Time series forecasting has long been pivotal in sectors such as finance, healthcare, meteorology, and supply chain management. Its primary objective is to predict future data points based on historical observations. However, the intricate and variable nature of time series data presents significant challenges in this endeavor. Recent advancements in machine learning, particularly the emergence of foundational models, have revolutionized this field by developing universal models capable of handling diverse time series without the need for specialized, case-specific training. These foundational models signify a major shift from traditional methods, which typically require multiple models tailored to specific datasets. Nonetheless, the diversity of time series characteristics, including frequency, seasonality, and underlying pattern variations, continues to pose substantial challenges for unified model training.

A major issue in time series forecasting is effectively managing data heterogeneity. Time series data from different sources exhibit significant differences in frequency, distribution, and structure. Current forecasting models often rely on human-defined, frequency-based specialization to address this diversity. However, frequency alone cannot reliably indicate time series patterns, as data with similar frequencies may exhibit different behaviors, while data with varying frequencies might display similar patterns. This approach must capture the inherent complexity and diversity present in real-world time series. Another challenge is the non-stationary nature of time series data, where statistical properties change over time, making accurate modeling using frequency-based grouping difficult.

Existing time series forecasting methods attempt to address data variability through various approaches. Models like TEMPO and UniTime integrate language-based prompts to help distinguish different data sources, achieving limited dataset-level specialization. Other models, such as TimesFM, maintain frequency-based embedding dictionaries to differentiate data types based on frequency. However, many models, including the widely recognized Chronos series, opt for general structures without specialized modules, increasing model complexity and requiring a large number of parameters. These approaches struggle to fully capture the diverse nature of time series data because frequency does not always correlate with underlying data patterns, leading to inefficiencies and reduced model accuracy.

Researchers from Salesforce AI Research, the National University of Singapore, and the Hong Kong University of Science and Technology have introduced an innovative model named MOIRAI-MoE. MOIRAI-MoE integrates a Mixture of Experts (MoE) within its Transformer architecture, allowing for token-level specialization without the need for manually defined frequency heuristics. This data-driven approach minimizes reliance on predefined frequency layers and utilizes a single input/output projection layer, enabling the model to automatically capture and represent diverse patterns. By implementing token-level specialization, MOIRAI-MoE offers a more flexible and effective solution, better representing the unique characteristics of different time series data without the need to develop separate models for each frequency category.

The architecture of MOIRAI-MoE employs a gating function that assigns each token to the appropriate expert within the Transformer layer based on token clustering derived from a pre-trained model. This clustering method is guided by the Euclidean distance to centroids, ensuring that tokens with similar patterns are handled by the same expert, while diverse tokens are managed by specialized experts. By incorporating 32 expert networks, each focusing on unique time series features, MOIRAI-MoE effectively reduces computational overhead and enhances its ability to generalize across different data types. This approach dynamically adapts to pattern changes within the data, enabling MOIRAI-MoE to excel in representing non-stationary time series data.

Extensive testing across 39 datasets demonstrates that MOIRAI-MoE outperforms in-distribution and zero-shot forecasting scenarios. In in-distribution forecasting, MOIRAI-MoE exceeds its dense model counterparts by up to 17%, achieving significant accuracy improvements while activating up to 65 times fewer parameters compared to leading models like TimesFM and Chronos. In zero-shot forecasting, where the model is tested on datasets not included in the training data, MOIRAI-MoE surpasses traditional models, showing a 3-14% improvement in Continuous Ranked Probability Score (CRPS) and an 8-16% improvement in Mean Absolute Scaled Error (MASE). These results highlight the model's robust generalization capabilities without the need for task-specific training.

This study provides key insights, underscoring the advancements of MOIRAI-MoE in time series forecasting:

· Data-Driven Specialization: MOIRAI-MoE overcomes the limitations of manually defined frequency specialization by achieving token-level specialization through sparse mixture of experts, allowing for more nuanced representation of time series diversity.

· Computational Efficiency: The model's sparse expert activation significantly reduces computational demands, cutting the number of activated parameters by up to 65 times while maintaining high accuracy.

· Performance Enhancement: Testing across various datasets confirms that MOIRAI-MoE surpasses dense models and foundational models like TimesFM and Chronos, achieving up to a 17% improvement in in-distribution tests.

· Scalability and Generalization: MOIRAI-MoE exhibits strong zero-shot performance, making it highly adaptable to real-world forecasting tasks without the need for specialized training for each application, which is critical for diverse applications in finance, healthcare, and climate modeling.

In conclusion, MOIRAI-MoE overcomes the limitations of frequency-based specialization by introducing a flexible, data-driven approach, representing a significant advancement in time series forecasting. Through its sparse mixture of experts architecture, MOIRAI-MoE addresses the diversity and non-stationarity of time series data, achieving substantial improvements in computational efficiency and performance. This novel approach highlights the potential of token-level specialization, paving the way for future enhancements in foundational time series models and expanding the practicality of zero-shot forecasting across various industries and applications.