WP-MIP Standardizes Weather Model Comparisons

Machine-learning weather models now compete with established numerical weather prediction systems on some medium-range scores, but the comparison problem has not been solved. Centers initialize models differently, verify against different analyses, archive different fields, and emphasize different use cases. WP-MIP addresses that gap as infrastructure rather than as a new forecasting model: it defines a shared intercomparison project for physically based, AI-based, and hybrid prediction systems.

The paper’s central claim is pragmatic. If national centers are going to use AI weather prediction in operations, they need a common archive and verification workflow that can reveal where each model class is skillful, physically credible, and operationally useful. The first WP-MIP instance focuses on global deterministic forecasts, with later expansion paths for probabilistic and ensemble prediction.

Core Contribution

WP-MIP’s main contribution is the protocol design. The project separates forecasts into two initialization streams: an own initial conditions stream, where each center uses its operational or preferred initialization, and a same initial conditions stream, where models start from common ECMWF initial states. That split is the important design choice. OIC reflects what a forecasting center can deliver in practice; SIC removes part of the initialization confound and lets evaluators focus more directly on model behavior.

Figure 1 summarizes this structure, including the core protocol and two subprojects.

The paper also defines WP-MIP as a distributed evaluation effort, not a single leaderboard. Forecasts are stored centrally, but verification is meant to be developed by participating groups. That matters because the open questions are broader than headline error scores: physical consistency, extremes, spatial structure, tropical phenomena, subseasonal signals, and regional forecast value all require different diagnostics.

Technical Approach

The project compares three model families: conventional physical models, AI weather prediction systems, and hybrid systems that combine data-driven components with physical-model machinery. The first phase uses global deterministic forecasts and a common 0.25° WP-MIP grid for shared diagnostics. The figures show evaluation against multiple reference analyses, with uncertainty estimated by a 1000-member bootstrap in the spectral diagnostics.

The protocol’s verification program is wider than pointwise error. The paper discusses conventional bias and RMSE for global 500 hPa temperature, spectral kinetic-energy diagnostics for 250 hPa winds at day 10, falsification tests for physically impossible states, spatial verification for feature structure and displacement errors, and event-oriented evaluation for extremes. The falsification section is especially relevant for AI systems: the authors cite negative precipitation as a clear example of an impossible atmospheric prediction, then broaden the point to conservation and process-level consistency.

Figure 7 shows the spectral amplitude-ratio diagnostic used to compare kinetic-energy structure against the analysis reference. It is a useful example of the paper’s broader argument: a model can have competitive scalar scores while still putting too much or too little variance at specific spatial scales.

Results and Analysis

This is not a paper that reports a finished WP-MIP ranking. Its evidence is a project archive, protocol specification, and sample diagnostics from early contributed forecasts. The 500 hPa temperature figures compare bias and RMSE across physical, AI, and hybrid models, with ECMWF’s physical forecast plotted as a reference in the AI and hybrid panels. Those plots are designed less to crown a winner than to show how OIC and SIC comparisons expose different sources of error.

The spectral figures are more diagnostic. At day 10 for global 250 hPa winds, the AI-model panels show a common loss of small-scale kinetic energy at high global wavenumber, while hybrid systems are closer to the analysis spectrum in the shown cases. Physical models vary more by center at small scales, which is expected because their numerics, diffusion, and resolution choices differ. The paper’s interpretation is careful: these diagnostics identify behavior that standard RMSE can hide, but they do not by themselves determine operational value.

The strongest part of the work is the protocol: OIC versus SIC, multi-analysis verification, common gridding, 6-hourly fields, and a planned set of phenomenon-based studies. The weaker part is timing. The archive covers a 1-year period, which the authors explicitly describe as limited for rare or extreme events. The protocol also lacks ensemble-based prediction in its first instance, even though recent AI successes suggest that future intercomparisons will need probabilistic components. For operational centers, the paper is most useful as a blueprint for how to evaluate next-generation forecast systems without reducing the question to one global score.