Data assimilation: where a forecast starts

Numerical weather prediction (NWP) models require a precise and comprehensive understanding of the atmosphere's current state to initialise a forecast. This initial state, known as the analysis, is constructed by integrating a vast array of observations from diverse sources. These observations provide critical real-world data that correct and refine the model's previous forecast.

Key observational platforms include:

• Radiosondes: Weather balloons launched globally, typically twice daily (00:00 and 12:00 UTC), carrying instruments that measure pressure, temperature, humidity, and wind speed and direction at various altitudes up to 30 km. Met Éireann launches radiosondes from Valentia Observatory in County Kerry.
• Surface stations: Thousands of automated and manual stations on land reporting temperature, pressure, humidity, wind, and precipitation. In Ireland, Met Éireann operates a network of synoptic and climatological stations.
• Ships and buoys: Marine observations are crucial for ocean areas, providing surface conditions. The Marine Institute's network of weather buoys off the Irish coast offers vital data points.
• Aircraft: Commercial aircraft contribute valuable in-situ measurements, particularly wind and temperature, during ascent and descent, and at cruising altitudes.
• Satellites: Geostationary and polar-orbiting satellites provide a wealth of remote sensing data, including atmospheric temperature and humidity profiles, cloud motion vectors (used to derive winds), sea surface temperature, and land surface characteristics. These are particularly important for data-sparse regions like oceans.

Each observation type has its own characteristics regarding coverage, frequency, accuracy, and the specific atmospheric variables it measures. The assimilation system must account for these differences.

Data assimilation is not simply replacing a model's prediction with observations. Instead, it is a sophisticated statistical process that blends observations with a short-range forecast from the previous model run. This previous forecast, typically 3 to 12 hours old, is called the background field (or 'first guess'). It provides a physically consistent estimate of the atmosphere, even in areas where observations are sparse.

The assimilation system then calculates analysis increments, which are the adjustments made to the background field based on the new observations. These increments are not just simple differences between observation and background; they are weighted by the estimated error characteristics of both the observations and the background field. For example, if a radiosonde reports a temperature significantly different from the background, and the radiosonde is known to be highly accurate, the system will give more weight to the observation.

Consider a scenario where a model's background field predicts a 10 m wind speed of 15 km/h, but a nearby surface station measures 20 km/h. If the observation error variance is, for instance, 1 (km/h)² and the background error variance is 4 (km/h)², the optimal weight for the observation W_obs can be calculated as W_obs = BackgroundErrorVariance / (BackgroundErrorVariance + ObservationErrorVariance). In this case, W_obs = 4 / (4 + 1) = 0.8. The analysis increment would be 0.8 * (20 km/h - 15 km/h) = 4 km/h. The resulting analysis wind speed would be 15 km/h + 4 km/h = 19 km/h. This weighted average ensures that neither the model nor the observation fully dominates, leading to a more robust initial state.

This process ensures that the analysis is not only accurate where observations exist but also physically coherent and smooth across the entire model domain.

Modern data assimilation systems are highly complex, employing advanced mathematical techniques to optimally combine observations and the background field. Two prominent approaches are 4D-Var (Four-Dimensional Variational assimilation) and Ensemble Kalman Filters (EnKF).

4D-Var systems seek to find the atmospheric state that best fits both the observations over a specific time window (typically 6 to 12 hours) and the model's forecast trajectory. The 'four-dimensional' aspect refers to the three spatial dimensions plus time. It essentially runs the model forward and backward in time within the assimilation window, adjusting the initial state until the forecast trajectory best matches the observations. This requires the adjoint of the forecast model, which is computationally expensive but allows for a very consistent analysis.

Ensemble Kalman Filters (EnKF) take a different approach. Instead of a single background forecast, they use an ensemble of forecasts, each perturbed slightly to represent the uncertainty in the initial state. The observations are then used to update this ensemble, producing an ensemble of analyses. This approach naturally provides an estimate of the analysis uncertainty, which is crucial for ensemble forecasting systems.

Both methods aim to produce an initial state that is as close as possible to the true state of the atmosphere, given the available observations and the model's dynamics. The choice of method depends on computational resources, model complexity, and the specific requirements of the forecasting centre.

Satellite data and aircraft reports play a crucial role in filling observational gaps, particularly over oceans and remote areas where conventional surface and radiosonde networks are sparse.

Satellite winds are derived from tracking the movement of cloud features or water vapour patterns in sequential satellite images. These 'Atmospheric Motion Vectors' (AMVs) provide wind estimates at various atmospheric levels. While not direct measurements, their vast coverage makes them indispensable for global models. The accuracy of AMVs can vary depending on cloud type, height assignment, and the underlying atmospheric conditions.

Aircraft reports, primarily from commercial flights, are known as Aircraft Meteorological Data Relay (AMDAR) or Aircraft Communications Addressing and Reporting System (ACARS) data. These automated reports provide highly accurate measurements of wind and temperature along flight paths. They are particularly valuable during ascent and descent, offering detailed vertical profiles, and at cruising altitudes, providing observations in regions often devoid of other in-situ data. The density of these reports is highest over well-travelled air routes.

These data sources are continuously assimilated into NWP models, providing a dynamic and evolving picture of the atmosphere. Without them, the initial conditions for forecasts, especially for weather systems originating over the Atlantic, would be significantly less accurate.

Ireland's weather is predominantly influenced by systems moving in from the North Atlantic. This vast ocean basin is notoriously data-sparse compared to landmasses with dense networks of surface stations and radiosonde launch sites. While satellite and aircraft data help, they cannot fully compensate for the lack of direct, in-situ measurements across the entire ocean.

The consequences of these data gaps for Irish forecasts are significant:

• Increased initial condition uncertainty: Weather systems developing or tracking across the Atlantic have fewer observations to constrain their initial state in the assimilation process. This means the analysis, the starting point of the forecast, is inherently less certain.
• Faster error growth: Small errors in the initial conditions tend to grow over time. With less accurate initial states over the Atlantic, these errors can amplify more rapidly, leading to greater forecast divergence and reduced predictability as systems approach Ireland.
• Impact on ensemble spread: Ensemble forecasting systems, which explicitly model uncertainty, often show a wider spread for systems originating in data-sparse regions. This wider spread reflects the higher uncertainty in the initial conditions.

For example, a low-pressure system forming rapidly in the mid-Atlantic might be poorly observed in its nascent stages. If the initial position or intensity is slightly off in the analysis, the forecast for its track and severity as it impacts Ireland several days later can vary considerably between model runs. This is a common challenge for forecasters at Met Éireann.

Even with sophisticated data assimilation, the analysis is never perfect. There is always an analysis error, which represents the difference between the estimated atmospheric state and the true, unobservable state. This error arises from several sources:

• Observation error: All observations have inherent inaccuracies, whether from instrument limitations, representativeness errors (e.g., a point measurement representing a larger area), or transmission issues.
• Background error: The background field itself is a forecast and therefore contains errors from the previous model run.
• Representativeness error: The model grid cannot perfectly resolve all atmospheric phenomena, so even a perfect observation might not be perfectly represented within the model's discrete grid cells.
• Model error: The NWP model itself is a simplification of the atmosphere and contains approximations and parameterisations that introduce errors.

Analysis error is not static; it grows as the forecast progresses. This growth is a fundamental characteristic of chaotic systems like the atmosphere. Small initial errors can amplify rapidly through non-linear atmospheric dynamics, leading to significant divergence in forecasts over time. The rate of error growth is influenced by the atmospheric state itself; some weather patterns are inherently more predictable than others.

Understanding analysis error and its growth is critical for assessing forecast reliability. Ensemble forecasting systems explicitly attempt to quantify this uncertainty by running multiple forecasts from slightly perturbed initial conditions, reflecting the range of possible analysis states.

While operational forecasts are initialised with the most current analysis, reanalysis projects are distinct. Reanalysis involves running a fixed, state-of-the-art data assimilation system and NWP model over historical periods, often decades long, using all available observations (including those that were not available in real-time). The goal is to produce a consistent, comprehensive, and physically coherent record of the global atmosphere and ocean.

Key characteristics of reanalysis data:

• Consistency: Unlike operational forecasts, which evolve with model upgrades and changes in assimilation techniques, reanalysis uses a single, frozen system. This ensures that changes observed in the data reflect actual atmospheric variability rather than improvements in the forecasting system.
• Comprehensive: It incorporates a vast archive of historical observations, including those digitised long after their initial collection.
• Global coverage: Provides a continuous, gridded dataset for the entire globe, filling in gaps where observations were sparse or non-existent.

Reanalysis datasets, such as ERA5 from ECMWF or NCEP/NCAR Reanalysis, are invaluable for climatological studies. They allow scientists to:

• Characterise past climate variability and change: Identify trends, extreme events, and long-term patterns.
• Validate climate models: Compare model outputs with a consistent representation of the past atmosphere.
• Study atmospheric processes: Investigate the dynamics of weather systems and climate phenomena over extended periods.

For example, when The Wind Agent refers to typical wind speeds for Ireland based on ERA5 climatology, it is drawing on this consistent, high-quality historical record, which provides a more reliable baseline than a collection of disparate operational forecasts.