Observations versus model: how to compare fairly

Before any meaningful comparison between observed wind data and a numerical weather prediction (NWP) model forecast can occur, fundamental parameters must be aligned. The most critical are measurement height, averaging period, and units.

Height: Meteorological observations are conventionally reported at 10 metres above ground level (AGL). NWP models also typically provide a 10 m wind output. However, if an observation station is at a different height, or if the model output is for a different level (e.g., 80 m for a wind turbine hub), direct comparison is inappropriate. Wind shear means speed changes significantly with height; a 10 m observation cannot be directly compared to an 80 m forecast without accounting for this.

Averaging Period: The World Meteorological Organisation (WMO) standard for mean wind speed is a 10-minute average. Gusts are typically defined as the highest 3-second average within that 10-minute period. Observation stations adhere to these standards. NWP models, however, often output instantaneous values or averages over their model timestep (e.g., 1 hour). A 1-hour model average will almost always be lower than a 10-minute observed average during periods of varying wind, and will certainly miss short-term fluctuations that contribute to observed gusts. The Wind Agent standardises observed and modelled wind to 10-minute averages for mean speed and 3-second maximum for gust where possible.

Units: Ensure both datasets use consistent units (e.g., metres per second, knots, kilometres per hour). Inconsistent units will lead to incorrect conclusions, as will comparing a forecast in mph to an observation in km/h. The instrument allows selection of preferred units and converts all data accordingly.

A fundamental difference between an observation and a model forecast lies in their spatial representation. An observation station provides a point measurement: the wind conditions precisely at the anemometer's location. This measurement is influenced by local topography, obstacles, and micro-scale turbulence specific to that single point.

Conversely, an NWP model provides a grid-cell average. The model atmosphere is divided into a grid of cells, typically 2.5 km to 9 km horizontally for regional models used in Ireland (e.g., Met Éireann's HARMONIE-AROME has a 2.5 km resolution). The forecast for a given location represents the average wind conditions across that entire grid cell, not the specific conditions at a single point within it. This averaging process inherently smooths out local variations and small-scale features.

For example, a station located on a exposed headland within a 5 km grid cell might consistently record higher wind speeds than the model's average for that cell, which also includes sheltered bays or inland areas. Similarly, a station in a valley might record lower speeds. This difference is not necessarily a model error, but a consequence of differing spatial scales. The model's representation is a generalisation, while the observation is a specific detail.

This distinction is particularly relevant in areas with complex terrain or coastlines, where local effects can cause significant deviations from the broader grid-cell average. The Wind Agent attempts to select the most representative grid cell for a given location, but the inherent difference remains.

When evaluating model performance against observations, several metrics are commonly used:

• Bias: This is the average difference between the forecast and the observation. A positive bias means the model generally over-forecasts, while a negative bias means it under-forecasts. It indicates a systematic error. For instance, if the model consistently forecasts 12 m/s when the observation is 10 m/s, the bias is +2 m/s. Bias is simple to calculate: Bias = Σ(Forecast - Observation) / N.

• Mean Absolute Error (MAE): This is the average of the absolute differences between forecast and observation. It measures the typical magnitude of error, without regard to direction. MAE is useful because it is easy to interpret: an MAE of 1.5 m/s means, on average, the forecast is off by 1.5 m/s. It does not penalise large errors disproportionately. MAE = Σ|Forecast - Observation| / N.

• Root Mean Square Error (RMSE): This is the square root of the average of the squared differences between forecast and observation. RMSE gives more weight to larger errors due to the squaring. This makes it sensitive to outliers and extreme events. A model with fewer large errors will have a lower RMSE than one with many small errors, even if their MAE is similar. RMSE = √[Σ(Forecast - Observation)² / N].

Worked Example: Consider three forecasts and observations:

• Bias: (1 + 2 - 1) / 3 = 2 / 3 = +0.67 m/s (model slightly over-forecasts)
• MAE: (1 + 2 + 1) / 3 = 4 / 3 = 1.33 m/s (average error magnitude)
• RMSE: √[(1 + 4 + 1) / 3] = √(6 / 3) = √2 ≈ 1.41 m/s (penalises the +2 error more than MAE)

Understanding the nature of the discrepancy between observations and models is crucial for effective use of forecasts. Discrepancies can generally be categorised as either persistent bias or random noise.

Persistent bias refers to a systematic, consistent difference. For example, if a model consistently under-forecasts wind speed at a particular coastal location by 2 m/s, this is a persistent negative bias. Such biases are often linked to the model's representation of local topography, surface roughness, or thermal effects not fully captured by its grid resolution. Once identified, persistent biases can sometimes be compensated for through statistical post-processing or by a user's informed adjustment of the forecast.

Random noise, or unsystematic error, refers to unpredictable, short-term fluctuations where the model's error varies in direction and magnitude. This is often due to the chaotic nature of the atmosphere and the model's inability to perfectly resolve all scales of motion. Turbulence, small convective cells, or transient local effects contribute to noise. Random noise cannot be easily corrected for in the same way as bias; it represents the inherent uncertainty in forecasting. The ensemble system addresses this by providing a range of possible outcomes.

Distinguishing between the two is important. A model with a small, consistent bias might still be highly valuable if that bias is understood. A model with high random noise, even if its overall bias is zero, might be less reliable for specific, short-term decisions.

NWP models generally struggle more with accurately forecasting gusts than mean wind speeds. There are several reasons for this:

1. Spatial Resolution: Gusts are by definition short-lived, small-scale phenomena. They are often associated with turbulent eddies that are much smaller than the typical grid cell size of an NWP model. The model's equations cannot explicitly resolve these sub-grid scale processes. Instead, gusts are parameterised, meaning they are estimated using statistical relationships based on the mean wind and atmospheric stability within the grid cell.

1. Turbulence Parameterisation: The schemes used to represent turbulence and its impact on gusts are simplifications of complex physics. They may not perfectly capture the interaction of wind with specific terrain features or the dynamics of convective downdrafts that generate strong gusts.

1. Convective Gusts: In convective conditions (e.g., thunderstorms or strong showers), gusts are often caused by downdrafts bringing high momentum air from aloft rapidly to the surface. These are particularly challenging for models, as the precise timing and location of such events are hard to predict accurately at the grid-cell scale.

1. Observation Variability: Observed gusts are the maximum 3-second average in a 10-minute period. This is a highly variable quantity. Even small shifts in the timing or location of turbulent features can lead to large discrepancies between a point observation and a model's grid-cell average gust estimate.

Consequently, it is common to see larger errors, both in magnitude and timing, when comparing modelled gusts to observed gusts, compared to mean wind speeds. Users should interpret gust forecasts with an understanding of this inherent modelling challenge.

The past_week chart in The Wind Agent provides a direct visual comparison of observed wind data against model forecasts for your selected location over the preceding seven days. This chart is a powerful tool for understanding local model performance.

How to interpret it:

• Mean Wind Speed: Look for how closely the modelled mean speed (often a solid line) tracks the observed mean speed (often a dashed line or dots). Are there consistent periods where the model is too high or too low? This indicates bias.
• Gusts: Compare the modelled gust (often a shaded area or higher line) with observed gusts. Are the peaks aligned? Does the model capture the magnitude of the observed gusts? As discussed, you might see larger discrepancies here.
• Direction: The chart typically includes wind direction. Does the model correctly predict shifts in wind direction? A consistent directional offset can be as important as speed bias, especially for operations sensitive to crosswind components.
• Diurnal Patterns: Observe if the model accurately captures daily cycles of wind speed, such as lighter winds at night and stronger winds during the day, or vice-versa depending on stability.

By regularly reviewing the past_week chart for your specific operational location, you can develop an intuitive understanding of the model's strengths and weaknesses in that particular environment. This empirical knowledge is invaluable for making informed decisions, especially when forecasts are near your operational limits. For example, if the chart consistently shows the model under-forecasting gusts by 5 km/h in a specific wind direction, you can mentally adjust the forecast accordingly.

While NWP models offer a broad, comprehensive view of atmospheric conditions, there are specific situations where a local observation station's data should take precedence, or at least be given significant weight:

1. Real-time Decision Making: For immediate operational decisions, a live observation from a well-sited station at your exact location provides the most accurate and current information. The forecast, by its nature, is a prediction, not a real-time measurement. The Wind Agent's grounded agent provides this live data.

1. Sudden, Localised Events: Phenomena like sudden squalls, microbursts, or highly localised sea breezes can develop and intensify rapidly. A local station will register these immediately, whereas a model might only capture them with a delay or smooth them out due to its resolution.

1. Known Local Biases: If historical past_week charts and verification data consistently show a model bias at your specific location (e.g., always under-forecasting in a particular wind direction), and the observation deviates from the forecast in a way consistent with that bias, the observation provides a more reliable indicator of the true conditions.

1. Complex Terrain: In areas with highly complex topography (deep valleys, sharp ridges, specific coastal features), the model's grid-cell average may struggle to represent the highly localised flow. A properly sited station can capture these microclimates more accurately.

1. Discrepancy with Ensemble Spread: If the observed conditions fall outside the range of the ensemble forecast (e.g., below P10 or above P90), it indicates the atmosphere is evolving in a way not well captured by the model's initial conditions or physics. In such cases, the observation signals a departure from the modelled scenarios.

The continuous comparison of observations against model forecasts is not merely an academic exercise; it is a critical input for calibration and refinement of operational procedures. The Wind Agent's Agreement Spine and calibration features are designed to facilitate this process.

Calibration involves:

1. Identifying Systematic Errors: By regularly monitoring the past_week chart and the Agreement Spine, users can identify persistent biases in the model's performance for their specific location and operational height. For example, a consistent tendency for the model to over-forecast wind speed by 10% during offshore flows, or to miss nocturnal inversions that lead to strong shear.

1. Adjusting Operational Limits: If a persistent bias is identified, operational limits can be adjusted. For instance, if the model consistently under-forecasts gusts by 5 km/h, an operator might effectively lower their internal gust limit by 5 km/h when interpreting the forecast, knowing the actual risk is higher.

1. Refining Decision-Making Protocols: Understanding model limitations and biases allows for more nuanced decision-making. Instead of a rigid adherence to a single forecast number, operators can incorporate the probability of exceedance (from the exceedance fan) and their learned understanding of local model behaviour.

1. Improving Siting and Exposure: Discrepancies can also highlight issues with observation station siting. If an observation consistently differs from multiple models in an unexpected way, it might indicate the station is in an unrepresentative location or is affected by local obstacles. This feedback can inform better sensor placement.

This iterative process of observe-compare-learn-adjust enhances the utility of both observations and forecasts, leading to safer and more efficient operations.