Methodology

How every score is made

A scoreboard is only worth trusting if its rules are public. These are ours, including the ways they can flatter or penalise a model. The statistical methods our own forecasts are built from are documented separately, under post-processing.

Station forecasts vs reality

The question this site answers first: you were forecast X; what did you actually get? For every station we score, every day, we take what each forecast said before the day began and put it next to what the station officially recorded.

1. The truth: the official NWS daily record

The observed high, low and precipitation are the station's official NWS daily climate record (the same numbers as its CLI climate report), taken from RCC-ACIS. The day is the station's local standard-time calendar day, midnight to midnight; during daylight saving time that runs 1 a.m. to 1 a.m. on the clock. A trace of rain counts as dry; measurable rain is 0.01" or more. The record usually lands the next morning, and the board is rebuilt every hour, so a day is scored as soon as its official numbers exist. If the NWS corrects a value later, the score is recomputed.

2. Which forecast counts: issued by 12Z, N days ahead

Forecasts are revised all day, so "what did NBM say?" needs a rule. For a target day D and a lead of N days, we take the last issue of each forecast at or before 12Z (7–8 a.m. Eastern) on day D−N. "Same morning" (N = 0) is the forecast you would have read at breakfast on the day itself; "the day before" (N = 1) is what you read the previous morning; and so on to a week ahead. Every source is held to the same clock. Same-morning lows are not scored, because by 12Z the overnight low has usually already happened.

3. The forecasts

Some sources run on slightly different day definitions (NBM and the NAM/GFS MOS use NWS 12-hour max/min windows; models use their output intervals). We score them all against the same official calendar day, because that is the number people see and the number markets settle on. A source that defines the day differently pays for it in the score, as it would in practice.

4. The stations

Featured stations (about 60) are the major airports and city sites where we carry every source and rain amounts. The network is every ASOS station with an official daily record that our MOS covers, about 1,200, scored on the National Blend, HRRR-MOS, RRFS-MOS, GFS, ECMWF HRES, GEFS mean and ECMWF EPS mean. Missing forecasts or observations remain unscored.

Some older Weather Trader global archives retained only 48 hours of forecasts. They contribute complete daily windows within that span, with later leads left missing. Historical paired-case counts therefore vary by model and lead; each comparison displays its actual sample.

The four national raw model products are extracted from Weather Trader's archived global grids at each station's nearest land cell. Incomplete historical GEFS intervals are recovered directly from NOAA's original member GRIBs, with member identities and forecast periods checked. HRES uses the operational forecast on ECMWF's published 0.25° open-data grid; EPS uses 50 perturbed members, excluding HRES. GEFS includes its control and 30 perturbed members. Daily ensemble highs and lows are the average of each member's own daily extreme, with decoded temperature precision retained through scoring. Native 3- or 6-hour extreme intervals are grouped by midpoint onto the local standard calendar day, without daylight saving changes. These intervals can shift the assembled day's boundaries by up to three hours; that difference is recorded in station provenance. Partial days, missing intervals and incomplete members are excluded. At most one unavailable ensemble member is tolerated. Availability remains an explicitly estimated conservative lag where the original publication timestamp is unavailable.

5. The numbers

6. Horizons

The same rules run from the nowcast (same morning) through tomorrow and the days after it to a week ahead, so every source's baseline skill is known at every horizon. Weeks 3–4, monthly and seasonal outlooks (CPC and the seasonal models) will be scored against the same station record as they are added.

7. Caveats

8. Quality control of the official record

Every score depends on the observation it is checked against, and the official record occasionally carries a bad value: a sensor fault or a keying error. Most big misses, though, are real busts (a sea breeze that never arrived, a thunderstorm outflow), and hiding those would defeat the purpose of the site. So an official value is held out only on two independent pieces of evidence:

  1. it misses the consensus of the forecasts (NBM, HRRR-MOS, RRFS-MOS, NWS, ECMWF, LAMP, GFS MOS) by 8°F or more, and
  2. the station's own hourly METAR reports for the same standard-time day disagree with it (the official high may sit up to 6°F above the highest hourly report, since the official value catches short peaks, but not more than 3°F below it; the low the mirror image), and the reports sit closer to the forecasts than the official value does.

A value the station's own reports back up is never held, however badly the forecasts missed it. Where there are no hourly reports to check, only an extreme value (15°F or more off the forecasts, and 12°F or more unlike the misses at neighbouring stations within 150 km that day) is held. Rain totals of 4" or more that no forecast put above 0.10" are held for review. Values outside physical limits, or a low above the high, are always held.

Held values are not scored. They are shown struck through on the station page and listed below. The checks run on every hourly build, so an official correction clears the hold automatically.

Loading the current holds…

Global models

The global-model standings score the large-scale pattern rather than station weather.

G1. The Standings metric: Day-5 pattern score

The headline number is the anomaly correlation coefficient (ACC) of 500 hPa geopotential height over the Northern Hemisphere, 20–80°N, multiplied by 100. It is the standard measure centres use to track medium-range skill: how well a forecast places the ridges and troughs that steer the weather, relative to climatology.

ACC = Σ w (f − c)(a − c) / √[ Σ w (f − c)² · Σ w (a − c)² ]
f = forecast, a = verifying analysis, c = climatology, w = latitude (area) weight

100 is a perfect pattern forecast. By convention a score below 60 is no longer a useful synoptic forecast, and the lead time where a model crosses 60 is its useful-forecast limit. Scores are also computed for Day 1–10 and for seven regions (Northern Hemisphere, Arctic, North Pacific, North America, North Atlantic, Europe, Asia).

G2. One common truth, and home field

Every model is graded against the same verifying analysis: ECMWF's operational analysis. Centres usually verify against their own analysis, which makes cross-centre comparisons unfair. One truth removes that, with one exception we flag openly. ECMWF HRES plays at home (marked HOME): it is graded against an analysis produced by the same system, which gives it a small advantage, especially at short leads.

G3. Games, windows and freshness

Each model run (00Z and 12Z) is a game. A run's Day-5 score exists five days after it starts, so the latest results always describe a run from five days ago. Rolling windows ("last 30 days") are measured by verification date and end at the most recent verification, so a late pipeline never silently shrinks the sample. Data is rebuilt every hour.

G4. Elo ratings

On every game, each pair of models that both verified plays a match: the higher Day-5 score wins, and scores within 0.05 are a draw. Ratings start at 1500 and update with the standard Elo expectation. The K-factor of 32 is divided across the field, so one game moves a rating by at most about 32 points however many models play:

E(A beats B) = 1 / (1 + 10^((R_B − R_A)/400))
ΔR_A = 32 × Σ_B (S_AB − E_AB) / (n − 1)

Elo rewards consistently beating the field, not a high average: a model that beats everyone narrowly every day outrates one that alternates big wins and big losses. The Power Rankings sort by Elo. A model with no runs in the last three days is shown as OFF, its rating frozen and its rank removed.

G5. Records, wins, streaks, form and awards

G6. Known caveats (read before you tweet)

G7. Humans vs Machines vs Money: daily highs

Point forecasts of the daily high temperature at U.S. stations, verified against the official settlement value of the daily-high contracts (°F). The scores are mean absolute error by lead day. Each source is scored on every station-day it forecast, so samples are not identical. NWS gridded forecasts (NDFD, the human-edited product) publish to Day 7.

The market is treated as a forecaster. Its forecast is the expected value of the price-implied probability distribution across the daily-high contract ladder, taken one day before. It is compared on the same station-days with the National Blend, a multi-model consensus, ECMWF HRES and the GEFS mean. "Closest on the day" counts the station-days on which each source was nearest the observed high, splitting ties.

G8. Hurricane League

Hurricane scores follow National Hurricane Center verification conventions, computed from NHC's public ATCF files:

Sources and credit

Official daily highs, lows and precipitation: the NWS climate record via the NOAA Regional Climate Centers' Applied Climate Information System (RCC-ACIS). Station guidance: the National Blend of Models, NDFD and LAMP from NOAA/NWS; GFS and NAM MOS via the Iowa Environmental Mesonet MOS archive; DWD MOSMIX from DWD open data. HRRR-MOS and RRFS-MOS are produced by Weather Trader from NOAA HRRR and RRFS output. RRFS, HRRR, GFS, AIGFS, the National Blend of Models and the NDFD archive from the NOAA Open Data Dissemination program (AWS); ECMWF IFS and AIFS from ECMWF open data (CC-BY-4.0). Basemap © OpenStreetMap contributors via Protomaps, served by Forecast Eagle. Model data from ECMWF, NOAA/NCEP, DWD, Environment and Climate Change Canada, the UK Met Office and Google DeepMind (WeatherNext). Hurricane forecasts and best tracks: NOAA National Hurricane Center (ATCF). Daily-high contract prices and settlements: ForecastEx. Verification computed by the Weather Trader pipeline. Forecast Scores is independent and not endorsed by any of these organisations.