Week two: the score, good and bad
Blog · news

Week two: the score, good and bad

Lesson from:

Last week's electricity and gas forecasts against what was published, three lost days, and what LSN News saw in 2 million more articles

Last Sunday I published a week of LSN News in numbers, with forecasts for the days ahead, and promised to publish the score, good or bad. This is the score.

Short version: the electricity forecasts held their ranges about four hours in five, the gas forecasts landed within one percentage point, and two of the three price zones lost three days because the prices they depend on were not published.

Contents · 8 sections

The score: electricity #

The forecast is for the next day’s hourly prices, issued at 11:15 Brussels time before the auction closes. “Inside” counts the hours where the published price fell within the stated range, which is meant to hold 8 times in 10. “Simple guess” is repeating the same hour from a week earlier.

Romania

Day Forecast, day average Published Error per hour Hours inside Simple guess, error
Mon 16 Nov 231 247 29 21 of 24 141
Tue 17 Nov 248 233 25 21 of 24 177
Wed 18 Nov 232 191 42 15 of 24 162
Thu 19 Nov 197 216 36 22 of 24 79
Fri 20 Nov 202 not published cannot be scored
Sat 21, Sun 22 Nov not issued

Hungary

Day Forecast, day average Published Error per hour Hours inside Simple guess, error
Mon 16 Nov 232 242 25 23 of 24 131
Tue 17 Nov 249 245 13 24 of 24 156
Wed 18 Nov 233 196 39 20 of 24 159
Thu 19 Nov 225 234 21 24 of 24 58
Fri 20 Nov 218 not published cannot be scored
Sat 21, Sun 22 Nov not issued

Bulgaria

Day Forecast, day average Published Error per hour Hours inside Simple guess, error
Mon 16 Nov 229 246 25 24 of 24 125
Tue 17 Nov 255 227 33 18 of 24 176
Wed 18 Nov 225 182 44 16 of 24 167
Thu 19 Nov 187 196 27 23 of 24 93
Fri 20 Nov 186 123 72 14 of 24 73
Sat 21 Nov 148 176 29 21 of 24 32
Sun 22 Nov 172 188 31 20 of 24 28

All figures are in EUR/MWh. The Monday row replaces the provisional 249 / 244 / 249 I printed last week, which were averages of 23 of the 24 hours.

Over the scored days:

Zone Days scored Hours inside the range Average error per hour Simple guess
Romania 4 79 of 96 (82%) 33 140
Hungary 4 91 of 96 (95%) 24 126
Bulgaria 7 136 of 168 (81%) 37 99

Where it missed #

  • Wednesday 18 November, all three zones. Prices fell by about 40 EUR/MWh more than forecast. Romania had only 15 of 24 hours inside its range.
  • Friday 20 November, Bulgaria. The forecast said 186 and the market published 123. That is the worst day of the week: an error of 72 per hour, no better than the simple guess.
  • Sunday 22 November, Bulgaria. The simple guess was closer than the model, 28 against 31.

Why the comparison flatters the model #

The week before was unusually expensive. Repeating last week’s prices was therefore a very bad guess from Monday to Thursday, which makes the model look better than it is. By the weekend the two were level. One week does not show skill; the tests on hundreds of earlier days do, and this week neither confirms nor overturns them.

The three lost days #

For Friday 20 and Saturday 21 November, the market data source did not publish day-ahead prices for Romania and Hungary. Friday’s forecast was issued and cannot be scored. Saturday’s and Sunday’s were not issued at all, because the model needs the previous day’s prices.

The forecast pages said so for each zone, and now also say since when. One page got it wrong: for a few hours on Friday the energy page showed an older price as if it were current, until it was fixed.

The score: gas storage #

Forecasts made on 15 November for 20 November, in percent full:

Country Forecast (range) Published Off by “No change” guess, off by
Romania 82.9 (81.9 to 83.3) 83.4 0.5 1.4
Czechia 69.8 (68.5 to 71.0) 69.6 0.2 1.9
Netherlands 59.1 (57.9 to 61.0) 59.8 0.7 1.5
Spain 79.6 (78.5 to 80.1) 80.0 0.4 0.2
Poland 95.2 (93.7 to 96.3) 95.4 0.2 0.5
  • Four of five inside the range. Romania’s level came in 0.1 above the top of its range.
  • Spain is a miss in the other sense: storage there barely moved, so guessing “no change” was closer than the model.
  • Four of the five forecasts were slightly too low. Storage emptied a little more slowly than forecast.

The forecasts for 27 November from last week’s table still stand and will be scored next Sunday.

The week in numbers #

Seven days, 16 to 22 November, against the seven before. The last day was still running when I counted, at 15:30 UTC.

What 9 to 15 Nov 16 to 22 Nov
Articles read 2.05 million 2.03 million
News sites 16,140 16,298
Languages 74 74
Stories the articles were grouped into 1.34 million 1.04 million
Stories carried by 5 or more outlets 15,532 24,915

The busiest day was Wednesday 18 November, with 352,875 articles. English was again a third (672,000), then Spanish (222,000), German (110,000), Chinese (108,000) and Russian (103,000). Romanian was eleventh, with 44,400.

Fewer stories, and more of them shared, is a repair and not a change in the news. A fault had left many reports of the same event as stories of their own. About 70,000 of them were joined to the stories they belong to. In the same week about 29,000 pages that were never news were taken out: a parts catalogue, property listings, tag pages, job adverts. Horoscope columns were kept as articles but are no longer grouped into stories. Together that retired 28,800 “stories” that held nothing else.

The stories most outlets carried #

As last week: outlets that carried the story, and how many used a headline of their own.

Story Outlets Distinct voices Languages
Summit of the United States, China and Russia, and its aftermath 1,596 1,402 56
Death of a well-known hippo at a zoo in Cincinnati 1,265 665 34
Online banking outages at banks in the United States 1,091 620 39
A classified US government network down for days 1,084 688 45
Capture of a cartel leader in Mexico 1,061 692 48
Poland investigates a missile entering its airspace 967 844 47

These are events as outlets published them. I did not verify them elsewhere.

And the small stories that travel as one text:

Story Outlets Distinct voices
A report that younger managers will soon outnumber older ones 613 13
Parents complain that school photos look like police photos 578 5
A local election in Florida decided by a coin toss 563 13
Six women honoured on an entrepreneurship day 403 5

The school-photo story reached 578 outlets with five headlines between them. It is one text, published the same way by hundreds of local sites.

Protests and strikes: what was reported #

Counts of events reported in the news and coded by the GDELT project, for 15 to 21 November against the four weeks before. They say what was published, not what happened.

  • Reported protests were about a fifth lower: around 710 a day worldwide, against about 880. All reported events together were slightly up, at 104% of the earlier level.
  • The fall is still the end of October’s waves: France at 25 a day against 81, India 63 against 119, the United States 162 against 204.
  • Two countries moved the other way. The Philippines had 41 reported protests a day against a usual 5, and Bangladesh 25 against 10.
  • Reported strikes were flat, 48 a day against 46. In the United States they rose from 13 a day to 21.

No forecast is printed here. The counts are the record.

What the system forecasts next #

The rule is unchanged. A forecast is of a figure that will be published. It is shown only where the model beat a simple guess on days it had never seen.

Electricity, Monday 23 November. Issued Sunday at 11:15 Brussels time. Day average and range, EUR/MWh:

Zone Forecast Range (P10 to P90)
Romania 228 172 to 293
Hungary 235 183 to 294
Bulgaria 208 145 to 277
Austria 229 181 to 280
Belgium 216 169 to 263
Czechia 231 193 to 294
Germany and Luxembourg 216 167 to 296
France 171 58 to 229
Netherlands 204 159 to 281
Poland 177 78 to 271
Slovakia 239 182 to 307

The eight zones below Bulgaria are new. They passed the same test, so they are shown.

Gas storage, percent full, for the six countries where the model has skill. Denmark is new:

Country 28 November 5 December
Romania 81.0 (79.5 to 81.7) 78.1 (76.1 to 79.6)
Czechia 65.9 (65.2 to 67.3) 62.0 (61.0 to 64.9)
Netherlands 56.1 (55.0 to 58.6) 52.8 (49.9 to 56.2)
Spain 79.3 (78.7 to 79.8) 78.2 (76.6 to 79.2)
Poland 93.9 (92.5 to 95.0) 92.4 (90.5 to 94.1)
Denmark 84.6 (82.3 to 86.9) 82.3 (78.8 to 85.9)

Every other topic: what moved from “not yet” #

Topic Last week Now
Electricity prices in eight more zones In first test runs Shown. All eight passed
Electricity demand In development Shown in 25 of 32 zones
Reported protests, strikes, armed clashes and coercion, per country, 7 days ahead No skill yet Shown where the test is passed: protests in 27 of 33 countries, strikes in 13 of 16, clashes in 97 of 122, coercion in 104 of 113
Gas storage in more countries No skill in 12 Skill in 6 of 17; no skill in 11
Wind and solar output In development First tests
Prices per quarter-hour Not yet. The market publishes per quarter-hour; the forecast is hourly
How long a story stays in the news Not built Not built
Elections, markets, named people or outlets Never Never

For protests, strikes, clashes and coercion, the target is the number of reported events in a country, from coded news data. It is not a risk score for a country, and not a claim that something will happen. Those forecasts are on the site with their track record. I print none here until they have a score of their own to show.

What to take from week two #

  • The ranges held about as often as promised: 81 to 95% of hours, against the 80% they claim. The middle of the forecast was off by 24 to 37 EUR/MWh per hour.
  • The worst failure was not a wrong number. It was three days with no forecast, because a figure the model depends on was never published. The honest answer on such a day is “not issued”, and for a few hours one page did not give it.
  • Counting changed more than the news did. A fifth fewer stories, and sixty percent more of them shared by several outlets, came from fixing how articles are grouped.

Next Sunday: the score for this table.