Week two: the score, good and bad
Lesson from:
Last week's electricity and gas forecasts against what was published, three lost days, and what LSN News saw in 2 million more articles
Last Sunday I published a week of LSN News in numbers, with forecasts for the days ahead, and promised to publish the score, good or bad. This is the score.
Short version: the electricity forecasts held their ranges about four hours in five, the gas forecasts landed within one percentage point, and two of the three price zones lost three days because the prices they depend on were not published.
Contents · 8 sections
The score: electricity #
The forecast is for the next day’s hourly prices, issued at 11:15 Brussels time before the auction closes. “Inside” counts the hours where the published price fell within the stated range, which is meant to hold 8 times in 10. “Simple guess” is repeating the same hour from a week earlier.
Romania
| Day | Forecast, day average | Published | Error per hour | Hours inside | Simple guess, error |
|---|---|---|---|---|---|
| Mon 16 Nov | 231 | 247 | 29 | 21 of 24 | 141 |
| Tue 17 Nov | 248 | 233 | 25 | 21 of 24 | 177 |
| Wed 18 Nov | 232 | 191 | 42 | 15 of 24 | 162 |
| Thu 19 Nov | 197 | 216 | 36 | 22 of 24 | 79 |
| Fri 20 Nov | 202 | not published | cannot be scored | ||
| Sat 21, Sun 22 Nov | not issued |
Hungary
| Day | Forecast, day average | Published | Error per hour | Hours inside | Simple guess, error |
|---|---|---|---|---|---|
| Mon 16 Nov | 232 | 242 | 25 | 23 of 24 | 131 |
| Tue 17 Nov | 249 | 245 | 13 | 24 of 24 | 156 |
| Wed 18 Nov | 233 | 196 | 39 | 20 of 24 | 159 |
| Thu 19 Nov | 225 | 234 | 21 | 24 of 24 | 58 |
| Fri 20 Nov | 218 | not published | cannot be scored | ||
| Sat 21, Sun 22 Nov | not issued |
Bulgaria
| Day | Forecast, day average | Published | Error per hour | Hours inside | Simple guess, error |
|---|---|---|---|---|---|
| Mon 16 Nov | 229 | 246 | 25 | 24 of 24 | 125 |
| Tue 17 Nov | 255 | 227 | 33 | 18 of 24 | 176 |
| Wed 18 Nov | 225 | 182 | 44 | 16 of 24 | 167 |
| Thu 19 Nov | 187 | 196 | 27 | 23 of 24 | 93 |
| Fri 20 Nov | 186 | 123 | 72 | 14 of 24 | 73 |
| Sat 21 Nov | 148 | 176 | 29 | 21 of 24 | 32 |
| Sun 22 Nov | 172 | 188 | 31 | 20 of 24 | 28 |
All figures are in EUR/MWh. The Monday row replaces the provisional 249 / 244 / 249 I printed last week, which were averages of 23 of the 24 hours.
Over the scored days:
| Zone | Days scored | Hours inside the range | Average error per hour | Simple guess |
|---|---|---|---|---|
| Romania | 4 | 79 of 96 (82%) | 33 | 140 |
| Hungary | 4 | 91 of 96 (95%) | 24 | 126 |
| Bulgaria | 7 | 136 of 168 (81%) | 37 | 99 |
Where it missed #
- Wednesday 18 November, all three zones. Prices fell by about 40 EUR/MWh more than forecast. Romania had only 15 of 24 hours inside its range.
- Friday 20 November, Bulgaria. The forecast said 186 and the market published 123. That is the worst day of the week: an error of 72 per hour, no better than the simple guess.
- Sunday 22 November, Bulgaria. The simple guess was closer than the model, 28 against 31.
Why the comparison flatters the model #
The week before was unusually expensive. Repeating last week’s prices was therefore a very bad guess from Monday to Thursday, which makes the model look better than it is. By the weekend the two were level. One week does not show skill; the tests on hundreds of earlier days do, and this week neither confirms nor overturns them.
The three lost days #
For Friday 20 and Saturday 21 November, the market data source did not publish day-ahead prices for Romania and Hungary. Friday’s forecast was issued and cannot be scored. Saturday’s and Sunday’s were not issued at all, because the model needs the previous day’s prices.
The forecast pages said so for each zone, and now also say since when. One page got it wrong: for a few hours on Friday the energy page showed an older price as if it were current, until it was fixed.
The score: gas storage #
Forecasts made on 15 November for 20 November, in percent full:
| Country | Forecast (range) | Published | Off by | “No change” guess, off by |
|---|---|---|---|---|
| Romania | 82.9 (81.9 to 83.3) | 83.4 | 0.5 | 1.4 |
| Czechia | 69.8 (68.5 to 71.0) | 69.6 | 0.2 | 1.9 |
| Netherlands | 59.1 (57.9 to 61.0) | 59.8 | 0.7 | 1.5 |
| Spain | 79.6 (78.5 to 80.1) | 80.0 | 0.4 | 0.2 |
| Poland | 95.2 (93.7 to 96.3) | 95.4 | 0.2 | 0.5 |
- Four of five inside the range. Romania’s level came in 0.1 above the top of its range.
- Spain is a miss in the other sense: storage there barely moved, so guessing “no change” was closer than the model.
- Four of the five forecasts were slightly too low. Storage emptied a little more slowly than forecast.
The forecasts for 27 November from last week’s table still stand and will be scored next Sunday.
The week in numbers #
Seven days, 16 to 22 November, against the seven before. The last day was still running when I counted, at 15:30 UTC.
| What | 9 to 15 Nov | 16 to 22 Nov |
|---|---|---|
| Articles read | 2.05 million | 2.03 million |
| News sites | 16,140 | 16,298 |
| Languages | 74 | 74 |
| Stories the articles were grouped into | 1.34 million | 1.04 million |
| Stories carried by 5 or more outlets | 15,532 | 24,915 |
The busiest day was Wednesday 18 November, with 352,875 articles. English was again a third (672,000), then Spanish (222,000), German (110,000), Chinese (108,000) and Russian (103,000). Romanian was eleventh, with 44,400.
Fewer stories, and more of them shared, is a repair and not a change in the news. A fault had left many reports of the same event as stories of their own. About 70,000 of them were joined to the stories they belong to. In the same week about 29,000 pages that were never news were taken out: a parts catalogue, property listings, tag pages, job adverts. Horoscope columns were kept as articles but are no longer grouped into stories. Together that retired 28,800 “stories” that held nothing else.
The stories most outlets carried #
As last week: outlets that carried the story, and how many used a headline of their own.
| Story | Outlets | Distinct voices | Languages |
|---|---|---|---|
| Summit of the United States, China and Russia, and its aftermath | 1,596 | 1,402 | 56 |
| Death of a well-known hippo at a zoo in Cincinnati | 1,265 | 665 | 34 |
| Online banking outages at banks in the United States | 1,091 | 620 | 39 |
| A classified US government network down for days | 1,084 | 688 | 45 |
| Capture of a cartel leader in Mexico | 1,061 | 692 | 48 |
| Poland investigates a missile entering its airspace | 967 | 844 | 47 |
These are events as outlets published them. I did not verify them elsewhere.
And the small stories that travel as one text:
| Story | Outlets | Distinct voices |
|---|---|---|
| A report that younger managers will soon outnumber older ones | 613 | 13 |
| Parents complain that school photos look like police photos | 578 | 5 |
| A local election in Florida decided by a coin toss | 563 | 13 |
| Six women honoured on an entrepreneurship day | 403 | 5 |
The school-photo story reached 578 outlets with five headlines between them. It is one text, published the same way by hundreds of local sites.
Protests and strikes: what was reported #
Counts of events reported in the news and coded by the GDELT project, for 15 to 21 November against the four weeks before. They say what was published, not what happened.
- Reported protests were about a fifth lower: around 710 a day worldwide, against about 880. All reported events together were slightly up, at 104% of the earlier level.
- The fall is still the end of October’s waves: France at 25 a day against 81, India 63 against 119, the United States 162 against 204.
- Two countries moved the other way. The Philippines had 41 reported protests a day against a usual 5, and Bangladesh 25 against 10.
- Reported strikes were flat, 48 a day against 46. In the United States they rose from 13 a day to 21.
No forecast is printed here. The counts are the record.
What the system forecasts next #
The rule is unchanged. A forecast is of a figure that will be published. It is shown only where the model beat a simple guess on days it had never seen.
Electricity, Monday 23 November. Issued Sunday at 11:15 Brussels time. Day average and range, EUR/MWh:
| Zone | Forecast | Range (P10 to P90) |
|---|---|---|
| Romania | 228 | 172 to 293 |
| Hungary | 235 | 183 to 294 |
| Bulgaria | 208 | 145 to 277 |
| Austria | 229 | 181 to 280 |
| Belgium | 216 | 169 to 263 |
| Czechia | 231 | 193 to 294 |
| Germany and Luxembourg | 216 | 167 to 296 |
| France | 171 | 58 to 229 |
| Netherlands | 204 | 159 to 281 |
| Poland | 177 | 78 to 271 |
| Slovakia | 239 | 182 to 307 |
The eight zones below Bulgaria are new. They passed the same test, so they are shown.
Gas storage, percent full, for the six countries where the model has skill. Denmark is new:
| Country | 28 November | 5 December |
|---|---|---|
| Romania | 81.0 (79.5 to 81.7) | 78.1 (76.1 to 79.6) |
| Czechia | 65.9 (65.2 to 67.3) | 62.0 (61.0 to 64.9) |
| Netherlands | 56.1 (55.0 to 58.6) | 52.8 (49.9 to 56.2) |
| Spain | 79.3 (78.7 to 79.8) | 78.2 (76.6 to 79.2) |
| Poland | 93.9 (92.5 to 95.0) | 92.4 (90.5 to 94.1) |
| Denmark | 84.6 (82.3 to 86.9) | 82.3 (78.8 to 85.9) |
Every other topic: what moved from “not yet” #
| Topic | Last week | Now |
|---|---|---|
| Electricity prices in eight more zones | In first test runs | Shown. All eight passed |
| Electricity demand | In development | Shown in 25 of 32 zones |
| Reported protests, strikes, armed clashes and coercion, per country, 7 days ahead | No skill yet | Shown where the test is passed: protests in 27 of 33 countries, strikes in 13 of 16, clashes in 97 of 122, coercion in 104 of 113 |
| Gas storage in more countries | No skill in 12 | Skill in 6 of 17; no skill in 11 |
| Wind and solar output | In development | First tests |
| Prices per quarter-hour | Not yet. The market publishes per quarter-hour; the forecast is hourly | |
| How long a story stays in the news | Not built | Not built |
| Elections, markets, named people or outlets | Never | Never |
For protests, strikes, clashes and coercion, the target is the number of reported events in a country, from coded news data. It is not a risk score for a country, and not a claim that something will happen. Those forecasts are on the site with their track record. I print none here until they have a score of their own to show.
What to take from week two #
- The ranges held about as often as promised: 81 to 95% of hours, against the 80% they claim. The middle of the forecast was off by 24 to 37 EUR/MWh per hour.
- The worst failure was not a wrong number. It was three days with no forecast, because a figure the model depends on was never published. The honest answer on such a day is “not issued”, and for a few hours one page did not give it.
- Counting changed more than the news did. A fifth fewer stories, and sixty percent more of them shared by several outlets, came from fixing how articles are grouped.
Next Sunday: the score for this table.