[Grid Flow Trade] When the price feed stopped: a postmortem

 

On 3 October the market index feed GridFlow forecasts stopped for nine hours, then fell behind six more times. Nothing false was stored, but the outage showed how the system degraded and reported. What I changed.

 

GridFlow Trade

 

When the price feed stopped: a postmortem

On 3 October 2026 the APX market index prices that Elexon publishes stopped arriving at 01:00 UTC. They came back nine hours later, and then fell behind six more times before the morning of 5 October, twice for 7 to 10 hours.

GridFlow forecasts that price for the next day, so the feed is the most important input it has. This is a blameless review of what happened, what worked, and the eight gaps it exposed.

What happened

Time (UTC, 3 Oct)	Event
01:00	Last APX price published (settlement period 4). Every other Elexon dataset kept updating
01:05 to 03:05	The hourly collector runs normally and stores the 4 prices that exist
04:20	The hourly data check raises a freshness alert: prices 7 periods behind (threshold 6)
09:10	I notice the alert and start investigating
09:14	Direct API queries show the APX rows are missing upstream. Cause classified as upstream
10:00	The next-day forecast runs with 4 of the 18 morning prices it normally uses
13:06	APX resumes from 10:00 onwards. The missing morning is back-published later

Detection took 3 hours 20 minutes from the last good price. It was about eight hours from that same point before I saw the alert.

Impact

• No wrong data. No placeholder or false prices were stored. Every period of 3 to 5 October was eventually published and collected automatically.
• One degraded forecast. The forecast for 4 October was made with 4 morning prices instead of 18. Its "D-1 morning mean" feature was the mean of four overnight prices. It still beat the naive baseline (MAE £24.50 vs £37.12).
• No rollback. The automatic rollback rule did not fire.

Root cause

Upstream. The APX market index feed into Elexon stopped. I checked from three API paths and three networks: HTTP 200 throughout, no rate limiting, other Elexon datasets current. I found no Elexon notice, so whether the provider or Elexon's ingestion failed is unknown.

A detail worth knowing if you use this data: Elexon keeps publishing zero-volume N2EX placeholder rows even when no prices exist. During the outage those rows kept arriving on time. A collector that takes every row as a price would have stored the outage as a run of £0 prices and raised no alert at all.

Why it hurt more than it should have

The data was fine. The way the system degraded and reported was not.

1. The alert was not where I look. Notices showed only on the Market Data page.
2. "Is it me or them?" was manual. It took hand-written API queries across networks to answer.
3. The forecast did not check its price inputs. It waited for the wind forecast only.
4. A feature degraded silently. The morning mean was computed over whatever periods existed: 4 instead of 18.
5. Thin days counted as evidence. A day with 4 scored periods would have counted towards the rollback rule.
6. The re-collection window was short. Only today and yesterday were re-read, so a later back-publication could have been missed.
7. Retraining did not check its data. The quality gate judged the model, not the data it learned from.
8. Alerts were counted as rows, not incidents. Every hourly update wrote a row, so the monthly report said "44 alerts" and hid that the feed failed seven times.

What I changed

Gap	Change
Alert placement	Active notices show on every page
Manual classification	The alert classifies a price gap itself: upstream feed, my collector, or API unreachable
Forecast inputs	The forecast waits for 14 of 18 morning prices. A forced run is recorded as incomplete, labelled in the product, and refreshed once the inputs arrive, only before the delivery day
Silent degradation	The morning-mean feature is missing, not a different number, when fewer than 14 periods exist
Thin evidence	Days with fewer than 40 scored periods do not count for the degradation alert or the rollback rule
Short window	Incomplete days in the last 7 days are re-collected daily
Retraining	Training-data validation is now stage 0 of retraining: schema, duplicates, calendar, recency, completeness, price range, placeholder zeros, forecast inputs
Counting	Alerts carry an incident id and a resolved time; reports count incidents with their duration

Email and chat alerting is the next item on this list, and a second price source to fall back on is the one after it.

What went well

• The collector filters zero-volume placeholder rows, so the outage showed up as a visible gap instead of as £0 prices.
• The hourly freshness check fired within its threshold.
• Late data was collected with no manual step once Elexon published it.
• Accuracy is only scored on forecasts stored before the delivery day, so the numbers above are honest.

Lessons

1. A filter that refuses bad data turns silent corruption into a visible gap. That is the right trade, but then every consumer of the data needs a stated behaviour for "missing".
2. An alert that needs a person to visit the right page is a log, not an alert.
3. Classification should ship with detection. The evidence I gathered by hand (other feeds current, placeholder rows present, the newest traded period upstream against the one in the database) was cheap to compute automatically.