Part 2 was me being fussy about the data. This part is where I actually make the thing run.
The aim is fairly simple: pull it, keep the raw source, clean it, rebuild the analysis and publish the result every day without me having to remember to prod it.
The whole lot runs on AWS in one CloudFormation stack. Two Lambda functions do the work, and the site reads the finished output straight from S3.
The shape of it
- Ingest runs daily at 06:00 UTC. It refreshes the last three months of REMIT data, the current and previous NESO curtailment seasons, and the Elexon unit reference. It writes the raw responses to a private bucket and compacts each REMIT month into a small events file.
- Publish runs daily at 07:00 UTC. It reads the compacted events, deduplicates them, builds the daily table, runs the storm-window tests and writes three JSON files to a public bucket.
- The site reads that public bucket directly. The chart updates without a site deploy.
Raw data is never rewritten by the publish step. Everything downstream of raw can be rebuilt from it.
The design in Part 2 is where I originally wanted this to end up. Once I started building it, the public data pushed the first working version somewhere different. ERA5 needs a Copernicus account I didn’t have, the HSE incident records only cover 2016 to 2021 and only oil and gas, and I didn’t have site locations I trusted enough to join on. So the version that runs keeps the same overall shape, but joins outages to storms by date window and uses the national storm catalogue.
The problem that shaped the design
The raw REMIT history is about 930 MB of JSON, one file per month.
My first local version did the wonderfully subtle thing of loading the lot into memory at once. Fine on a reasonably chunky laptop. Considerably less clever when you want the same thing to run reliably in Lambda.
The fix was to move the heavy lifting into ingest. Each month is compacted as soon as it’s downloaded. The compaction keeps only generator units, drops nothing that the later dedupe needs, and writes a CSV of a few megabytes. The publish step never sees the raw JSON.
That split became the main design decision: do the expensive, ugly bit once at ingest, then make the thing I run over the full history small and boring.
I like small and boring in production.
Dedupe has to span months
REMIT messages, just to make life interesting, get revised. The same message ID can arrive several times with different revision numbers, and a revision can land in a later month than the original.
So the compacted files keep every revision for generator units, not just the latest. Publish deduplicates across all of them, keeping the highest revision per message. Deduplicating inside each month would miss a revision that lands in the next one.
One little wrinkle I didn’t expect: when I checked the cloud version against my local run, the counts differed by 31 events out of about 92,000. The cause is the order of two filters. Locally I deduplicated before filtering to generator units. In the cloud I filter first, so a message whose newest revision names a non-generator unit is kept in the cloud and dropped locally. I’ve left it as it is and noted it, because either rule is defensible and the difference is small.
Failures that only show up on one machine
Encoding. On Windows, Python’s default text encoding is not UTF-8. The raw files contain bytes the default codec can’t read, so a local run crashed on perfectly valid data. Every read now sets UTF-8 explicitly. Exactly the sort of tiny machine-specific nonsense that can happily eat an afternoon while you question your life choices.
Placeholder timestamps. Some REMIT records carry a 0001-01-01 end time, which pandas can’t store as a nanosecond timestamp. Bad timestamps are coerced to missing values, and the rest of the record is kept.
Network. The Elexon and NESO hosts both stopped resolving from my laptop partway through the backfill, which is why the first full backfill runs inside AWS rather than locally. A pipeline that depends on a network you don’t control needs a retry, and it needs a place to run that isn’t your laptop.
Storms are a manual input
The storm catalogue is the one thing the pipeline doesn’t find for itself. Nothing scans for new storms. Each season someone adds them to storms.json and uploads the file to the data bucket. The next publish run picks it up.
That is deliberate. I could automate it, but I’d be automating something against a source that isn’t really offering itself as a feed. The Met Office publishes named storms as press releases rather than a feed, so automating the list would mean scraping a page that’s not designed to be scraped.
The code
The full project is public at audiblebutter-ltd/StormtoOutage. These are the parts that do the work, shown as they are in the repo.
The core: compaction, dedupe and the tests
stormcore.py holds everything both Lambdas share. It compacts each REMIT month, deduplicates revisions across months, builds the daily table and runs the permutation tests.
Loading…
Ingest: pull, store raw, compact
The ingest Lambda runs daily. It writes every raw response unchanged to S3, then compacts each REMIT month to a small events file.
Loading…
Publish: rebuild and write the public results
The publish Lambda reads the compacted files, rebuilds the analysis and writes the three JSON files the site reads.
Loading…
Infrastructure
Two schedules, two functions and two buckets, in one CloudFormation template.
Loading…
The bits I haven’t pretended are finished
ERA5 storm severity needs a Copernicus account, which I haven’t set up. Until then the storm strength is a national peak-gust figure, and it is weaker than the location-level measure I wanted.
The count model is ordinary least squares on counts. A Poisson or negative-binomial model would fit the data better, and it’s the next change to make.
What I’d take from this
The bits that caused the grief weren’t the clever statistical bits. They were encoding, revisions, nonsense placeholder dates and deciding where the heavy work should happen.
Which is very data engineering.
Keeping the raw data untouched and making every stage rerunnable means that when a number looks odd, I can walk backwards through the pipeline and find out why. That is far more useful to me than a pipeline that looks elegant on a diagram but becomes archaeology the moment something goes wrong.
Gareth Winterman