Part 1 ended with a question and no data.
As North Sea storms become more severe, do offshore outages become longer or more frequent?
On paper, this sounds almost insultingly straightforward.
Find the storms. Find the outages. Join the two together. Job done.
Except public data has a habit of smiling politely while making that considerably harder than it sounds.
Before I write a single Lambda, I want to unpick the request as an architect.
What are we actually being asked to measure? What would I need to answer it properly? What can I genuinely get hold of? And, probably the most important question of the lot: what can that data actually prove?
That last bit matters more than the technology.
I’m not collecting datasets because their names sound vaguely stormy or outage-ish. I need something I can actually access, understand, join to something else later and defend if somebody asks where the number came from.
That ruled out rather more than I expected.
Storm Severity: The List Isn’t the Data
The Met Office, Met Éireann and KNMI jointly name storms that could cause medium or high impacts.
That gives me a nice, human-readable list of storm names and dates going back to 2015.
What it doesn’t give me is much I can actually calculate with.
There isn’t a bulk API where I can ask for:
- Storm Babet’s minimum pressure
- maximum wind speed
- significant wave height
- storm track
- conditions over individual offshore assets
The named-storm list is useful, but it is basically a collection of announcements.
It tells me when to look.
It doesn’t tell me what happened.
So I am treating the named-storm list as an event index, not as the source of storm severity data.
ERA5 does the actual measuring
ERA5, provided through the Copernicus Climate Data Store, is hourly global reanalysis data going back to 1940.
It works on a grid of roughly 31km and includes wind speed, mean sea-level pressure and significant wave height, along with hundreds of other weather variables.
It is free, although registration is required. You accept the licence for each dataset, create a personal access token and query it using the cdsapi Python client.
The results come back as NetCDF or GRIB files.
That is much more my sort of thing: actual numbers, attached to actual coordinates and timestamps, rather than a sentence in a news article telling me it was windy.
That difference matters here.
I can ask ERA5 what the wind speed and pressure were over Dogger Bank at 3am during Storm Babet and get a number back.
I can’t ask that of a Met Office press release.
Two other sources I looked at
I also looked at two perfectly good weather sources which didn’t quite fit as the primary dataset.
Met Office Weather DataHub
This is a proper API and would make plenty of sense for current observations, forecasts or perhaps a live version of the project later.
The problem is history.
Its historical land-observation data covers a recent rolling window rather than giving me the five-to-ten-year backfill I need now.
Perfectly good source. Just the wrong shape for this particular job.
CEFAS WaveNet
WaveNet collects data from real Datawell Waverider buoys around the UK coast, including sites in the southern North Sea near the Lincolnshire offshore wind cluster.
Historic CSV files can be downloaded for free under the Open Government Licence, although registration is required.
This is genuinely useful because it is measured wave height from a physical buoy rather than a modelled estimate.
The problem is coverage.
A buoy measures one point. ERA5 gives me a queryable grid covering the whole North Sea.
That means WaveNet is useful for checking ERA5 at locations where a buoy happens to be nearby, but it can’t sensibly become the main storm dataset.
Verdict: the named-storm list tells the pipeline where to look. ERA5 measures what happened. WaveNet gives me something physical to spot-check those measurements against.
Outages: Nobody Publishes a Field Called “Storm Damage”
This is where I have to stop myself getting carried away with the nice version of the project and deal with what the data really gives me.
There is no public dataset containing a helpful field that says:
Asset X was unavailable because Storm Y broke it.
Operators and insurers may have that information.
I don’t.
What exists publicly is a collection of signals sitting around the thing I actually want to measure. Each one tells me something useful, but each one also has a fairly important blind spot.
BMRS and REMIT
BMRS/Elexon publishes REMIT market messages covering planned and unplanned generation unavailability.
It also includes changes to the actual availability of parts of the offshore electricity network.
The API is public, returns JSON and doesn’t require a paid subscription.
This is probably the closest public source I can get to dated, asset-level evidence that something stopped working.
There is, of course, a catch.
The units are identified using BMRS codes rather than nice, friendly asset names, and the message doesn’t necessarily say why the unit became unavailable.
An unavailable unit could mean storm damage.
It could also mean maintenance, a grid fault, an unrelated mechanical failure or an outage that simply happened to overlap with some bad weather.
REMIT tells me that something changed.
It doesn’t prove the storm caused it.
NESO curtailment data
The NESO Data Portal publishes wind curtailment and constraint data, including downloadable CSV files showing how much wind generation was turned down and why.
At first glance, this looks like an obvious outage number.
It isn’t. And if I quietly treated it as one, I could make myself a very convincing-looking chart that was also rather misleading.
Curtailment during a storm is often caused by there being lots of wind generation and not enough grid capacity to move it.
That isn’t equipment failure. It can be a perfectly functional wind farm being asked to generate less.
If I used curtailment as the headline outage measure, wind speed would be driving both sides of my own correlation. I would, in effect, have built an expensive way of proving that windy weather contains quite a lot of wind.
So curtailment stays in the project, but as a labelled confounder and a piece of context. It is not the dependent outage variable.
HSE hydrocarbon releases
The HSE Hydrocarbon Releases system contains individual incident records for offshore oil and gas platforms going back to October 1992.
It is more useful than the annual RIDDOR summaries because it gives me incident-level records rather than a yearly total.
This is probably the strongest public safety-incident signal available for individual platforms.
Its limitation is fairly obvious: it covers oil and gas, not offshore wind.
That still makes it useful. It just means I need to be very clear about which part of the offshore estate it describes.
NSTA open data
The NSTA open-data collection is genuinely open, well-presented and full of useful information about production, field boundaries, licences and offshore infrastructure.
What it doesn’t contain is an incident feed.
I can use it later to confirm whether a platform was active during a particular storm or add useful context around production.
I can’t use it as the outage source.
Ofgem REGO data
Ofgem’s Renewable Energy Guarantees of Origin data is annual and certificate-based.
That is far too coarse to line up with a storm lasting a few days.
Ruled out.
Verdict: REMIT becomes the main outage proxy for offshore wind and relevant offshore infrastructure. HSE hydrocarbon releases provide a separate incident signal for oil and gas platforms. NESO curtailment comes along as context, but stays clearly labelled as something influenced directly by wind conditions.
What This Means for the Architecture
This is exactly why I wanted to unpick the question before building anything: the data has changed the shape of the architecture.
The honest version of this project was never going to prove that storms cause outages.
The public data simply can’t take me that far.
REMIT doesn’t reliably record the cause of an outage, while curtailment is tangled up with wind speed by definition.
So I need to build an architecture that tests for correlation without dressing it up as causation.
For me, this is the architecture bit.
It isn’t just choosing AWS services and drawing increasingly confident boxes with arrows between them. It is working backwards from the question, finding the limits in the data, and then designing something that doesn’t promise an answer the evidence can’t support.
The design, as deployed
The pipeline runs on AWS in one CloudFormation stack. It has two Lambda functions sharing one dependency layer, two private and public S3 buckets, and two daily schedules.
Ingest (06:00 UTC). Pulls the Elexon BMU reference, the current and previous NESO BOA curtailment seasons, and the last three REMIT months. Each raw response is written unchanged to a private data bucket, so the source is always recoverable. Each REMIT month is then compacted at ingest time to a small events file. That turns roughly 930 MB of raw REMIT JSON into a few megabytes of CSV, so the daily job never needs the full history in memory.
Publish (07:00 UTC). Reads the compacted events and the BOA seasons, deduplicates REMIT revisions, builds the daily table, runs the storm-window tests, and writes results.json, daily.json and storms.json to a public bucket. The site reads that bucket directly, the same way the oil price chart reads its data.
Every step can be rerun safely. Raw files are overwritten only for the months being refreshed, and the publish step rebuilds everything from the compacted files.
Storms are a manual input
The storm catalogue is reference data, not a live feed. Nothing in the pipeline finds new storms. Updating the catalogue means uploading a new storms.json, and the next publish picks it up.
Outages join to storms on time
REMIT identifies units by BMU code, and the reference list doesn’t give me a site location I can trust. So outage records join to storms by date window, not by coordinates. A storm is a national event in the catalogue, so the join asks whether an outage started in the window around the storm, not whether the asset was in its path.
Would I prefer a proper location-aware join against the storm track? Absolutely.
Do I currently have trustworthy public location data to support it? No.
So this is the weaker join, but it is the honest one.
What the design can’t do yet
ERA5 storm severity needs a Copernicus account, so storm strength still comes from a national peak-gust figure. Point-level exposure would need site locations I haven’t verified. The HSE incident data isn’t in the pipeline at all.
The awkward data stays visible
The curtailment data doesn’t disappear just because it complicates the story.
It is wind-only curtailment from NESO’s balancing data, and it stays in the model as a labelled control: a possible consequence of high wind and grid conditions, not proof of physical damage.
That matters. If an underwriter, engineer or suspicious reader looks at the results, they should be able to see that curtailment was considered and that I haven’t quietly mixed it into the outage count to make the relationship look stronger.
A Result I Can Stand Behind
And after all that, the final correlation may still turn out to be weak. That is allowed.
Storms may line up with more offshore outages. They may line up with longer ones. Or the public outage data may simply contain too much noise for a clear pattern to emerge.
Any of those outcomes is acceptable.
The point isn’t to force the chart into saying something dramatic.
The point is to build a pipeline using public, traceable data and be able to explain exactly what each number measures, what it doesn’t measure and where it came from.
If somebody challenges the result, I want to be able to follow it all the way back through the pipeline to the original ERA5 value, REMIT message or HSE incident record.
That is the standard I am aiming for.
Not the most exciting answer.
Not necessarily the answer I expected when I started.
But an answer I can actually defend.
Gareth Winterman