Where the Risk Lives — Part 4: Going Live

Part 3 finished with a deliberate gap.

“The buckets are encrypted, public access is blocked and the processed bucket only allows CORS from the Data & Grit site. Publishing the processed object for the frontend is deliberately separate work; I did not weaken the storage policy just to make a demo easier.”

Basically, I had built the pipeline and produced the data, but nobody outside AWS could actually see it.

That was fine at that stage. I didn’t want to start knocking holes through the security just because I wanted a pretty map for the website.

This part is about sorting that final bit out.

The processed data is now public, and more importantly, there is finally a map to look at.

energy-asset-exposure-map — offshore wind / oil & gas / power vs flood, earthquake, wildfire & storm hazard

// total assets —

// earthquake exposed —

// flood exposed —

// wildfire exposed —

// severe storm history —

hazard-exposed not exposed (this filter)

$ fetching asset/hazard data

// "power" (OpenStreetMap Overpass substations/generators) is at 0 assets — the public Overpass API was down for the entire build session, and the pipeline correctly reports this as a per-source failure rather than faking data. Everything else — onshore wind, solar, battery, bioenergy, hydro, nuclear — comes from DESNZ's Renewable Energy Planning Database and a hand-curated nuclear station list. See the write-up below.

Making some of it public, without making all of it public

The processed bucket started completely locked down.

BlockPublicAcls, BlockPublicPolicy, IgnorePublicAcls and RestrictPublicBuckets were all set to true.

That was the right place to start. It is much easier to open up exactly what you need later than discover you accidentally published the lot.

To let the website read the map data, I only needed to relax two settings:

BlockPublicPolicy
RestrictPublicBuckets

I then added a bucket policy allowing public s3:GetObject access, but only for objects under the processed/* prefix.

The raw bucket is still completely locked down.

There is no reason for anything in raw/ to be public. The website doesn’t need it, the person looking at the map doesn’t need it, and therefore it stays private.

CORS had already been configured for dataandgrit.energy in the CloudFormation template from Part 3. So once the bucket policy was in place, the frontend could fetch the data directly from S3.

No API was needed. No second version of the data. Just the output already being produced by the pipeline.

What the map showed me that the JSON didn’t

Every asset is displayed using a centre point.

That matches the geometry_centroid() logic used by the join Lambda. I didn’t want the pipeline deciding an asset was in one place and then the frontend making up its own answer.

One definition of location is enough.

Once I put the 395 real assets onto an actual map, something became obvious that wasn’t particularly obvious from looking at the JSON.

Flood exposure was sitting at 0% across the entire dataset.

At first glance, that looks like something has gone wrong.

It hasn’t.

The assets currently in the dataset are offshore or coastal infrastructure, while the flood-zone layer covers inland flooding. The spatial join is doing exactly what I told it to do; the two datasets just don’t meaningfully overlap yet.

That’s really a coverage gap, not a join bug: there’s no onshore energy infrastructure in this dataset at all yet - no nuclear stations, no solar or onshore wind farms, no battery storage, no refineries. Adding that properly means sourcing from an authoritative register (the DESNZ Renewable Energy Planning Database, or National Grid ESO’s Embedded Capacity Register for the smaller sites), not typing out a list by hand and hoping it’s still accurate. Flood exposure will only mean something once assets that are actually inland are in the dataset to test it against.

So the result is technically correct, but not especially interesting.

That is still a useful result.

A JSON file can happily report zero all day without making you question it. Put the same result on a map and you immediately start asking why.

That is one of the reasons I wanted the map in the first place. Not just to make the project look nicer, but to expose where the data is thin, missing or simply not useful yet.

The OpenStreetMap power-infrastructure gap is still there as well. The public Overpass API was unavailable throughout the build, so the pipeline recorded that source as failed instead of pretending everything worked and returning made-up data.

I would rather show a gap than hide one.

So that is where the project currently stands: a working AWS pipeline, real British energy assets, real hazard data and a live map that makes both the useful results and the awkward gaps much easier to see.

It isn’t finished because the data will never really be finished.

But it is live.

Gareth Winterman