Where the Risk Lives — Part 3: Building an Energy Asset Exposure Pipeline on AWS

Where Are Britain’s Energy Assets Exposed?

An energy asset is not risky simply because it is expensive.

It is risky because it exists in a place.

Offshore wind farms sit in the path of North Sea storms. Oil and gas platforms operate miles from the coast. Substations, generators and other power infrastructure can be surrounded by flood water long before anything is technically wrong with the asset itself.

The useful question is not just where are Britain’s energy assets?

It is:

What are they exposed to, and how close is the hazard?

That is what I built this pipeline to answer.

It collects public data on British energy infrastructure, brings in flood, earthquake and storm data, normalises the sources into a common shape, and joins each asset to its nearest hazards.

The result is a versioned exposure dataset that can feed an interactive map, risk analysis or an underwriting workflow.

The map is the visible bit.

The interesting bit, for me, is the data engineering underneath it.

What I Built

The pipeline runs in AWS and follows a simple event-driven pattern:

1. EventBridge triggers the ingestion Lambdas once a month
2. one Lambda collects energy assets from several public sources
3. another collects natural-hazard data
4. every source writes a dated raw snapshot into S3
5. an S3 ObjectCreated event triggers the join Lambda
6. the join finds the latest available snapshot from each source
7. each asset is enriched with its nearest flood and earthquake hazard
8. the pipeline writes a dated output and replaces processed/latest.json

The first deployed run produced 395 real energy assets:

• 72 Crown Estate offshore wind leases for England, Wales and Northern Ireland
• 58 Crown Estate Scotland leases
• 265 active NSTA oil and gas platforms

Each is published with flood, earthquake and storm exposure information.

This is not a mock architecture or a local notebook. The stack is deployed in eu-west-2 and has been run end to end against the real sources.

Why I Used Raw and Processed S3 Layers

I did not want the frontend calling five public APIs and hoping they were all healthy.

Public data sources have different schemas, different availability and different ideas about pagination. They also change independently. Tying the website directly to them would make somebody else’s outage my outage.

Instead, each ingestion writes its own dated snapshot:

raw/<source>/YYYY/MM/DD/snapshot.json

The join then publishes two outputs:

processed/YYYY/MM/DD/exposure.json
processed/latest.json

The dated files give me traceability. I can see what the pipeline knew on a particular run and compare outputs over time.

latest.json is the stable data product. The eventual frontend only needs one predictable object and does not need to understand how the source systems work.

The raw bucket also has a 90-day lifecycle policy. Raw snapshots are useful, but keeping every intermediate file forever would be storage by accident rather than design.

The Infrastructure as Code

One CloudFormation stack creates the raw and processed buckets, IAM role, three Lambda functions, monthly EventBridge schedule and the S3 notification that starts the join.

The buckets are encrypted, public access is blocked and the processed bucket only allows CORS from the Data & Grit site. Publishing the processed object for the frontend is deliberately separate work; I did not weaken the storage policy just to make a demo easier.

    Loading…
  

There was a genuine CloudFormation problem hiding in this design.

The raw bucket notification needed the join Lambda ARN. The Lambda needed its IAM role. The role’s S3 policy referenced the bucket. That closed the dependency loop.

The fix was to use deterministic !Sub bucket names and ARN strings in the IAM policy instead of !Ref and !GetAtt on every edge. Because the bucket names are deterministic and include the AWS account ID, I could break the CloudFormation dependency without broadening the permissions.

That issue was caught by aws cloudformation validate-template, which is precisely why infrastructure should be validated rather than admired as YAML.

Ingesting Assets From Several Different Sources

The asset Lambda brings together:

• Crown Estate offshore wind leases for England, Wales and Northern Ireland
• Crown Estate Scotland offshore wind leases
• active NSTA oil and gas installations
• OpenStreetMap power infrastructure through Overpass

The two Crown Estate datasets and the NSTA data are available through ArcGIS FeatureServer endpoints. They are all geospatial datasets, but that does not make them consistent.

Names live in different fields. Identifiers mean different things. Geometry can be a point or a polygon. NSTA includes inactive infrastructure that I do not want in the current exposure view.

The ingestion layer turns them into one common asset contract containing:

• a source-qualified ID
• source
• asset type
• name
• latitude and longitude
• original geometry
• original source properties

I kept the raw properties as well as the normalised fields. Normalisation makes the join possible; retaining the source detail means I have not thrown useful context away to achieve it.

    Loading…
  

I considered the US Energy Information Administration during source research, but dropped it. It is a good source for US energy data; it is not the right source for locating British energy assets.

That sounds obvious written down. It is still an important data-engineering decision: source authority and geographic coverage matter more than whether an API is familiar.

Ingesting Hazards Without Pretending the Data Is Cleaner Than It Is

The hazard Lambda combines three quite different things:

• Environment Agency flood zones from a WFS service
• modern and historical BGS earthquake catalogues
• a hand-curated catalogue of significant UK and North Sea storms

Flood zones are polygons. Earthquakes are points with magnitudes. Storms are events, but there is no single clean public API giving me consistent historical storm-track geometry for this use case.

I chose not to invent precision that the source data does not contain.

The storm catalogue is therefore maintained as a small, explicit dataset and bundled with the deployment. It is treated as national-scale context rather than being assigned a made-up distance from every asset.

    Loading…
  

This Lambda exposed one of the more useful lessons in the project.

The Environment Agency pagination completed in roughly 90 seconds from a normal network, but hung until Lambda killed the invocation in AWS.

urllib’s timeout is an inactivity timeout. It is not a maximum wall-clock duration. A remote server that keeps trickling bytes can remain technically active while consuming the whole Lambda runtime.

I added a hard SIGALRM around individual requests and an overall time budget around pagination. If the source is slow, the ingestion now returns clearly identified partial data instead of blocking the entire pipeline.

That is a better failure mode. A bounded, observable partial result is more useful than no result and a timeout message.

Joining Assets to Hazards

Once a new raw snapshot lands, S3 invokes the join Lambda.

For each source, it finds the latest available snapshot. Asset geometries are reduced to representative centroids, then the join uses the haversine formula to calculate great-circle distance to the nearest flood-zone centroid and earthquake epicentre.

The exposure rules are explicit:

• flood exposure: nearest flood-zone centroid within 10 km
• earthquake exposure: nearest recorded epicentre within 50 km
• storms: national catalogue summary attached as wider risk context

The output is GeoJSON, which means the processed data can move directly into a mapping frontend without another transformation layer.

    Loading…
  

The geospatial functions live in a small shared module rather than being buried inside the Lambda handler.

    Loading…
  

This first version uses polygon centroids, not true point-in-polygon or boundary-distance calculations. That is a known modelling limitation.

The important design choice is that the limitation is isolated. geometry_centroid() and nearest_feature() are the seam where I can later introduce proper polygon intersection or a spatial index without rewriting ingestion, storage or publishing.

A Failed Source Must Not Delete Good Data

One of the easiest ways to corrupt a consolidated dataset is to confuse the source returned nothing with the source failed.

During development, the public Overpass API and its fallback mirror were overloaded. Returning zero power assets as if that were a valid observation would imply that Britain suddenly had no power infrastructure.

The pipeline records the source error instead.

More importantly, the join can seed that source from the previously published output when the latest raw snapshot is missing. A temporary upstream failure therefore does not silently remove an entire asset class from the data product.

    Loading…
  

This is a small piece of resilience, but it matters. The last known good result is often safer than confidently publishing an empty one.

It also makes the failure visible rather than hiding it. The Overpass source remains at zero in the current run and will be picked up by a later scheduled ingestion when the service recovers.

The Lambda Was Not Stuck Where I Thought It Was

After fixing the network timeout, the hazard Lambda still failed.

CloudWatch showed that approximately 26,000 flood features had been fetched quickly. Then there were no more log lines until the function timed out.

That changed the diagnosis.

The network request was no longer the problem. Serialising and uploading the large JSON payload at 512 MB was consuming the remaining runtime. Lambda allocates CPU in proportion to memory, so I increased the hazard ingestor to 2,048 MB.

The pipeline completed.

I did not simply extend the timeout again. The logs showed which phase had completed, which meant I could change the resource that was actually constrained.

That is one of the reasons I deploy these projects properly. A local run proves that the Python works. It does not reproduce Lambda’s CPU allocation, AWS network path, IAM graph or CloudFormation dependency model.

Running the Whole Pipeline Locally

The repository includes a local runner that calls every real fetch function, performs the join and writes the intermediate and final JSON files to a gitignored output directory.

It does not need AWS credentials and does not touch S3.

That gives me a quick way to inspect source schemas, test normalisation and examine the final exposure document before deploying infrastructure.

    Loading…
  

The full run calls the live services. --skip-slow avoids Overpass and the large national hazard pulls when I only need to test the pipeline shape.

What Is Still Limited

There are two limitations I have deliberately left visible.

First, the flood-zone ingestion currently caps itself at 26,000 features from a national dataset of more than 813,000. Pulling the whole country as one paginated request is not the right long-term design. The next version should divide Great Britain into regional bounding boxes, ingest the tiles independently and then de-duplicate their boundaries.

Second, centroid distance is an approximation. A large offshore lease or flood polygon can overlap an asset even when the two centroids are some distance apart. A production risk model should use real polygon intersection and distance to polygon boundaries.

Neither limitation invalidates the pipeline. They define what the current exposure flag means and give the next iteration a clear engineering target.

Why I Built It This Way

This project is about more than plotting markers.

I wanted a pipeline that demonstrates the decisions behind a usable energy-risk data product:

• choose authoritative UK sources rather than convenient but irrelevant ones
• preserve raw snapshots for traceability
• normalise inconsistent schemas without discarding source detail
• separate ingestion from joining and publishing
• use events to connect stages without creating a permanently running service
• degrade source by source instead of failing the entire dataset
• retain the last known good data when an upstream service disappears
• keep modelling assumptions explicit
• validate the system in its real cloud runtime

The output is a map-ready GeoJSON file.

The real product is a repeatable answer to a risk question: which energy assets are exposed, to what, and based on which evidence?

That is the part that can grow. The frontend can change. Better spatial methods can replace centroids. More hazards and asset classes can be added as independent sources.

The backbone does not need to be thrown away to do it.

Gareth Winterman