Where the Risk Lives — Part 2: Why I Chose This Architecture

One thing I’ve learned over the years is that there are hundreds of ways to build almost any data engineering project.

Sometimes that’s exciting.

Sometimes it’s paralysing.

Every cloud provider has dozens of services.

Every engineer has a favourite architecture.

Every blog post tells you their way is the “modern” way.

So before writing a single line of code, I wanted to answer a different question.

What is the simplest architecture that solves this problem well?

It Doesn’t Need To Be Clever

When people hear “geospatial pipeline”, it’s easy to imagine enormous GIS systems, Kubernetes clusters and distributed processing engines.

They’re brilliant tools.

I don’t need them.

My entire dataset is measured in megabytes rather than terabytes.

The public data changes relatively slowly.

The joins are spatial rather than computationally expensive.

This isn’t Google Maps.

It’s a portfolio exposure map.

Building a huge platform would simply be engineering for the sake of engineering.

Static Data Is Your Friend

Unlike the oil price project, nothing here changes every few minutes.

An offshore wind farm doesn’t move.

Neither does an oil platform.

Flood maps are updated occasionally.

Storm archives arrive after the event.

OpenStreetMap evolves over time rather than every second.

That changes the design completely.

Instead of streaming data…

…I can simply take regular snapshots.

Each ingestion creates a dated copy that lives forever.

If something goes wrong six months from now, I can reproduce exactly what the pipeline saw on that day.

Storage is cheap.

Historical data is valuable.

Why Lambda?

I like Lambda for projects like this because it naturally encourages small pieces of software.

Rather than one enormous ETL job, every dataset gets its own ingestion function.

If the Crown Estate changes their data format…

…only that Lambda needs updating.

If OpenStreetMap is unavailable…

…everything else continues to work.

Each component has one responsibility.

Download.

Clean.

Store.

Done.

Small functions are easier to understand, easier to test and much easier to replace later.

Why S3?

Object storage feels like the natural home for public datasets.

Every raw download is stored exactly as it arrived.

Nothing is overwritten.

Nothing disappears.

Later stages of the pipeline can always go back to the original source rather than trusting an intermediate transformation.

That also makes debugging far easier.

If something looks wrong on the map, I can trace it all the way back to the original file that produced it.

Why GeoJSON?

The finished product is an interactive map.

GeoJSON already understands geographical features.

Almost every mapping library can read it directly.

That means the pipeline doesn’t have to perform any clever transformations for the frontend.

It simply produces one file.

The website loads it.

The browser draws the map.

Simple.

Why No Database?

This might be the decision that surprises people most.

There isn’t one.

At least… not yet.

For a relatively small dataset, introducing PostgreSQL with PostGIS would certainly work.

So would DynamoDB.

So would Aurora.

But they also introduce operational overhead.

Backups.

Maintenance.

Networking.

Authentication.

Monitoring.

For a static dataset containing only a few thousand assets, a single GeoJSON file is perfectly capable of doing the job.

Sometimes the best database…

…isn’t having one.

Why No API?

The website simply reads the latest processed file directly from S3.

No API Gateway.

No web server.

No additional Lambda.

No REST endpoints.

The browser asks for one file.

S3 returns one file.

That’s all that’s required.

Reducing moving parts generally makes systems faster, cheaper and easier to maintain.

Why CloudFormation?

Infrastructure is part of the project.

I didn’t want a tutorial that says:

“Click this button… then this one… then this one…”

I wanted everything reproducible.

If someone cloned this repository tomorrow, they should be able to create exactly the same infrastructure I did.

That’s what Infrastructure as Code gives you.

Version control.

Repeatability.

Confidence.

Why This Shape?

If you’ve looked at my previous Global Shock project, you’ll notice this architecture looks very familiar.

That’s deliberate.

I’m trying to build a library of reusable patterns rather than isolated projects.

Once you know how one pipeline works, the next one should feel familiar.

Different data.

Different business problem.

Same engineering principles.

There’s something satisfying about solving completely different problems using the same dependable building blocks.

Could This Grow?

Absolutely.

If the datasets became significantly larger, I’d probably introduce PostGIS for spatial indexing.

If updates became more frequent, event-driven processing would make sense.

If multiple consumers needed the data, an API would become useful.

If analytical workloads appeared, Athena or Databricks would be strong candidates.

But adding those today would be premature.

Good architecture isn’t about using the most services.

It’s about using the fewest services necessary to solve today’s problem while leaving tomorrow’s options open.

For this project, I think that’s exactly where this architecture lands.

Gareth Winterman