Building eolas: a data platform for New Zealand official statistics

New Zealand publishes a lot of good official data. Getting hold of it is the tedious part.

Stats NZ has one portal, the Reserve Bank has another, LINZ has a third. MBIE, MSD, Police and NZTA each do their own thing. Councils publish through ArcGIS services with wildly varying conventions. Some sources offer a clean API. Others offer a CSV behind a form, or an Excel file with merged header cells and a footnote in row 4. The OECD has a proper SDMX API, which is a relief until you read the specification.

None of this is unusual, and none of it is anyone’s fault. Agencies publish for their own purposes and their own audiences. But the practical result is that a good chunk of any analysis project gets spent on retrieval and cleaning before the actual work starts. I have done that work often enough, in consulting and in government, that I got tired of doing it again.

eolas is my attempt to do it once. It is a platform that pulls together over 700 datasets from New Zealand, Australian and OECD sources, cleans them to a consistent shape, and serves them through a single typed API with Python and R clients and a CLI. This post covers how it is built and which decisions turned out to matter.

Three planes

It helps to think of the system in three parts: getting data in, storing it, and serving it out.

Getting data in. Each source has its own tap, written to the Singer specification. Singer defines a simple contract between a tap (which extracts) and a target (which loads), and the useful property is that it makes every source a small, independently testable unit. A tap for RBNZ interest rates has nothing to do with a tap for LINZ parcels, and neither can break the other. Councils are the exception to the one-tap-per-source rule: enough of them publish through ArcGIS that a single generic base tap covers most, parameterised per council.

Meltano orchestrates the taps. They run as scheduled ECS Fargate tasks, triggered by EventBridge, with one task definition per pipeline grouped by refresh cadence. Daily sources run daily; quarterly ones do not.

Storing it. Everything lands in Apache Iceberg tables on S3, catalogued in AWS Glue. A custom loader, target-iceberg, handles the write.

Serving it out. Two FastAPI applications share one codebase. One serves the marketing site, dashboard, authentication and billing. The other serves the data API. They run on ECS Fargate behind an ALB, with Cloudflare in front, and a Postgres database on RDS holds customer and account data.

The decisions that mattered

Most architectural choices are reversible and not worth much discussion. A few were not.

An open table format

Choosing Iceberg on object storage rather than loading everything into a warehouse was the call that shaped everything downstream. It means the same physical data can be read three ways without copying it: through the eolas API, through a Snowflake zero-copy share, and directly by anything that speaks Iceberg, such as dbt, Athena or pyiceberg.

The Snowflake share is the clearest illustration. Customers on Snowflake query our S3 and Glue directly. There is no nightly sync job, no second copy drifting out of date, and no Snowflake warehouse of our own to pay for. Column documentation and table descriptions are materialised into the Glue catalog, so someone browsing the share sees the same clean names and definitions as someone reading the API docs.

The cost is that Iceberg is a lower level of abstraction than a managed warehouse. You think about compaction, snapshot expiry and manifest files. For a platform whose product is the data itself, that seemed like the right trade.

Keeping history

Official statistics get restated. A quarterly figure published in March may be revised in June, and a dataset that looks static will quietly change underneath you. If you overwrite on each load, you lose the ability to answer what a series looked like at a point in time, which is exactly the question an analyst reproducing a published result needs to ask.

So target-iceberg keeps slowly changing dimension history rather than overwriting. Deletes expire a row, additions append a new one, and the full history stays queryable.

This has been the single largest source of subtle bugs in the project. The failure mode is nasty because it is quiet: a bookmark handled wrongly, or a merge that takes the wrong branch, does not raise an error. It just writes less data than it should, and you find out weeks later when someone asks why a series has a hole in it. The fix was not cleverness in the merge logic but canaries around it. Loads now check for unexpected shrinkage and compare against source fidelity, and fail loudly when the numbers do not line up.

If I were starting again, I would write those checks first.

Keeping batch away from serving

Early on, extraction ran on the same box that served the API. This works until a large backfill runs, at which point the box becomes unresponsive and the API goes with it. Moving batch onto Fargate fixed it permanently, and had the side benefit of making pipeline scheduling a first-class thing rather than a crontab.

A related lesson, less architectural and more practical: the ETL image builds in CI and pushes to ECR. Building and pushing locally wedged the Docker daemon often enough that I made it a rule.

Outgrowing SQLite

Customer and account data started in SQLite on the application box. This was fine for a single instance and became the binding constraint on everything else. One writer, one host, backups that amount to copying a file, and foreign keys that are more suggestion than guarantee. It also made horizontal scaling impossible, because two application instances cannot share a local file.

Moving to Postgres on RDS was the enabler for running more than one instance, which in turn was the enabler for high availability. The migration itself was unremarkable, done with SQLAlchemy Core and Alembic, with a reversible cutover gated on smoke tests. The lesson is not about the mechanics. It is that a storage choice made for convenience early on quietly set a ceiling that had to be removed before anything else could improve.

Where it is now

The platform runs across multiple availability zones, with rolling deploys gated on a readiness check, and a transform layer built on dbt against Athena producing derived marts as Iceberg tables. The first of those joins parcels and titles with council hazard and zoning data, which is the sort of question that is straightforward to ask and genuinely tedious to assemble from source.

There is a reasonable amount still to do. More sources, better coverage of the ones already there, and continued work on the contract tests that keep the whole thing honest.