Cloud data platforms: the architecture, the migrations, and the systems that keep them running.
We design the platform and we operate it. Agents are the primary interface. The sections below are production systems: the shape of the work, and how it runs.
The boundaries come before the jobs. Ingest takes credentials and a destination. Account metadata stays in its own service and is called only when the extract needs it. Records that are too large to join in the database are hashed into buckets in object storage, so one bucket can fail alone. The spec fixes the storage layout, the trigger, and what is allowed to query the result. The pipeline is built to that shape.
The older warehouse was Redshift. Production reporting now sits on Snowflake. The classic adapter path is what grew up loading that earlier warehouse.
A later move took a high-volume CDN log feed off New Relic. The files were already in object storage, and a function pushed them into New Relic, which bills on volume. Snowflake loads them when the object is created. Queries cover the old dashboard. The function is gone, so the remaining cloud charge on that path is storage, and the monthly cost fell to a small fraction of the New Relic bill.
A large feed runs as Python on Airflow. Loads start when the data arrives. The orchestrator emits traces, and the jobs record duration, memory, and counts.
Pull requests and the main branch run on GitHub Actions. The builds run on CodeBuild. A change publishes the stacks it touches.
Pipeline runs and delay rules feed a metrics stack: exporters, alert rules, and a check that a job still has an alert. Traces, metrics, and logs go through an OpenTelemetry collector into a columnar store.
An agent takes the first pass. It gathers that evidence and investigates before a person is told. When the fix is one the agent is allowed to make, it handles the problem, and the notice says what happened and that it is already taken care of. When a person has to look, the notice carries the related evidence and the first checks, in the form of a playbook.
One path ingests metadata, performance reports, and large datasets, including feeds of hundreds of millions of records. A run starts when the data arrives, or it runs on a clock. An adapter knows one API. Format, compression, and destination are configuration.
Each process streams. A Python generator yields records, and the next step consumes that stream, so the process does not hold the feed in memory.
The orchestrator divides the work with a dynamic expand. Each piece is its own task, and a failed piece retries alone. A reduce then consolidates the pieces: they are sorted and stream-merged, and the join happens in object storage, outside the database. The database receives the load.
Finding accounts is its own catalog. A manager account is walked for its children: a Google MCC, a Facebook Business Manager, a TikTok business center, and others of that shape. Ingest receives credentials for the accounts it should pull. The catalog stays beside the extract.
Some datasets already live in a warehouse. Pulling them account by account through an adapter is the slow path. Those exports now leave on the warehouse's own transfer and land as files in object storage, which is the handoff into the warehouse load. The performance gain on that path is large.
The cluster, the observability stack, and the hosts for the assistants are infrastructure as code. Dev and prod are separate stacks. A stack changes only when its files change.
We design the system for an agent first and a person second. The interface an agent calls is the one we finish first. A screen is there so a person can look, or check a result.
The trace store is built that way. OpenTelemetry lands in a columnar store. The API and the MCP in front of it are what an agent calls. The UI is for poking around.
Written knowledge follows the same rule. The pages stay up, and a person can still read them. The fast check is an agent over those pages, the tickets, and the code. Each tool is limited to the caller's role.
Account work follows it too. A form can grant and manage access. An agent that does the same job is faster, and a person confirms the writes.
A warehouse MCP lets a model list records, measure a combination, and find related ones. It is read-only, authenticated, and capped on concurrent queries.
Modeled SQL and a semantic layer, when a report needs them. The models are SQL templates over tables the pipeline already lands.