Zillow Runs 100 Million Notifications a Day on One Configuration Layer
Zillow turned twenty notification campaigns and ten generator services into one configuration-driven platform that runs each message as a short Step Functions workflow.
TL;DR
- AWS Step Functions Express Workflows is a managed service that runs short workflows on demand; Zillow's unbatched path starts one execution per notification, and its batch mode combines multiple messages into one execution to send more than 100 million emails and push messages a day.
- The platform has two parts: a Campaign Orchestrator that reads queued events and starts the right workflow, and a Universal Content Generator that holds the shared, stateless modules every campaign calls.
- Zillow built it after twenty-plus campaigns spread across more than ten generator services duplicated the same work and slowed every release.
- A router sits in front of the workflows and owns queue polling, batching, idempotency checks, replay, speed control and circuit breakers, so a single campaign can be paused without touching the others.
- Express mode provides the higher default start quota and the short-run latency this traffic needs, with at-least-once delivery that the idempotent modules make acceptable.
- Batching is where the cost argument lands: Zillow reports about 1.2 seconds for one message and 1.4 seconds for a batch of ten, which it puts at roughly 90% cheaper than ten separate executions.
- The measured engineering gains are an estimated 9 development weeks versus 32 for an in-house engine, and about 2 hours of maintenance a week versus 8.
Notification volume is an awkward problem for a small engineering team, because the interesting failures appear only at scale. Zillow published its answer on 11 November 2026 in the AWS Architecture Blog: one configuration-driven platform that runs every email and push as a short workflow and consolidates the workflows, data models and business rules previously carried by separate campaign services.
The shape of the workload is ordinary. A trigger arrives, several services enrich it, rules apply, content assembles, a message goes out. What makes the case study worth reading is the engineering around that shape: how the team isolated campaigns from each other, how it priced the decision between two execution modes, and what it measured after the migration.
What the Campaign Orchestrator and the Universal Content Generator do
Campaign Orchestrator is a configuration-driven platform. Each message is input data combined with persisted campaign configuration, and a shared orchestration layer reads queued campaign events, groups work where that helps, and starts the matching Step Functions Express Workflow. Campaign metadata carries product constants, ownership and environment features; workflow configuration carries step topology, per-step settings, mappings, branching, delays and send behaviour.
The Universal Content Generator is the library behind those workflows: stateless, schema-driven modules for property hydration, hidden homes, common data, recommendations, A/B testing and content assembly. Zillow calls the workflows directly through Step Functions HTTP Invoke, and a base component named content_module_call carries the resource, method, headers and connection reference that every step shares.
The credential handling in that design is worth copying. An EventBridge connection holds the API key in AWS Secrets Manager and adds it at call time, so the workflow definition and the application code never contain the key. A rotation happens once in the connection, and every workflow picks it up.
Why twenty campaign services became one shared pipeline
The old system supported more than twenty email and push campaigns from more than ten microservices. Each one owned its workflow, models, business rules and assembly logic, and each implemented the same pattern separately. Zillow describes the consequence in operational terms: duplicated engineering work, behaviour that drifted between campaigns, and overhead in releases, monitoring, logs, dashboards and on-call.
The original choice made sense when each campaign was built. Scale was the hard part then. As products multiplied, the trade-off inverted, because a new capability had to be rebuilt and retested in every service. Zillow's own summary of the fix is that shared logic moved into reusable modules and campaign decisions moved into configuration before the team evaluated workflow engines at all.
How a notification moves through the router and the workflow
The router is the part that decides when a workflow starts. It reads from Amazon SQS and owns the controls that campaigns need before any step runs: queue polling, batching, idempotency checks, pause and resume, replay coordination, speed control and routing decisions. Keeping those in one service gives Zillow a single place to pause a campaign, replay a batch or change send speed without editing a workflow definition.
Isolation is explicit in the design. Every campaign keeps its own state machine and its own SQS queue, and the router gives each queue its own polling threads, worker pool and circuit breakers. When one campaign backs up, only its breaker trips, and its queue can be paused while the rest keep running.
The configuration layer is what makes the platform self-service. A campaign owner declares which modules run, in what order, and how data moves between them; the system translates that declaration into a Step Functions definition and a CloudFormation template. Campaigns each get their own workflow built from shared modules, and the team is considering a generic workflow that assembles modules from configuration at run time.
What Express mode provides at this volume
Zillow compared Temporal, Flowable, an in-house engine and both Step Functions modes against reliability, total cost of ownership, development speed and cross-team scale. Temporal and Flowable would have added infrastructure to operate, which worked against the goal of removing release, monitoring and on-call overhead. Step Functions is serverless, so a team could pilot without provisioning anything, and Zillow estimates the service met its orchestration needs at roughly a quarter of the build effort and a quarter of the ongoing maintenance of the alternatives.
Express mode fits the traffic shape. The plan ramped from 14 million workflows a day to more than 100 million, several thousand executions per second at peak, with each workflow running for a few seconds. Express carries the higher default start quota, and its at-least-once delivery is sufficient because Zillow builds the modules around idempotent calls. Standard mode offers exactly-once delivery and optimisation for durable, long-running work, which is a guarantee this workload pays nothing for.
How batching cuts the cost of each execution
Batching is the optimisation with the clearest arithmetic. The router can group several messages and invoke one workflow for the resulting batch. Processing 100 messages individually could produce 100 executions and 400 HTTP requests to the content modules; with batching, the same work fits in one workflow with four requests.
Express charges for requests and for run time, so the trade is between fewer, longer runs and many short ones. Zillow's rule of thumb is that a batch of ten costs less than ten separate runs as long as it takes no more than ten times as long as one run. One message takes about 1.2 seconds, which sets a 12-second ceiling for the batch, and in practice a batch of ten runs only 10 to 20 percent longer, about 1.4 seconds, which Zillow puts at roughly 90% cheaper.
The batch has to stay inside two limits. The payload stays under 256 KiB, which keeps memory billed at the 64 MB minimum, and every step must accept bulk input, a requirement built into the module API because a module that ignores it fails to build. Batching also has to preserve per-message visibility: each batch is keyed by tracking ID, the modules return a result per tracking ID, and the router delivers, retries or drops each message individually, so one bad message cannot fail the whole batch.
What the team measured after the migration
The largest reported gain is velocity. Testing a variant or launching a campaign used to mean changing a dedicated generator service, often several. Now a team edits configuration, reuses modules other teams already built, and validates the workflow before production.
The engineering estimates are specific. Zillow puts the build at 9 development weeks, against 32 for a comparable in-house engine, and ongoing maintenance at roughly 2 hours a week, against 8. Testing supports those numbers: shared JSONata expressions live in their own files, each has unit tests, and the runner executes them through the Step Functions TestState API so a failing case blocks the deploy.
Quotas received the same treatment. The team reviewed StartExecution rate limits, HTTP Invoke concurrency and CloudWatch logging throughput before traffic ramped, sized the concurrency it needed from a capacity model, and worked with AWS to raise the HTTP Invoke quota before each rollout phase so throughput limits never gated the ramp.
What a small team can copy from this design
Four practices transfer to a much smaller estate. The first is describing each message as input data plus configuration, which turns a campaign into a file a product team can own. The second is keeping modules stateless and schema-validated, so the same module serves many workflows and every input contract is checked before deployment.
The third is making every step accept a batch. That single requirement is what allows the router to group work and what keeps the per-message guarantees, since a batch keyed by tracking ID still returns a result per message. The fourth is planning service quotas before traffic arrives, because HTTP Invoke concurrency is the limit that gates a high-volume Express deployment first.
The human part of the design is where most small implementations fail. A configuration layer only stays trustworthy when the mappings behind it are tested against the real runtime, and Zillow's habit of running every shared expression through the TestState API before deploy is the practice worth lifting whole. Designing an automation pipeline that a human approves at the step that matters depends on the same foundation: a declaration a person can read, tests that prove it, and a runtime that stops when the tests fail.
The rest of the stack is familiar to anyone who has replaced bespoke code with a declarative layer, and the AWS post is unusually generous with the numbers that make the decision comparable. For a team sending only a few thousand notifications a day, this architecture can be excessive. Zillow still operates a custom router, deployment tooling, per-campaign queues and circuit breakers. Express delivery also requires idempotency across every module. Batching and configuration-driven workflows earn that operational cost when reuse and throughput are large enough to justify a platform team; a simple queue consumer may be the safer choice at smaller scale.
Sources
Source: How Zillow runs 100 million daily notifications on AWS Step Functions — AWS Architecture Blog, 11 November 2026, aws.amazon.com/blogs/architecture, retrieved 2026-11-12 (Zillow sends more than 100 million personalized email and push notifications a day, each as its own short-lived Step Functions Express Workflow execution; the previous system supported more than 20 campaigns from more than 10 microservices, each with its own workflow, models and business rules; Campaign Orchestrator is a configuration-driven orchestration layer and the Universal Content Generator is a library of stateless, schema-driven modules for property hydration, hidden homes, common data, recommendations, A/B testing and content assembly; steps call modules through Step Functions HTTP Invoke with an EventBridge connection that keeps the API key in AWS Secrets Manager; a router reads from Amazon SQS and owns queue polling, batching, idempotency checks, pause and resume, replay coordination, speed control and routing; each campaign keeps its own state machine and SQS queue with its own polling threads, worker pool and circuit breakers; Zillow evaluated Temporal, Flowable, an in-house engine and both Step Functions modes against reliability, total cost of ownership, development speed and cross-team scale, estimated roughly a quarter of the build and maintenance effort against the alternatives, and ramped from 14 million to more than 100 million workflows a day at several thousand executions per second at peak; Express mode offers the higher default start quota, latency for short-running workflows and at-least-once delivery sufficient with idempotent modules, while Standard offers a lower default quota, tuning for durable long-running workflows and exactly-once delivery; batching turns 100 individual messages and 400 HTTP requests into one workflow with four requests, a single message takes about 1.2 seconds, a batch of ten runs 10 to 20 percent longer at about 1.4 seconds and roughly 90% cheaper than ten separate runs, the payload stays under 256 KiB with memory billed at the 64 MB minimum, and results are returned per tracking ID so each message can be delivered, retried or dropped separately; Zillow estimates a 9-week build against 32 weeks for a comparable in-house engine and maintenance of about 2 hours a week against 8; shared JSONata expressions have unit tests executed through the Step Functions TestState API in continuous integration, and a failing case blocks the deploy; the team reviewed StartExecution rate limits, HTTP Invoke concurrency and CloudWatch logging throughput and raised the HTTP Invoke quota before each rollout phase).
Source: the AWS Architecture Blog case study of 11 November 2026, retrieved 2026-11-12. Every volume, duration, cost comparison and engineering estimate above traces to that page; the reading of which practices transfer to a small team is Netics analysis. Images: three architecture figures from the AWS post and one capture of it, reproduced with credit, plus three Netics editorial diagrams. Netics editorial, neticslabs.com.