99 Problems and Cold Start Ain't One

If you're building voice AI, you're going to hit this wall. Here's how it breaks, in the order it'll break for you.

You'll start with always-on containers, because they work. Every customer on your platform gets a Fargate task running 24/7. Latency is great. Isolation is clean. Then your finance team runs the numbers.

A single always-on Fargate task for a voice pipeline -- STT, LLM, TTS, all loaded in memory -- costs around $29 per month in our configuration.¹ Ten customers: $290. A hundred: $2,900. A thousand: $29,000. And most of those containers are sitting idle, waiting for a phone call that might come once every few hours.

So you'll scale to zero. And that's where the real problem starts.

The sixty-second wall

When a Fargate task has been sleeping, waking it up takes time. Engineering teams publishing real-world numbers report cold starts of 20-60 seconds for typical production workloads, broken down into ENI provisioning (10-30s), image pull (5-60s for large images), and layer extraction (2-15s).² Voice AI images sit at the large end of that range because you're shipping model weights alongside your code. Our pipeline averages 60-90 seconds cold.

Sixty seconds of silence. In a domain where quality is measured in milliseconds.

Human conversation breaks at 500 milliseconds -- the natural pause between speakers.³ Voice AI platforms target under 800ms voice-to-voice latency as the floor for "sounds like a person."⁴ Industry reporting suggests customers hang up 40% more frequently when agents take longer than one second to respond.⁵ Call center research shows abandonment spikes at the 30 and 60-second marks,⁶ and the industry treats an abandonment rate above 10% as a serious operational problem.⁷

A cold start is roughly 100x past the threshold where callers start perceiving something is wrong.

Why the obvious answers weren't enough

When we hit this wall, we worked through the standard playbook. Each option has a real answer; none of them were sufficient.

Lambda with provisioned concurrency. The natural first thought: skip Fargate and run the voice pipeline on Lambda with a warm pool. This fails on two fronts. Lambda's 15-minute execution limit is awkward for long calls (think insurance intake, technical support). More importantly, provisioned concurrency costs you whether you use it or not -- you've reinvented always-on with extra steps, and paid Lambda premium pricing for the privilege.

Fargate with SnapStart-style snapshots. Fargate doesn't have SnapStart. Even if it did, snapshotting a container with loaded STT/LLM/TTS models and then restoring GPU memory state isn't a clean operation. The models don't live at the container boundary.

Seekable OCI (SOCI) lazy image loading. SOCI is real and it helps -- AWS reports up to 50% faster task starts for large images by lazy-loading filesystem layers.⁸ But SOCI optimizes image pull, not model load. A voice container has to download weights, allocate memory, and warm inference before it can take a call. SOCI takes us from 90 seconds to maybe 45. Still 90x past the threshold.

Pre-warmed Fargate pool. Keep a pool of generic warm containers ready to receive any customer's traffic. This works for cold-start timing but breaks everything else. The moment a "generic" container starts serving a specific customer, it's loaded that customer's data into memory, and now it can't be returned to the pool without either destroying it or sharing memory across customers. You've rebuilt Option C below without the operational benefits.

Shared multi-tenant execution. Run everyone on the same always-warm containers. This solves cold start and cost at the same time. But your customers' data now shares a process, connection pools, and memory with everyone else's. Isolation depends entirely on application-layer discipline. A single bug in prompt scoping, session handling, or cache key generation can leak one customer's context into another's. It works until it doesn't, and when it doesn't, you find out from the customer, not a monitor.

Every voice AI platform today picks a point on this triangle:

Pick two of: low cost, no cold start, full isolation. That's the conventional wisdom.

The approach

Here's what we built. I'll state it plainly, then explain it.

We keep a small pool of permanently warm multi-tenant gateway containers in front of per-customer isolated containers. Each customer has their own Redis Cloud instance and their own DynamoDB tables -- real, separate infrastructure, not a shared database with prefix scoping. When a customer's container is scaled to zero, the Twilio webhook for that customer's phone number points to the gateway. When a call arrives, the gateway identifies which customer the number belongs to, opens connections to that customer's isolated Redis and DynamoDB, loads the customer's configuration, answers the call using the customer's logic, and triggers the customer's container to start spinning up. When the container is ready, we hand off mid-conversation at a natural pause. The caller never notices.

Once you see the shape of it, the name of the trick is: compute and data are separable, but isolation is at the data layer, not the compute layer. The gateway is shared compute running isolated data paths. That's the whole idea.

What makes this actually hard

The idea takes a paragraph. The engineering took considerably longer. Three parts were harder than they look.

Session state across the handoff. The gateway is writing conversation history, extracted parameters, and function states to the customer's Redis while the call is happening. The customer's container, once it's warm, needs to read that state and continue without a gap. The non-obvious failure mode: the handoff signal fires, the customer's container starts reading, but the gateway's last write hasn't been flushed yet. You get a container that believes the caller hasn't said the thing they just said. We solved this with a commit-then-signal pattern -- the gateway finalizes its write and acknowledges it before emitting the handoff signal, and the customer's container reads from a consistent snapshot. Not novel in distributed systems terms, but easy to get wrong.

Finding the natural pause. The handoff can't happen in the middle of a sentence. It has to land during a moment where the caller expects a brief beat -- "let me check availability for you," a tool call, a database lookup. Those moments exist naturally in voice AI because the LLM often dispatches a backend query mid-conversation. We detect these pause windows from the pipeline state and hold the handoff until one opens. If the container warms before a pause opens, we wait. If a pause opens before the container is warm, we keep the gateway serving. The handoff is opportunistic, not scheduled.

The failure modes. What happens if the customer's container never comes up? The gateway keeps serving the call on the customer's isolated databases until the call ends. The conversation completes. The customer gets a monitoring alert. What happens if the gateway loses its connection to the customer's Redis mid-call? We fail closed: the call ends with a graceful message rather than falling back to shared storage. What happens on Twilio webhook retries if the gateway is slow? The gateway is sized for the retry window with margin; we've never hit the ceiling in production, but the failover path routes the retry to a second gateway instance in a different AZ. None of these are glamorous. All of them had to be thought through before we could trust the architecture.

Isolation, concretely

This is the part I'd want to see most, if I were evaluating a partner.

Every customer on the platform has their own dedicated Redis Cloud instance and their own dedicated DynamoDB tables. Not a shared Redis with keys like custA:session:123 and custB:session:456. Two separate Redis Cloud instances, with different connection strings, different credentials, living in different network paths.

When the gateway receives a call, it:

1. Identifies the customer from the Twilio number.
2. Fetches that customer's connection credentials from our control plane.
3. Opens connections to that customer's Redis and DynamoDB.
4. Serves the call, reading and writing only through those connections.
5. Closes the connections at handoff.

During the call, the gateway's process holds open connections to exactly one customer's infrastructure. Another customer calling at the same moment hits a different gateway container (or a different invocation on the same container, with its own connection scope). Two customers' data never share a connection pool, a cache, or a process-local variable.

The practical consequence: a bug in the gateway can't leak one customer's state into another's, because there's no shared state to leak into. The worst case of a bug is that the current call fails. It can't cross-contaminate.

The trade-offs we accepted

Every architecture has them. Here are ours.

First-call latency after long idle. The gateway pools its connections, but establishing a fresh connection to a customer's Redis Cloud instance after hours of idle can add 100-200ms to the first call. We hide this in the gateway's greeting but it's real.

~0.7 second of lost buffered audio at handoff. The Twilio media stream handoff is not atomic -- buffered audio in the outgoing stream can be lost when the WebSocket disconnects. Sometimes the transition is clean; sometimes the assistant's last word is clipped. This is a protocol limitation, not a timing bug, and it's the primary reason we need a natural conversational pause to hand off. The pause masks the discontinuity.

Gateway cost. The gateway pool isn't free. We need enough capacity to absorb concurrent cold-start calls across all scaled-to-zero customers. In practice this is a small fraction of the always-on cost it replaces, but it's a non-zero floor. The cost chart below reflects this.

Container wake failure consumes shared capacity. If the customer's container never comes up, the gateway keeps serving the call on the customer's isolated databases until the call ends naturally. The conversation completes -- the caller is unaffected -- but the gateway is occupied beyond the typical 60-90 second handoff window, consuming shared capacity that would otherwise serve other cold-start calls.

Infrastructure failure fails closed. If the control plane can't resolve a customer's database credentials, or the customer's Redis is unreachable, the call ends with a graceful message rather than falling back to shared storage. This preserves isolation but means our availability is capped by the availability of each customer's dedicated infrastructure.

Scope. This approach works because voice calls have natural pauses and because our voice pipeline's "identity" lives in data (prompts, config, voice selection, tool definitions) rather than in code. A workload where per-customer logic required custom binaries would need a different solution.

The numbers

For a platform serving 100 customers:

Mode	Monthly Compute	Savings
Always-on (24/7)	$2,884	--
Working hours (9-5 M-F)	$688	$2,196 (76%)
On-demand (scale to zero)	$144-288	$2,596-2,740 (90-95%)

These are pure Fargate compute costs for the customer tasks. The gateway pool is shared infrastructure already running for the platform, and its cost is included in our platform fee rather than billed per customer.

Scaled up: a platform running 1,000 customers on always-on spends roughly $350,000 a year on compute. Working-hours mode drops it to $82,000. On-demand drops it to roughly $20,000. Whether or not you use Remote Assistant, this is the order of magnitude a voice AI platform is leaving on the table when it treats cold start as unsolvable.

If you're evaluating us

The reason I wrote this post at this length, with this much of the architecture exposed, is that I think voice AI platforms are converging on a set of assumptions that aren't actually forced. "Always-on is the price of good latency." "Scale to zero means hold music." "Multi-tenant means giving up isolation." None of these are laws. They're the consequence of thinking about cold start as a container-startup problem instead of a call-answering problem.

Once you reframe the question -- how do we answer the phone instantly while the container starts in the background? -- the architecture writes itself. The hard part is execution: session continuity, natural-pause detection, graceful failure modes, real per-tenant infrastructure. But the shape of the idea is available to anyone who asks the right question.

We're sharing it because we'd rather the industry's voice AI get better than keep this as a moat. If you're building on Remote Assistant, you get this architecture for free. If you're building your own platform and this post helped, tell us -- we'd like to know.

---

Sources

---

Remote Assistant is an AI operations platform that deploys autonomous AI employees for businesses. Learn more about building on our developer platform.

Footnotes

1. Based on our standard voice pipeline Fargate task configuration in us-east-1, running 730 hours/month. Your mileage varies with model choice and memory footprint. ↩

2. Dalai, Dikhyant Krishna. "Taming Cold Starts on AWS Fargate." AWS in Plain English, October 2025. https://aws.plainenglish.io/taming-cold-starts-on-aws-fargate-the-architecture-behind-sub-5-second-task-launches-622ebd73b051 ↩

3. AssemblyAI. "The 300ms rule: Why latency makes or breaks voice AI applications." https://www.assemblyai.com/blog/low-latency-voice-ai ↩

4. Retell AI. "AI Voice Agent Latency Face-Off 2025." https://www.retellai.com/resources/ai-voice-agent-latency-face-off-2025 ↩

5. Telnyx. "Voice AI agents compared on latency." 2026. https://telnyx.com/resources/voice-ai-agents-compared-latency ↩

6. Sprinklr. "Call Center Statistics." https://www.sprinklr.com/blog/call-center-statistics/ ↩

7. GetVoIP. "What is Call Center Abandonment Rate." https://getvoip.com/blog/call-center-abandonment-rate/ ↩

8. AWS. "Reducing AWS Fargate Startup Times with zstd Compressed Container Images." AWS Containers Blog, 2022. https://aws.amazon.com/blogs/containers/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/ ↩