Building a voice AI that answers the phone in Bangla

Most "voice AI" demos are a microphone in a browser tab. A phone call is a different problem. The moment you put an AI on a real phone line, three things that a demo hides become the entire engineering problem: latency, telephony, and a voice that sounds human in the caller's own language.

Here is what actually mattered building a Bangla-speaking call agent.

The latency budget is smaller than you think

On a real call, people start to feel something is wrong at around 300 milliseconds of dead air. That is your entire round-trip budget: speech in → transcription → model reasoning → speech out → back down the phone line. Every hop spends part of it.

The naive pipeline (record a turn, send it up, wait for a full response, synthesize it, play it) blows the budget on the first sentence. The fix is to stream every stage and overlap them: begin synthesizing the start of the answer while the model is still generating the end of it.

Telephony is the unglamorous hard part

The model is the easy 20%. Bridging a SIP telephony gateway into a realtime audio pipeline (call setup, codecs, jitter, and the fact that phone audio is narrow-band and lossy) is where the real work lives. A model that sounds great on studio audio can fall apart on 8 kHz phone-band sound.

Why a custom voice

No hosted TTS produced conversational Bangla we were willing to ship. So the voice is a custom-trained model, tuned for phone-band audio specifically.

That is a real tradeoff: a custom voice is far more work and has to be maintained, versus a hosted one you never think about. But there was no hosted option that made the call feel like a person, and on a phone call the voice is the product.

What I'd tell someone starting

• Design the whole thing around the 300 ms budget from day one; you cannot bolt streaming on afterwards.
• Spend your hardest week on the SIP/telephony bridge, not the prompt.
• Test on real phone audio early. Studio audio lies to you.