Speech Pipeline & LLM Systems

Emitrr runs live voice conversations for other businesses, which means the pipeline gets exactly one chance to get a sentence right before the moment it was said has already passed. I spent my internship inside that pipeline, mostly in the 300–500 milliseconds between someone talking and the system understanding them.

The transcription layer was the first target. I migrated the real-time voice pipeline off GCP Speech-to-Text and onto Deepgram, then spent most of the tuning time on endpoint detection and audio buffering — the unglamorous knobs that decide whether a pause means "the sentence is over" or "they're still thinking." That work cut average transcription latency by 30%, a 350ms end-to-end reduction that's small enough to sound trivial and large enough that people on the call actually noticed.

On the output side, I integrated ElevenLabs for text-to-speech and optimized the streaming path, bringing TTS latency down from 500ms to 350ms — enough to make the voice on the other end sound like it was keeping up rather than catching up.

The harder problem was what happens when a call doesn't go cleanly. I built a multi-threaded state manager that correlates five concurrent call states in real time, which made isolating pipeline failures far faster than it had been. Then I replaced the hard-coded fallback rules for incomplete calls with a reward-driven agent that learns, rather than a lookup table, what the next best action should be.