ElevenLabs Voice Agent - Build Experience & the Question of Where Intelligence Should Live
An honest field report on building “QC Sahayak”, a quick-commerce delivery-partner helpline, on ElevenLabs Conversational AI - what worked, what fought back, and the architectural question every team will face: where should the agent’s intelligence actually live?
The problem
Quick-commerce delivery partners - the riders who get groceries, food deliveries to your door in ten minutes - lose a surprising amount of their working day just trying to get help from their parent organisation. Address nahi mil raha. Store ne pickup ready nahi kiya. Payout kab aayega? Leave kaise lagaun? Most of this lands in noisy WhatsApp groups or on human support agents handling hundreds of riders at once. It’s repetitive, it’s mostly lookups, workflow execution and status updates, and it happens while the rider is on a bike in traffic - exactly the situation where a phone call beats tapping through an app.
The question worth answering: can a voice agent handle the common cases well enough that a rider would rather call it than wait for a human? And - just as important for anyone evaluating this space - what does it take to build, where does a horizontal voice platform shine, and where does it start fighting you? This is a build report, not a pitch.
🎥 Demo video - a 3-min screen recording of the agent handling all three flows in Hinglish:
What I built
QC Sahayak is a voice agent for delivery partners. A rider calls in (a web widget for now; can be exposed via telephony for actual technology demonstration) and self-serves three things, all in natural Hinglish - the Hindi-English code-mix riders in majority of Indians actually speak, switching languages inside a single sentence:
- Policy queries - “payout kab milta hai?”, off-time rules, incentive structure, helmet policy. Public information; no identity should ideally be needed.
- Account details - today’s earnings, pending payout, last settlement, shift timings, hub. Specific to the rider; pulled live from the system of record where workflow and business rules live. The rider identifies themselves once.
- Delivery issues - address not found, pickup not ready, running late. The rider gives an order ID; the agent logs the issue and reads back the next step.
A deliberate design decision for the experiment design underpins all three: each bucket uses a different mechanism, because each job has a different cost and shape.
- Policy is retrieval. Answered directly from a knowledge base (RAG) - no tool call, no reasoning step, effectively free per answer. This is the highest-volume, lowest-complexity path, so it gets the cheapest path.
- Account is a managed connector. A live lookup against a CRM record through ElevenLabs’ native Salesforce integration - one connector call, no integration endpoint to write for the read path.
- Delivery is a custom tool layer. Order data and issue logging run through a custom MCP server and a small webhook - a clean, reusable tool boundary rather than logic stuffed into the prompt.
Splitting it this way keeps token cost down (policy questions never invoke the model’s tool machinery), keeps each piece independently testable, and maps the architecture to how the business actually thinks about these requests.
The stack, concretely
No abstraction - here’s exactly what’s under the hood:
| Layer | Choice |
|---|---|
| Voice AI platform | ElevenLabs Conversational AI - bundles STT, LLM orchestration, TTS, turn-taking and RAG |
| Voice | Monika Sogam - a Hindi-native female voice from the ElevenLabs library, built for IVR/support flows |
| Language model | Gemini 2.5 Flash Lite (selectable inside ElevenLabs) - cheapest tier that handled Hinglish + tool-calling |
| Language | Hinglish via the native hinglish_mode flag + language: hi |
| Policy knowledge | ElevenLabs Knowledge Base (RAG) over a markdown policy doc |
| Account data | Salesforce custom object (QC_Rider__c) via the native Salesforce connector → SOQL |
| Order data | A custom MCP server exposing get_order, list_orders_for_rider, get_pickup_status |
| Issue logging | A tiny Node webhook that appends to a log and returns next-step guidance |
Two real transcripts from the platform’s own analytics. First, the account-lookup flow - the Salesforce tool fires and the agent answers settlement and hub/shift in Hinglish:
And the order flow, where the MCP get_order tool fires and the agent reads back the delivery address:
Those screenshots are the platform’s built-in analytics view, not something I built - which itself turned out to matter (more below).
What I liked
Being objective: a lot of this was genuinely good, and a few things were better than I expected.
The CLI and config-as-code experience - the standout
This deserves top billing because it changed how the whole build felt. The entire agent - system prompt, voice, model settings, every tool, the knowledge base, turn-taking, ASR config - is a single JSON file, deployed with one command: elevenlabs agents push. Pull it back down with elevenlabs agents pull. That’s it.
That sounds mundane until you live in it. It meant:
- The agent is version-controlled like any other code. Every prompt tweak, every tool change, every voice setting is a git diff. I could see exactly what changed between “the agent that over-asked for the rider ID” and “the agent that remembered it” - and roll back instantly when an experiment made things worse.
- I drove almost the entire build from the terminal, with an AI coding assistant. Edit the prompt in the file, regenerate the Salesforce connector setup script, run a smoke test against the webhook, push the agent - all without opening the dashboard. Because the whole agent is a readable file, an AI pair-programmer can ingest the entire agent definition, reason about it, and make safe, reviewable changes. A click-only console locks that workflow out completely; you can’t diff a dashboard, and an assistant can’t “see” it. Add to that, the skill from Elevenlabs turbocharged the entire process.
- Reproducibility. The agent is recreatable from the repo. Nothing important lives only in a UI that someone has to remember to re-click.
If you’re a builder who lives in a coding workflow, this is the feature that compounds. Most agent platforms are dashboard-first; ElevenLabs’ CLI-first posture was, for me, the single biggest day-to-day productivity win - and it’s exactly the surface that makes AI-assisted development of the agent itself practical.
Native Hinglish
hinglish_mode: true plus a Hindi-native voice produced believable code-mix on the first try - no custom STT tuning, no phonetic workarounds just to get started. For an Indian-language use case, that’s a real head start over wiring up your own speech pipeline.
Knowledge-base RAG that’s basically free
Upload a policy doc, reference it, done. The most common questions never trigger a tool call - they’re answered straight from retrieval. The cheapest possible path for the highest-volume queries, with zero infrastructure.
Out-of-the-box analytics and post-call analysis
This pleasantly surprised me. Every conversation is captured with a full transcript, per-turn latency breakdowns (LLM, TTS, RAG, tool time), tool-call success/failure, and structured data collection - with zero setup. The two screenshots above are the platform’s analytics. For debugging “did the agent actually call the right tool” and for understanding latency, this was invaluable.
Fine-grained behaviour controls
ElevenLabs exposes the knobs that actually matter for voice UX: turn-taking eagerness (so the agent doesn’t cut off a rider mid-sentence), a soft-timeout nudge (“Hello, aap line par hain?”) when the caller goes quiet, ASR keyword boosting for domain terms and IDs, a “typing” tool-call sound to cover latency, and an LLM cascade/fallback. These are the difference between a demo and something that survives a real, noisy phone call. There are many hyper-parameter tuning one can do to make the agent sound really nice!
Tool+MCP support
Plugging in a custom MCP server for the order tools - rather than registering a dozen one-off webhooks - kept the tool layer clean and portable. The same MCP server could front a different agent tomorrow. In addition to that, there are many out of the box connectors such as Salesforce and others, which dev+integration work fast.
What didn’t work well
The honest part. None of these are dealbreakers, but they’re real, and anyone building a multilingual voice agent or a tool-rich agent will hit them.
English-word pronunciation - still my main open issue
Mid-Hinglish, the TTS sometimes Anglicizes or mis-pronounces English words, and occasionally mangles transliterated place/hub names. It’s the most noticeable rough edge to a native ear, and I want to be straight that it’s unsolved in my current build. That said, the levers exist - pronunciation dictionaries, spelling tricky words phonetically in the reply text, choosing terms the voice handles cleanly - and I’d expect it to be tractable with focused effort. It just wasn’t a single-flag fix.
Gender drift in Hindi
The agent persona is female, and Hindi conjugates verbs by speaker gender (“main batati hoon” vs “main batata hoon”). Telling the model “you are female” wasn’t enough - it drifted to masculine/neutral forms. I had to be prescriptive and supply the exact verb forms, and even then it occasionally slips. For a brand-controlled voice, this is an ongoing QA surface, not a one-time prompt line.
Indian number formatting
Amounts should sound natural - ₹2,180 as “do hazaar ek sau assi rupaye”, not “two thousand one hundred eighty” - and IDs must be read digit by digit. None of this is automatic; the cleanest fix is to pre-format numbers into words on the server before the model sees them. This is also why I switched to short, voice-friendly IDs like RD101 and ORD1001 - the original dashed alphanumerics were an ASR/TTS minefield.
Integration challenges - where I lost the most time
The integration surface is where “managed” stops meaning “frictionless,” and it’s worth detailing because these are the hours that don’t show up in any demo:
- The Salesforce connector setup is a multi-step dance. Create a connection, inspect which operations it exposes, register the right tool, wire it to the agent. The connector’s generic query tool worked, but getting there meant discovering the correct request schema largely from error messages. The CRM side needed an OAuth client-credentials flow with a “Run As” user - a specific, easy-to-miss toggle that silently fails until it’s set.
- MCP transport was trial-and-error. ElevenLabs connects to MCP servers over SSE / Streamable-HTTP. Getting the handshake right - the correct transport type, and crucially an absolute endpoint URL (the SDK defaults to a relative one that the platform can’t resolve) - took several iterations against opaque “failed to connect” errors.
- The MCP tool-approval policy defaults to “always ask.” In a voice call there’s no one to click “approve,” so tool calls silently hang until you switch the server to auto-approve. Obvious in hindsight; an hour lost finding it.
- Tool routing took real prompt iteration. With a small, fast model, getting the agent to call the right tool - and to actually call a tool instead of narrating “main check kar rahi hoon” and stopping - needed blunt, explicit routing instructions (“for order data use this tool, never the CRM query”). A larger model would likely need less hand-holding, at higher cost per turn.
- An Alpha feature fought me. I tried the platform’s new Procedures (SOP-style step sequencing). It worked - and overrode my tool flow, running its own script instead of calling the order tool. Since the prompt + tools already encoded the logic, I removed it. Worth watching as it matures.
The throughline: the conversational layer is solved and easy. The integration layer - connecting the agent to real data and real actions - is where the genuine deployment effort lives. Hold that thought; it’s the bridge to the bigger question.
The bigger question: where should the agent’s intelligence live?
Everything above is about ElevenLabs as a conversational front end, and on that count it’s excellent - best-of-breed at the hard parts of voice (speech quality, turn-taking, latency, multilingual handling), and fast to stand up. In fact, ElevenLabs is a chosen vendor for TTS segment of Agentforce Voice by Salesforce.
A headless, horizontal agentic strategy - plug the best voice engine into whatever you’re building - is genuinely attractive. But the moment the agent has to do something in the business - not just answer, but log a case, reassign an order, check it against a rule, trigger a workflow, route an escalation, write a status back - the centre of gravity shifts. In this build, the rider data lives in Salesforce, and the operational logic should live there too. With the horizontal approach, every one of those workflows has to be re-coded inside the voice agent: I write the SOQL, I stand up the MCP server, I build the webhook, I encode the decision flow in a prompt, and I own all of that glue forever.
The platform that already holds the data and the business rules - Salesforce, ServiceNow, and their peers - can ground the agent in that data and that decision flow natively, without re-implementing it. That’s the crux: a horizontal agent re-creates the workflow; an OEM-native agent inherits it.
Here’s the honest, parameter-by-parameter comparison - drawn from actually building the horizontal side and weighing it against what a native platform agent (e.g. Salesforce Agentforce, ServiceNow’s agentic layer) provides:
| Dimension | Horizontal voice platform (ElevenLabs) | OEM-native agent (Salesforce / ServiceNow) |
|---|---|---|
| Voice quality (TTS) | Excellent - its core strength | Often the same TTS engine under the hood - not a differentiator. Some features could be gated |
| Time to a talking demo | Fast - the voice layer is its strength | Fast too, if you’re already on the platform |
| Time to a business-ready agent | Slower - workflow, write-back, testing all custom integrations | Faster - actions, flows, approvals already exist |
| Data grounding | Hand-built connector / SOQL / MCP per source; breaks on schema change | Native grounding, zero-copy to records and metadata; adapts as schema evolves |
| Knowledge / RAG | Smooth, but siloed to the one agent | Platform RAG over enterprise knowledge + connected sources, reused across channels |
| Complex workflow & business rules | Re-coded in the bot + middleware (the wrong layer) | Inherited - triggers existing flows/rules/approvals |
| Write-back to system of record | Generic create/update; blind to validation, triggers, sharing. Could be limited by OEMs exposability | Platform writes to itself - respects all automation natively |
| Observability | Genuinely good OOTB analytics + transcripts | Supervisor console, call records, native analytics, Agentic Observability |
| Omnichannel | Voice-siloed; chat/email are separate builds | One agent model across voice/chat/email/messaging |
| Governance & compliance | You assemble and certify each vendor in the chain | Inherited platform posture (retention, residency, certifications) |
| Maintenance | You own the glue forever; every schema change is a risk | Platform-managed; agents inherit improvements |
| Control / flexibility | Total - you can build anything | Bounded by what the platform exposes |
Why this feels familiar
This pattern has played out before. A decade ago, prediction - churn scores, propensity models, demand forecasts - was built on bespoke ML tooling: a data-science team, a custom feature pipeline, a model server, an MLOps stack, all outside the application, with enormous effort spent piping predictions back to where work happened. Then enterprise platforms embedded prediction in the flow of work - scoring computed natively inside the CRM record, the service ticket, the ERP transaction. For the majority of mainstream jobs, the bespoke external stack stopped being worth it - not because it was worse (often it was more capable), but because proximity to the data and the workflow beat raw capability. Specialist ML teams didn’t vanish; they moved up to the hard, differentiated problems and let the platform handle the commodity 80%.
Agents look like they’re on the same arc. A custom agent on a horizontal platform is the exciting, flexible path today - much like rolling your own ML pipeline was then. But where the workload, data, and business rules already sit on one platform, the in-the-flow-of-work agent will likely win the mainstream cases for the same decisive reason embedded prediction did: it’s already where the work is, and it doesn’t have to re-implement the work to act on it. For the larger scheme of things, these in-the-flow-of-work agent can connect via protocols such as A2A, MCP etc in a headless way.
So which should you choose?
Not either/or - a decision rule:
- Lead with the horizontal platform (ElevenLabs) when the use case is genuinely voice-first, the data needs are light or span systems no single OEM owns, you need control the platform can’t give you (custom audio handling, exotic routing, tight latency budgets), or you’re validating an idea before committing. Its voice quality, speed, and config-as-code workflow are real advantages.
- Lean toward the OEM-native agent when the majority of the relevant data and business logic already lives on that platform, you need omnichannel consistency, you’re in a regulated industry where inherited compliance is an accelerant, and the agent’s job is mostly to act on the business rather than just talk.
For this delivery-partner agent specifically, the honest read is a hybrid: the voice-first triage and policy Q&A are a natural fit for ElevenLabs; the account lookups and - especially - the write-back workflows (log a case, reassign an order, update an SLA) point toward the platform that owns the rider data and the operational rules. The likely end-state is a horizontal voice front end handing off to an OEM-native agent for the business-logic back end, with a clean seam between them.
Where this lands
The commodity layers of voice AI - can it speak, can it understand, can it call a tool - are solved, and impressively easy to assemble on ElevenLabs; the CLI-and-config-as-code workflow makes building the agent (and building it with an AI assistant) a pleasure. The remaining thoughts splits in two:
- Localisation fidelity - pronunciation is the open item, alongside gender and number handling, a focused grind I’m optimistic about
- Strategic question - where the business logic and ultimately agentic reasoning should live.