Sub-600ms voice API enables sellable AI phone agents

What to build

A vertical AI phone-agent product for one niche (dental/medical clinics, restaurants, or car dealers) built on ElevenLabs' new Realtime speech-to-speech models: it answers inbound calls, handles bookings/FAQs, and hands off edge cases, sold as a monthly subscription to small business owners who can't staff a 24/7 front desk.

ElevenLabs' sub-600ms speech-to-speech models cross the latency threshold where AI phone calls finally feel natural, opening a short window for solo builders to wrap the API into niche voice-agent products before larger platforms commoditize it.

Demand

Small clinics, restaurants, car dealers, and real-estate offices lose bookings to missed calls and can't afford round-the-clock reception staff; a natural-sounding AI phone agent that doesn't feel laggy is now cheap enough and good enough for them to actually adopt.

• ElevenLabsProduct launch

  ElevenLabs announced two new speech-to-speech models purpose-built for real-time AI conversations, positioned as an upgrade over prior text-to-speech-based pipelines.

• ElevenLabsProduct spec

  Reported sub-600ms end-to-end latency for the new speech-to-speech API, the threshold widely cited in voice-AI circles as the point where conversations stop feeling laggy.

• Industry analysisMarket context

  Low-latency speech-to-speech APIs are already enabling a wave of niche voice-agent products for clinics, restaurants, car dealers, and real estate, built by small teams or solo developers wrapping the underlying model API.

Stack

• ElevenLabs Realtime API
• Twilio Voice (SIP/PSTN)
• OpenAI or Claude for dialogue logic
• Supabase
• Cal.com or Google Calendar API for booking
• n8n or Make for call-flow orchestration

Solo + AI difficulty

Wiring ElevenLabs Realtime to Twilio for inbound calls and a basic booking webhook is straightforward with AI-assisted coding; the hard part is tuning turn-taking, interruption handling, and keeping real-world latency (phone network, ASR/TTS handoffs) under the natural-feeling threshold the benchmark implies. Realistic MVP for a single vertical: 1-3 weeks solo.

Entry threshold

Low build barrier since it is an API call away, a solo builder can prototype a working voice agent in days using existing telephony integrations like Twilio; the real work is picking a vertical, building the prompt/flow logic and handling edge cases in calls.

Window

3-6 months

Where to find first users

• Direct outreach to local clinics, restaurants, or dealerships
• r/smallbusiness and niche Facebook groups for dentists/restaurant owners
• Indie Hackers and a Product Hunt launch
• Voice-AI reseller/agency directories and local business associations

Competitors

• Bland AI
• Vapi
• Retell AI
• Synthflow AI

Counter-signals & risks

• Benchmark latency figures from vendors are often measured under ideal conditions; real-world phone network latency, ASR/TTS handoffs, and concurrent load could push actual response times well above the advertised sub-600ms figure.

• Competitive pressure from OpenAI's Realtime API, Google, and other speech-to-speech entrants could commoditize this latency advantage quickly, compressing margins for solo builders reselling thin wrappers.

• Vertical voice-agent products touching bookings and customer data (e.g., clinics, real estate) may face regulatory/compliance burdens (HIPAA, call-recording consent laws) that raise the barrier for solo developers beyond just technical integration.