health-desk-ai
A voice agent that books hospital appointments, with a lip-synced avatar.
- LiveKit
- Deepgram
- Cartesia
- Gemini
- PostgreSQL
- Simli
See it working
Problem
Hospital front desks answer the same calls all day: book, cancel, reschedule. A voice agent can take that load, but only if it behaves like a careful receptionist rather than a chatbot with a microphone.
Hallucination in voice is a different problem than in chat. In chat, a user scrolls past a wrong answer. In voice, a patient hears that their appointment is booked for Monday and believes it. So this build treats correctness and timing as architecture problems, not prompt problems.
Architecture
Decisions and trade-offs
1. Turn detection has two layers, and most builds only use one.
Silero VAD runs every 32ms but only says that silence happened. A transformer end-of-utterance model reads the actual transcript and predicts whether the sentence is semantically complete: I want to book for, followed by a pause, is not done. 10% completeness means keep waiting, 95% means respond now.
2. Endpointing that adapts to each speaker.
The silence timer after VAD uses an exponential moving average of the callers speaking pattern. A fast talker gets about a 300ms delay, a slow talker up to 2.5s, and the same agent adapts in real time. This dropped P95 response time from around 4 seconds to under 2.
3. Nothing waits for anything.
STT streams interim results while the caller is still speaking, the LLM starts generating before the turn is confirmed, and TTS synthesizes sentence one while the LLM writes sentence three. Pipeline latency approaches the maximum of the three stages, not their sum. P95 matters more than average: if 5% of callers wait 6 seconds, the product is broken.
4. Double-booking prevented at the database, not the prompt.
A UNIQUE constraint on slots plus an availability check in the service layer. The model can be wrong; the database cannot.
5. Tool-gated answers, streamed to the screen.
The agent never states availability from memory: every fact is a tool call against real database rows. Each tool publishes structured JSON status over the same WebRTC data channel as the audio, so the frontend shows what the agent is doing while it does it.
6. Provider-agnostic by design.
The LLM sits behind a factory: three environment variables switch providers. This came from a real bottleneck, ten concurrent calls on one API key made P95 jump to 8 seconds. A rolling context window keeps every prompt under 8K tokens.
What I measured
P95 response under 2 seconds, down from about 4
7 database-gated tools
rolling context under 8K tokens
false-interruption recovery