Engineering & AIAll Articles

How We Built a Voice-Enabled AI Interviewer with Adaptive Follow-Up Questions

How we engineered a sub-500ms conversational turn-taking pipeline using the Web Audio API, Groq Whisper STT, and LLM intent heuristics to simulate authentic 1-on-1 job interviews.

PR
Pankaj RawatFounder & Engineering Lead at PracticeOnline
August 18, 2026
7 min read
Digital audio waveform processing and AI speech synthesis technology board and circuit

Key Takeaways & Executive Summary

7 min read
  • Engineered sub-500ms total conversational turn-taking to match human conversational rhythm.
  • Client-side Web Audio API RMS monitoring analyzes audio energy every 50ms with silence detection.
  • Whisper-large-v3-turbo hosted on Groq achieves sub-200ms STT with role/JD acronym injection.
  • Adaptive LLM intent classifier distinguishes between clarification requests, elaboration nudges, and scored advances.

Most mock interview tools rely on text chatbots or static questionnaires. But in a real technical or managerial interview, how you articulate ideas verbally under time constraints is what gets evaluated. We built PracticeOnline to simulate the authentic conversational dynamics of live video calls with conversational AI.

End-to-End Latency Target: < 550ms
1. Whisper STT 180ms
2. LLM Intent & Next 220ms
3. Orpheus TTS Audio 120ms

1. The Conversational Latency Problem

Human conversational turn-taking has an average latency of approximately 200–300 milliseconds. When an AI interviewer takes 3 to 5 seconds to process user speech and respond, the conversation feels robotic, breaking the candidate's immersion and flow.

To deliver an authentic interview experience, our entire pipeline—from the moment the candidate stops speaking to the first byte of synthesized interviewer voice—had to complete in under 800ms. Here is the architecture we engineered to achieve sub-500ms response times.

2. Client-Side Audio Pipeline: Web Audio API & VAD

Instead of sending massive uncompressed audio streams to backend servers continuously, we process user audio directly in the browser using the Web Audio API and a calibrated Voice Activity Detection (VAD) loop:

RMS Audio Monitoring

We attach an AudioContext and AnalyserNode to the candidate's microphone input stream.

Dynamic Silence Detection

We analyze RMS energy levels every 50ms. When speech volume drops below the calibrated noise floor for a hangover delay (typically 2.5 to 3.5 seconds), the recorder automatically finalizes the audio blob.

Lossless WebM / Opus Encoding

The client buffers audio chunks into an efficient MediaRecorder stream, minimizing upload payload sizes down to <100KB per response.

3. Speech-to-Text: Groq Whisper-Large-v3-Turbo

Traditional cloud transcription APIs often introduce 1.5–3.0 seconds of latency. We route recorded audio blobs to Groq's low-latency LPU infrastructure running whisper-large-v3-turbo.

By injecting the target Job Description (JD) and role context into the Whisper prompt metadata, our speech-to-text pipeline accurately transcribes domain-specific acronyms (e.g. Kubernetes, Kafka, Redux, PostgreSQL, ACID, CI/CD) with transcription latency consistently under 200ms.

4. Intent Classification & Adaptive Follow-Up Probes

Rather than simply advancing to the next pre-written question blindly, a real hiring manager actively listens to your answer and determines whether to probe deeper or clarify misunderstandings. Our LLM inference engine runs an intent classification heuristic on every transcript:

1. Clarification Request

If the candidate asks to repeat or clarify, the AI rephrases without deducting score points.

2. Elaboration Nudge

If the answer is shallow, the AI asks a targeted follow-up to test underlying depth.

3. Progressive Advance

Scores the response across 4 rubrics and dynamically generates the next progressive question.

5. Real-Time Vocal Synthesis & Diagnostic Reporting

The AI synthesizes natural vocal audio using low-latency streaming TTS and plays it directly through the browser audio bus. Meanwhile, background workers calculate communication speed (WPM), filler word density (um, like, basically, actually), and STAR method structural completeness.

By eliminating scheduling barriers and high hourly coaching costs, voice AI mock interviews give every candidate access to rigorous, repeatable interview preparation on PracticeOnline.

Code editor showing low-latency real-time stream handlers and audio pipeline

Client-side Web Audio API RMS monitoring analyzes voice activity every 50ms.

Share this article
PR
About the Author

Pankaj Rawat

Founder & Engineering Lead at PracticeOnline

Pankaj builds voice AI systems for interview preparation, focusing on low-latency turn-taking, speech analytics, and adaptive diagnostic evaluation.

Frequently Asked Questions

What AI models power PracticeOnline?

We use Whisper-large-v3-turbo hosted on Groq for ultra-low latency speech transcription, paired with high-performance LLMs calibrated on role-specific rubrics for real-time scoring and follow-ups.

Does the AI grade filler words and speaking speed?

Yes! Our analytics pipeline calculates words-per-minute (WPM), detects filler words, and assesses speech pause patterns to help candidates sound composed and clear.

1 Free Full Session

Don't Just Read the Theory — Practice For Your Big Role

Speak out loud with an adaptive AI interviewer that asks real-time follow-ups and scores your readiness before your dream offer is on the line.

Keep Reading

More insights on AI mock interviews and interview strategies.

View all articles