Speech To Speech: Open-Source Realtime Voice Agent
Speech To Speech is Hugging Face's framework for chaining VAD, speech-to-text, an LLM, and text-to-speech into one low-latency voice agent that speaks the OpenAI Realtime WebSocket protocol. Reach for it if you want a fully local, self-hosted pipeline built from swappable pieces; skip it if you want a five-minute demo without picking a CUDA wheel for the TTS backend.
What is the Speech To Speech Pipeline?
An open-source voice-agent pipeline, Speech To Speech, is constructed from four swappable stages: voice activity detection, speech-to-text, a language model, and text-to-speech. Each of these components operates within its own dedicated thread, and queues manage their interconnections. This entire cascade is exposed via an OpenAI Realtime-compatible WebSocket API, allowing clients developed for OpenAI's own Realtime protocol to connect without needing custom transport code.
Key Capabilities
- ✓VAD -> STT -> LLM -> TTS cascade where each stage runs in its own thread and hands off audio and text through queues, with Silero VAD v5 handling turn-taking.
- ✓OpenAI Realtime-compatible WebSocket API at /v1/realtime, supporting both WebSocket and WebRTC transport so existing Realtime clients connect unmodified.
- ✓Six STT backends to pick from with --stt, including Parakeet TDT (default), Whisper, Faster Whisper, Lightning Whisper MLX, MLX Audio Whisper, and Paraformer.
- ✓Five TTS backends with --tts: Qwen3-TTS (default), Kokoro-82M, Pocket TTS, ChatTTS, and MMS TTS, with different defaults on macOS versus Linux/CUDA.
- ✓LLM slot speaks OpenAI's responses-api or chat-completions protocol, so it can target OpenAI, HF Inference Providers, OpenRouter, or a self-hosted vLLM/llama.cpp server.
- ✓Four run modes — realtime, local (mic and speakers), raw-websocket, and socket (TCP) — covering everything from a WebRTC client to a bare microphone/playback pair.
- ✓Optional --enable_llm_proxy exposes the configured LLM in the capacity of a plain OpenAI-compatible endpoint, thereby enabling a client to run summaries or background agents concurrently with the live voice session.
- ✓Direct audio input mode (--stt none --llm_backend chat-completions) skips STT entirely and streams VAD-segmented audio straight to an audio-input model.
Typical Applications
- •Running a voice agent fully offline by pairing local STT and TTS with an LLM served through llama.cpp or vLLM on your own machine.
- •Adding a conversational voice layer to a robot or embedded device — the README says this pipeline is the production conversation backend for Reachy Mini robots.
- •Prototyping against the OpenAI Realtime protocol without paying for OpenAI's own hosted Realtime model.
- •Running background LLM tasks, like summaries or titles, concurrently with a live voice session through the LLM proxy endpoint.
- •Benchmarking TTS backend latency and quality locally with the included benchmark_tts.py script before picking a default.
Installation Guide
Speech To Speech needs Python 3.10 or newer. The base install is `pip install speech-to-speech`, which pulls in Parakeet TDT for STT, the OpenAI-compatible API path for the LLM, and Qwen3-TTS for speech output — GGML on non-macOS platforms, mlx-audio on Apple Silicon, resolved automatically through platform markers in pyproject.toml. On Linux, the Qwen3-TTS GGML backend's default wheel targets CUDA 12.8; if your CUDA runtime doesn't match, install a specific `qwentts-cpp-python` wheel (CUDA 13.x, CUDA 12.4, or CPU-only) from the Hugging Face wheelhouse before the main package. Optional backends come as pip extras: `[kokoro]`, `[pocket]`, `[chattts]`, `[faster-whisper]`, `[whisper-mlx]`, `[paraformer]`, and `[mlx-lm]`. To work from source: `git clone https://github.com/huggingface/speech-to-speech.git`, `cd speech-to-speech`, `uv sync` — that installs the package in editable mode and puts the `speech-to-speech` CLI on your path.
Running the Voice Agent
The quickstart is three commands: set `OPENAI_API_KEY`, run `speech-to-speech`, and it starts an OpenAI Realtime-compatible server at `ws://localhost:8765/v1/realtime` using Parakeet TDT for STT and Qwen3-TTS for output. That's the whole quickstart. From a source checkout, talk to it with `python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765` in a second terminal. Prefer to keep the LLM local? Serve Gemma 4 with `llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full`, then point `speech-to-speech` at it with `--model_name` and `--responses_api_base_url`. On a Mac, `speech-to-speech --local_mac_optimal_settings` switches to MPS, MLX LM, and mlx-audio TTS in one flag. `--mode raw-websocket` gives you a minimal client target (16 kHz, int16, mono PCM), and `--mode socket` streams raw PCM over plain TCP with no interruption handling or transcript events. `docker compose up` (with the NVIDIA Container Toolkit installed) brings up a llama.cpp server plus the TCP socket server on ports 8080, 12345, and 12346.
Strengths
- ✓Every stage — VAD, STT, LLM, TTS — is swapped with a CLI flag instead of forked code, so testing a different backend is a one-line change.
- ✓Speaking the actual OpenAI Realtime protocol means SDKs and clients built for OpenAI's voice API connect to a self-hosted server without a custom transport layer.
- ✓The LLM slot is backend-agnostic: point it at a hosted provider, HF Inference Providers, OpenRouter, or a local vLLM/llama.cpp server with the same two flags.
- ✓The Docker Compose setup pairs a llama.cpp server with the TCP socket mode out of the box, giving you a working fully-local reference to copy from.
- ✓Apache-2.0 licensing keeps commercial use and forking straightforward.
Known Considerations
- △The realtime server and its optional LLM proxy ship with no authentication and no rate limiting of their own — the README is explicit that you need a trusted network or a gateway in front before exposing either.
- △The default Qwen3-TTS GGML wheel on Linux is pinned to CUDA 12.8; anything else means manually finding and installing a matching wheel from a separate Hugging Face dataset before the main package will even import.
- △TCP socket mode is deliberately bare: no interruption handling, no live transcript events, and no tool-call events, so it's not a substitute for realtime mode if your client needs those.
- △DeepFilterNet's audio-enhancement option needs numpy<2, which conflicts with Pocket TTS's numpy>=2 requirement — you can't have both installed in the same environment.
- △Older implementations like MeloTTS have been moved to `archive/` and are no longer wired into the CLI, so a project that depended on them means switching backends to upgrade.
Alternative Voice Agent Frameworks
Frequently Asked Questions
Speech To Speech runs four stages — voice activity detection, speech-to-text, a language model, and text-to-speech — each in its own thread, connected by queues, forming a VAD-STT-LLM-TTS cascade exposed over an OpenAI Realtime-compatible WebSocket API.
Speech To Speech supports two OpenAI-compatible API modes, responses-api and chat-completions, that can target OpenAI, HF Inference Providers, OpenRouter, or a self-hosted vLLM or llama.cpp server, plus in-process local inference, which uses Transformers for CUDA/CPU and mlx-lm for Apple Silicon.
Speech To Speech can run fully local by combining local STT (Parakeet TDT) and local TTS (Qwen3-TTS or Kokoro-82M) with an LLM served through llama.cpp, vLLM, Transformers, or mlx-lm on Apple Silicon, without any external API calls.
Speech To Speech implements the core OpenAI Realtime event set in both directions — input_audio_buffer.append, session.update, conversation.item.create, response.create, and response.cancel inbound, plus speech events, streaming transcription, audio deltas, and response.done outbound — so OpenAI Realtime clients connect to it directly.
Speech To Speech needs Python 3.10 or newer. The default Qwen3-TTS GGML backend targets CUDA 12.8 on Linux, with separate wheels available for CUDA 13.x, CUDA 12.4, or a CPU-only build, while macOS runs through mlx-audio on Apple Silicon.
Speech To Speech is released under the Apache-2.0 license, a permissive license that allows commercial use, modification, and redistribution.
The problem it solves
Wiring voice activity detection, speech-to-text, an LLM, and text-to-speech into a working real-time conversation loop usually means gluing together four separate open-source or commercial pipelines yourself, then handling the threading and queueing so a slow LLM pass doesn't stall audio playback. Speech To Speech pre-builds that cascade and puts a standard protocol — the OpenAI Realtime API — in front of it, so a client already written for OpenAI's own voice API works against it without a custom transport layer.
Who should try it — and who should skip
Try Speech To Speech if you're building a Python-based voice interface and want the STT, LLM, and TTS layers to stay swappable — especially if the plan is to serve the LLM yourself through vLLM or llama.cpp instead of paying per token to a hosted API. Skip it if you want a managed, zero-setup voice API: you're still on the hook for picking backends, matching CUDA wheel versions on Linux, and standing up the WebSocket transport your client will talk to.
Related repositories
Want a second opinion on speech-to-speech?
Ask an AI that can read this page — one click and you get its take on speech-to-speech.