STT Module

Overview#

Transcribes an audio file to text. It is used in three places, all of them entrypoints where audio arrives already decoded as WAV:

src/modules/STT/
├── faster_whisper_stt.py  Whisper on this machine
├── groq_stt.py            Groq Whisper
├── openrouter_stt.py      OpenRouter Whisper
└── openai_compat_stt.py   any OpenAI-compatible transcription server

The provider is chosen by stt_provider in config and instantiated in src/cli.py. It is optional: with stt_provider set to anything else, stt is None and the audio paths degrade rather than crash.

Interface#

class STTInterface(ABC):
    def transcribe(audio_path: str, language: Optional[str] = None) -> str
    def reload_config(config: BrainConfig) -> None

Returns the transcript, or an empty string on failure. Both methods are @abstractmethod — omitting either raises TypeError at instantiation.

language falls back to config.stt_language (Settings → Hearing → Language), resolved through src/core/language.py before it reaches any provider: jp, it-IT and Italiano all mean the same thing on all three, and auto becomes a missing field rather than a literal.

A pin is only worth setting when the language is known in advance. A wrong one is worse than none: whisper transcribes into the language it was told, so Italian pinned to ja comes back as Japanese and Italian pinned to en comes back as English. No error is raised, so build_stt() logs the effective language at startup (src/modules/STT/factory.py):

STT backend: faster_whisper, listening in Italian (it)
STT backend: faster_whisper, listening in whatever language it hears

The default is auto. Turns shorter than MIN_DETECT_SECONDS reuse the last reliable detection — see Languages. On a call in one known language other than English, pin it: detection misplaces a quarter of Italian turns even on small — see Pin or detect.

Local Whisper (faster_whisper_stt.py)#

Whisper through faster-whisper, running on the machine she runs on. No key, no account, no per-minute bill, and the audio never leaves the room — which is the whole reason to prefer it for a Discord call with people who did not agree to be sent anywhere.

• Config: stt_provider: "faster_whisper", stt_model (default small)
• Key: none

The weights are downloaded on first use, not at install time:

The same model costs very different amounts on different machines. Every turn pays for a full 30-second encoder window however short it is, so the floor is set by the CPU, not by the turn: on an Apple M5, base takes ~0.5s a turn and small ~1.1s, and an older x86 laptop is several times that. Turns that arrive together queue behind each other on the one model.

cpu_threads is the number of physical cores (src/core/perf.py). More threads than cores is not free: on 10 cores, 20 threads made small 3x slower at the median and 4.5x at the worst turn.

tools/stt_bench.py measures all of this on the machine it runs on — models, pinned against detected, noise, thread counts:

uv run python tools/stt_bench.py --models base,small --lang it --threads 4,8

Any faster-whisper model on Hugging Face works too — a repo id with a slash in it (Systran/faster-distil-whisper-large-v3) is passed through untouched.

The weights come from a public repo: no account, no key. uv run bea --setup offers to download them at the end rather than leaving the wait to her first sentence, and a refusal — a shared IP against Hugging Face's anonymous rate limit — is logged with what would fix it, HF_TOKEN in .env.

Degenerate decodes. faster-whisper retries a decode that trips compression_ratio_threshold at increasing temperatures, so this provider passes a ladder — TEMPERATURES = (0.0, 0.2, 0.4, 0.6, 0.8, 1.0) (src/modules/STT/faster_whisper_stt.py). A single 0.0 would leave the retry loop with nowhere to go. On Groq/OpenRouter, temperature=0.0 already requests the server-side equivalent.

What it normalises, so the same config.json works on every provider:

• stt_model. The hosted spelling whisper-large-v3-turbo, or openai/whisper-large-v3-turbo, becomes large-v3-turbo — switching stt_provider does not also mean editing the model.
• language. Resolved by core/language.py for every provider alike, then checked here against the languages this build of whisper actually has. One it does not have falls back to auto-detection with a warning, rather than raising and taking the transcript down to an empty string.

Nothing else about her becomes local by choosing this. The mind is still a hosted model, and it is still sent what she heard.

When the device will not run#

A GPU ctranslate2 can see is not one it can use: on Windows the CUDA Toolkit DLLs do not come with the pip wheels, so the model loads and every transcription fails. Instead of staying deaf, she probes the device once at startup with a silent second — if it cannot hear, she rebuilds on cpu/int8 before anyone speaks and says so loudly in the log, with what to install for your OS. If a device breaks mid-call, the current turn is retried on CPU rather than dropped.

/status reports the honest state (stt.device, stt.degraded, stt.last_error), and the doctor's ears check names the fix.

Groq (groq_stt.py)#

Groq's Whisper endpoint. Fast enough for near-realtime.

• Config: stt_provider: "groq", stt_model (default whisper-large-v3-turbo)
• Key: GROQ_API_KEY env → config.json → None

The connection to Groq is kept for KEEPALIVE_SECONDS (90) between turns. The SDK's own default is five seconds, shorter than most pauses in a call, and a new connection is a handshake in front of every transcription.

OpenRouter (openrouter_stt.py)#

The same Whisper models through OpenRouter, useful when you already have a key there and would rather not add a Groq account.

• Config: stt_provider: "openrouter", stt_model
• Key: OPENROUTER_API_KEY

Model ids are namespaced here, so a bare whisper-large-v3-turbo is rewritten to openai/whisper-large-v3-turbo on both load and hot reload — an id copied from the Groq config keeps working.

Requests go through one requests.Session, so the connection is kept between turns. A request that finds the kept connection closed by the far end is sent once more; a timeout, a refused or unresolvable host and a bad certificate are not.

Custom OpenAI endpoint (openai_compat_stt.py)#

Any server speaking OpenAI's /audio/transcriptions: speaches, LocalAI, a whisper.cpp server, vLLM, or OpenAI itself.

• Config: stt_provider: "openai_compat", stt_compat_base_url (up to and including /v1), stt_model (default whisper-1)
• Key: STT_COMPAT_API_KEY, optional. Its own, never the custom LLM endpoint's: a transcription server is rarely the one that thinks

The audio goes up as multipart, the way the OpenAI api takes it. It asks for verbose_json, which says which language was heard and so feeds the short-turn language memory; a server that answers 400 to it is asked for plain json from then on, and asked again only when the endpoint changes. The connection is kept and a dead kept connection is retried once, as on OpenRouter. /status reports stt.degraded and stt.last_error.

Warming the connection#

Every hosted transcriber (Groq, OpenRouter, custom) has a warm(). When somebody in a call starts talking, the bot's hearing: start reaches Consciousness, which calls it on a thread: the socket is opened while they are still speaking, and their turn skips the handshake — ~200 ms measured on Groq. It runs at most once every 20 s (src/utils/warm.py), well inside the keepalive, and a failure is left for the real request to find. Local Whisper has nothing to open and no warm().

The Discord path#

The bot decodes Opus to PCM in Node (prism-media), resamples 48 kHz stereo to 16 kHz mono (Pcm.js) and posts WAV files. Python never handles raw Opus. Turn segmentation happens in the bot — see Discord.

Each turn is transcribed and deposited on the perception bus as its own VOICE perception, unless dropped by the echo guard (see Discord). The bus coalesces a burst into a single batch, and the attention gate decides what deserves a reasoning cycle — so two people talking at once become one batch, one turn, one answer.

Hot reload#

reload_config() updates the model and re-creates the client if the key changed. Switching stt_provider itself needs a restart — the object type changes.

Local Whisper compares the whole set — model, device, precision, cache — and rebuilds only when one of them actually moved. Loading it is seconds and possibly a download, and an unrelated save from the dashboard must not pay for that.

Discord Skill →