Avatar Module

Overview#

Two ports decide how Bea reaches the screen. Expression calls them with a mood and a state and never learns which backend is behind either.

src/modules/avatar/                          src/modules/caption/
├── factory.py   build_avatar()              ├── factory.py   build_caption()
├── png.py       PngAvatar                   ├── obs_text.py  ObsTextCaption
├── model3d.py   Model3DAvatar               ├── stage.py     StageCaption
└── vtube_studio.py  VTubeStudioAvatar       └── silent.py    SilentCaption

They are chosen independently: any avatar backend works with any caption backend.

AvatarInterface#

class AvatarInterface(ABC):
    def show(self, mood: str, state: str) -> None: ...
    def perform(self, clip: str) -> None: ...
    def mouth(self, envelope: Sequence[float], fps: int) -> None: ...
    def reload_config(self, config) -> None: ...
    def close(self) -> None: ...

The state she rests in. talking is the only state the speech path chooses for itself. The others are set by whoever knows about them — sleeping by the consciousness, listening by the voice surface while she sits in a call — and Expression remembers the last one: a line ends by going back to it, not to a hardcoded idle. A state that arrives mid-line is recorded and applied when she stops talking, so someone walking into the call cannot take her talking face off her halfway through a word.

Three rules a backend has to hold to:

• Every method is synchronous. A backend that needs the network queues the work. Blocking here blocks the turn she is speaking in.
• mouth receives the whole utterance. Do not expect a call per frame.
• Ignore what you cannot do. perform and mouth are no-ops on PngAvatar. Callers never branch on the backend.

CaptionInterface is say(text) and clear(). say is awaitable because the OBS backend animates over time and barge-in cancels it mid-sentence.

Configuration#

Everything lives in the stage block. Dashboard → Settings → Stream writes the same keys.

"stage": {
  "avatar_backend": "png",
  "caption_backend": "obs",
  "lipsync_fps": 30
}

Both are hot-reloadable. AIVtuberBrain._reload_stage rebuilds only the port whose backend changed, so switching the caption does not drop a loaded model or an authenticated socket.

png#

Swaps a file in an OBS image or media source. obs_source_type decides which.

"avatar_map": {
  "angry":    { "idle": "data/pngs/angry/idle.png", "talking": "data/pngs/angry/talking.png" },
  "sleeping": { "idle": "data/pngs/sleeping.png",   "talking": "data/pngs/sleeping.png" }
}

Keys are moods, plus optionally the two non-speech states, sleeping and listening. Resolution order for show(mood, state):

1. avatar_map[state] if state is sleeping or listening and has an entry
2. avatar_map[mood], taking talking or idle
3. avatar_map["neutral"]
4. any entry at all

A missing state entry logs one warning per state, not per frame.

close() blanks the source. The image is the only thing this backend leaves behind, and another backend taking over mid-stream would otherwise be drawn on top of the face it replaced.

model#

Renders a VRM in a page served at /stage, which you add to OBS as a Browser Source. The backend publishes to the stage channel; the page owns three.js.

"stage": {
  "avatar_backend": "model",
  "model_path": "data/models/VRM1_Constraint_Twist_Sample.vrm",
  "clips_dir": "data/clips",
  "shot": "bust",
  "mood_clips": { "angry": "lean_in" },
  "background": ""
}

Why the format matters#

A plain .glb names bones however the artist chose — Bip01_L_UpperArm, mixamorig:LeftArm. VRM fixes the names: 54 humanoid bones and 18 expressions (happy angry sad relaxed surprised, the visemes aa ih ou ee oh, blinks, look direction). Every mapping in this module is written against those names, which is what makes it work on a model the code has never seen.

The same applies to behaviours. A .vrma drives bones by standard name and carries no mesh, material or texture — a three-second clip is about 11 KB, and it plays on any VRM.

Checking a model#

uv run python tools/inspect_vrm.py data/models/bea.vrm

Prints the rig, the expressions and the licence fields VRM stores inside the file. Use it to find out before going live whether a model is missing the aa viseme (her mouth cannot move) or the emotion presets (one face for every mood). It exits non-zero when something required is absent.

make model downloads pixiv's VRM 1.0 sample and one clip into data/models and data/clips, both gitignored. Both are pinned to the three-vrm release the dashboard itself is built against and checked against a SHA-256: what arrives is loaded and run by a renderer, so anything that does not match is deleted rather than kept. A file already at that path is never overwritten — it is yours.

Notes for the renderer#

src/web/frontend/src/stage/avatar.js:

• The Live Loop: src/web/frontend/src/stage/life.js manages procedural animations. It adds natural breathing, blinking, and subtle look-around motions, so the avatar looks alive even when the engine is not actively sending expressions.
• Turning the model 180° is correct for VRM 0.x only. VRMUtils.rotateVRM0 applies it conditionally; an unconditional rotation.y = Math.PI faces every VRM 1.0 model away from the camera.
• Camera framing reads the head and hips world positions and solves the distance from the FOV. Hardcoding a position ties the shot to one model's height.
• The page is a separate Vite entry (stage.html). three.js is behind a dynamic import, so it is fetched only when avatar_backend is model. Importing from src/stage/ anywhere in the dashboard would pull it into the main bundle.

vtube_studio#

Drives a VTube Studio instance over its plugin API. Nothing is bundled; the model and the software are the user's.

"stage": {
  "avatar_backend": "vtube_studio",
  "vts_host": "127.0.0.1",
  "vts_port": 8001,
  "vts_expressions": { "angry": "furious.exp3.json" },
  "vts_clips": { "happy": "hotkey-id-or-name" },
  "vts_mouth_param": "MouthOpen"
}

Authentication. AuthenticationTokenRequest prompts the user inside VTube Studio; the token is written to data/vtube_studio_token.json and reused with AuthenticationRequest on later sessions. A token that stops authenticating is deleted so the next attempt asks again instead of failing forever. It is kept out of config.json because GET /config is unauthenticated.

Threading. Every request holds a lock for its send and its reply: a socket is one send/recv pair, and the mouth writes to it 30 times a second while the worker may be setting an expression. _run owns one connection and drains a bounded queue over it, reconnecting with backoff from 3s to 30s. When the queue fills, the oldest command is dropped — the newest face is the correct one. The mouth runs in its own task so a new line cancels the previous one.

A behaviour per mood. vts_clips is keyed by mood, exactly like mood_clips on the 3D backend, and fires when she starts talking rather than on every change of face. perform resolves through the same map, so a hotkey id passed straight in still works.

Two behaviours specific to this backend. VTube Studio holds an expression until told otherwise, so the adapter deactivates the previous one before activating the next. And it has no clock shared with the engine, so this is the only backend that paces the envelope itself.

Discovery. Expression files and hotkey ids are defined by whoever rigged the model. GET /vts/model returns both from the connected instance so the dashboard can offer a list.

Semantic Picker#

The moods and clips are no longer hardcoded strict matches. A semantic picker (src/core/expression/picker.py) uses a small local embedding model to map whatever expression she names to the closest valid mood or installed clip. This means if she decides to write <do:shrug> and the file is named 003_dismissive_wave.vrma, the picker understands the meaning and plays the right clip, rather than failing silently. It is extremely fast, cached in memory, and prevents her natural vocabulary from breaking the visual output.

Mood → expression#

src/core/expression/face.py. weights_for(mood) returns a weight for every VRM emotion, zeros included, so a face is never left wearing part of the last mood.

VRM standardises five emotions; MOODS has seven. disgusted and bored are blends.

tests/test_face.py asserts that the weights agree with the valence/arousal vectors in moods.py: a mood below -0.2 valence must weigh sad/angry above happy/relaxed, and the reverse above +0.2. This catches the easy mistake with relaxed, which means at ease and reads as contentment — weighting bored toward it produces a face that looks pleased rather than unbothered.

The table is also checked against the sample model, when it is present, so a weight cannot name an expression no real VRM has.

Lip sync#

envelope(audio, sample_rate, fps) in src/core/expression/pcm.py returns the per-frame RMS of the audio, normalised to [0, 1] and rounded to three decimals.

The line is normalised against its own peak, but never against a peak below MOUTH_FLOOR_RMS (0.05). Dividing by the peak alone makes every line as loud as every other, so a whisper is drawn exactly like a shout; the floor keeps a quiet line quiet while any normal one still opens her mouth all the way.

It is computed in Expression._speak_local between synthesis and playback, and on the call route it is accumulated per sentence as each one is synthesised. It runs on this machine because the audio is played here; the page never hears it.

Cost for six seconds at 30 fps: 180 frames, ~0.2 ms, ~1.2 KB of JSON. Small enough to send in one message ahead of playback rather than streaming it.

Expression._move_mouth catches and logs any failure. A broken mouth must not stop her from speaking.

The stage channel#

src/core/stage.py. A fan-out to however many browser sources are open, plus the current state.

channel.publish({"mood": "angry", "state": "talking"})   # merged into the state
channel.publish({"envelope": [...], "envelope_fps": 30}) # sent, not stored
channel.snapshot()                                       # what a new page receives

Keys in TRANSIENT (envelope, perform) describe a moment and are fanned out without being stored. A page connecting mid-utterance is not made to mouth a sentence that has already ended.

There is no backlog. OBS reloads a browser source whenever it is toggled, and a subscriber that received a replay would act the last minute out again. A queue that fills is dropped rather than awaited.

Saving settings publishes {"config": public_config(...)}, so a running page picks up a new shot or background without being reloaded by hand. The page reloads itself only when the change is structural — a different backend, or a different model, which model_id (the file name and its mtime) detects. That same push is what keeps the preview in the dashboard current.

None of these are authenticated, so none of them return a secret.

Adding a backend#

1. Implement AvatarInterface in src/modules/avatar/.
2. Add a builder to BUILDERS in factory.py. Keep the import inside the builder so other backends do not pay for its dependencies.
3. If it needs a publisher, add its name to NEEDS_PUBLISHER. Without one the factory falls back to png rather than publishing into nothing.
4. Add it to AVATARS in src/setup/wizard.py.

tests/test_avatar_backends.py iterates BUILDERS and asserts each one builds an AvatarInterface; tests/test_setup.py asserts the wizard never offers a name the factory does not have.