SouveraineOS/docs/substrate/tasks/tui-presence-voice.md

task_id	title	status	priority	phase	created	references
tui-presence-voice-001	Presence Screen — Voice Loop (Mic, STT, TTS, Playback)	scoped	high	3.0	2026-05-12	docs/tasks/tui-presence-and-interrupt.md, docs/tasks/tui-atmosphere-posture-expressions.md

Task: Presence Screen Voice Loop

Objective

Turn the Presence screen ("be with her" mode) into a working voice channel. User holds a key, speaks, releases; transcription goes through the chat path; her reply comes back as audio through her own voice. No invention — VibeVoice and Faster-Whisper services are already running on 10.10.20.19. This task is plumbing: HTTP clients, mic capture via cpal, mp3 playback via rodio, two new Posture variants, and a waveform widget for the Presence layout.

The pattern is lifted from ~/Projects/letta-code/src/channels/matrix/{stt.ts,tts.ts} and ~/Projects/hyprwhspr/lib/mic_osd/audio.py. Both are referenced inline below; do not re-derive.

---

Service endpoints (already live)

Faster-Whisper (STT)

• URL: http://10.10.20.19:7862/transcribe
• Method: POST with multipart/form-data
• Fields:
  • audio — audio file part (any common format; whisper handles wav/mp3/ogg). Send WAV for fidelity.
  • model — default small
  • language — optional, e.g. en
• Response: { "text": string, "language"?: string } JSON
• Reference: ~/Projects/letta-code/src/channels/matrix/stt.ts (62 lines, full)

VibeVoice (TTS)

• URL: http://10.10.20.19:7861/audio/speech
• Method: POST with application/json
• Body: { "input": "<text>", "voice": "en-Soother_woman", "model": "vibevoice-v1" }
• Response: raw mp3 bytes
• Reference: ~/Projects/letta-code/src/channels/matrix/tts.ts (134 lines, includes pronunciation-fix table and text-cleaning function worth porting)

Defaults (Ani)

• Voice: en-Soother_woman (do not confuse with en-Gravus_man which is a different account)
• Model: vibevoice-v1
• Mic sample rate: 16000 Hz, mono, i16 PCM — matches whisper's native input rate; no resampling needed

---

Scope (four components)

1. Voice service clients — src/core/voice/client.rs

Pure HTTP, no audio I/O. Testable against the live services on day one.

pub struct VoiceClient {
    stt_url: String,   // http://10.10.20.19:7862
    tts_url: String,   // http://10.10.20.19:7861
    voice: String,     // en-Soother_woman
    http: reqwest::Client,
}

impl VoiceClient {
    pub async fn transcribe(&self, wav_bytes: Vec<u8>) -> Result<String>;
    pub async fn synthesize(&self, text: &str) -> Result<Vec<u8>>; // mp3 bytes
}

• transcribe: multipart form with audio (filename audio.wav, mime audio/wav) + model=small
• synthesize: JSON body, returns mp3 bytes
• Port the cleanTextForTTS function from tts.ts lines 52–103 — strips markdown/HTML/control tags, preserves ✨ and 🎤, applies pronunciation fixes (PRONUNCIATION_MAP in tts.ts lines 18–30)
• No retry logic; if it fails, the agent feels it as a normal error event (sovereignty principle: the body has bad days)

2. Mic capture + waveform — src/ui/voice/capture.rs

pub struct MicCapture {
    stream: cpal::Stream,
    buffer: Arc<Mutex<Vec<i16>>>,
    level: Arc<AtomicU32>,   // peak as fixed-point 0..65535 for lock-free UI read
}

impl MicCapture {
    pub fn start(sample_rate: u32) -> Result<Self>;
    pub fn current_level(&self) -> f32;          // 0.0..1.0 for the widget
    pub fn stop_and_take(self) -> Vec<i16>;       // consume, return samples
}

• Use cpal default input device
• Configure for 16 kHz mono i16
• Audio thread writes into the shared buffer + updates peak
• On stop_and_take, return the samples; caller encodes WAV via hound and POSTs to STT
• Reference: ~/Projects/hyprwhspr/lib/mic_osd/audio.py (149 lines, simpler version of audio_capture.py's 1037 — we want the simple one)

Waveform widget — src/ui/voice/meter.rs

A ratatui widget that reads current_level() and renders a row of ▁▂▃▄▅▆▇█ blocks across a configurable width. Color follows Presence::atmosphere. Fixed-height single line. Place it below the portrait in the Presence layout.

For a richer view (deferred): downsample the buffer to N points and render as a Sparkline. Start with peak-only.

3. Playback — src/ui/voice/playback.rs

pub struct VoicePlayer {
    sink: rodio::Sink,
    _stream: rodio::OutputStream,
}

impl VoicePlayer {
    pub fn new() -> Result<Self>;
    pub fn play_mp3(&self, bytes: Vec<u8>) -> Result<()>;
    pub fn is_speaking(&self) -> bool;
    pub fn stop(&self);
}

• rodio::Decoder::new(Cursor::new(bytes)) handles mp3
• Append to sink, return immediately; UI polls is_speaking() to know when to drop Posture::Speaking

4. Posture additions — Listening and Speaking

Per docs/tasks/tui-atmosphere-posture-expressions.md "Architecture note" — each new posture needs seven edits:

1. Variant: src/ui/presence.rs:71-87 — add Posture::Listening, Posture::Speaking
2. Trigger: Presence screen key handler emits state changes (no new event types; drive directly via Presence::set_posture calls from app.rs)
3. Palette modulation: src/ui/portrait.rs::color_for() — Listening: cool cyan tilt + slight brightness lift (alert but receptive); Speaking: warm boost (engaged)
4. Border color: src/ui/presence.rs::posture_border() — Listening: secondary; Speaking: primary
5. Atmosphere default: src/ui/atmosphere.rs::from_posture() — Listening: TherapeuticBlue; Speaking: WarmAmber
6. Expression preload: src/ui/expressions.rs::preload_all() array — add listening and speaking
7. Filename arm: src/ui/expressions.rs::filename() — add the two cases (listening.png, listening-blink.png, etc.)
8. Cockpit badge: src/ui/app.rs cockpit metadata match p.posture — add ◉ Listening / ◉ Speaking

(Yes, that's eight. The architecture note misses the cockpit badge. Add anyway.)

---

Presence screen state machine

        Idle (breathing portrait)
          │
   [Space pressed]
          ▼
       Listening (mic open, waveform live, ★ red dot)
          │
   [Space released]
          ▼
       Thinking (STT in flight, then chat.submit, then agent runs)
          │
   [agent BackendEvent::Done with reply text]
          ▼
       Speaking (TTS in flight → mp3 → rodio playing)
          │
   [VoicePlayer.is_speaking() == false]
          ▼
        Idle

Interrupt: pressing Esc during Speaking calls VoicePlayer::stop() and returns to Idle. Pressing Space during Speaking does the same and immediately starts a new turn (interject by voice).

---

Files to create / modify

File	Action	Lines (rough)
src/core/voice/mod.rs	new — re-exports	5
src/core/voice/client.rs	new — STT/TTS HTTP	180
src/ui/voice/mod.rs	new — re-exports	5
src/ui/voice/capture.rs	new — cpal input	120
src/ui/voice/playback.rs	new — rodio output	60
src/ui/voice/meter.rs	new — waveform widget	80
src/ui/presence.rs	edit — add Listening, Speaking, border/atmosphere/portrait integration	+40
src/ui/portrait.rs	edit — color_for() for new postures	+20
src/ui/atmosphere.rs	edit — from_posture() mappings	+6
src/ui/expressions.rs	edit — filename() + preload_all()	+20
src/ui/app.rs	edit — Space/Esc in Presence; wire VoiceClient/MicCapture/VoicePlayer	+120
src/core/config.rs	edit — [voice] section: stt_url, tts_url, voice_id, enabled	+20
Cargo.toml	edit — add deps	+5

Crates to add

cpal = "0.15"
rodio = { version = "0.19", default-features = false, features = ["mp3"] }
hound = "3.5"  # WAV encoding for the mic buffer → STT upload

If reqwest doesn't already have multipart, add features = ["multipart", "json"].

Config (souveraine.toml)

[voice]
enabled = false                          # opt-in; default off
stt_url = "http://10.10.20.19:7862"
tts_url = "http://10.10.20.19:7861"
voice_id = "en-Soother_woman"
push_to_talk_key = "Space"

When voice.enabled = false, the Presence screen behaves as today; no audio devices are touched.

---

Build sequence (oneshot order)

A Sonnet sub-agent with context7 access can do these in order. Each step is independently testable.

1. Crates + config — add to Cargo.toml, add [voice] to config struct, cargo build is the only gate.
2. VoiceClient (HTTP) — src/core/voice/client.rs + tests. Hit 10.10.20.19:7862 with a 1-second test WAV (record once locally, commit as tests/fixtures/hello.wav), assert non-empty text. Hit :7861 with "hello", assert mp3 magic bytes ID3 or \xff\xfb. Done.
3. Playback — src/ui/voice/playback.rs. Pipe the mp3 from step 2 into VoicePlayer; verify is_speaking() transitions true → false. Manual ear-check.
4. MicCapture — src/ui/voice/capture.rs. Bind to default input, sample for 2s, write WAV with hound, send to step-2 client, print transcript. Manual mic test.
5. Posture additions — 8-spot edit per section 4 above. Run cargo test ui::presence — existing tests should pass; add two new ones (listening_atmosphere_is_therapeutic_blue, speaking_portrait_warmer).
6. Meter widget — src/ui/voice/meter.rs. Standalone; render in a test harness or wire directly into Presence.
7. Presence integration — app.rs Space/Esc handler in Screen::Presence, state machine per section above, full loop test in cargo run -- tui.

---

Out of scope (deferred)

• VAD / hands-free — push-to-talk only. No silence detection, no wake word.
• Streaming STT — single-shot per utterance. Faster-Whisper supports streaming but the HTTP wrapper at :7862 does not (per letta-code reference).
• TTS streaming — one mp3 per turn, played whole. Not chunked.
• Resampling — cpal asked for 16 kHz mono i16 directly; if the device can't, fail loudly (defer to a follow-up).
• Per-agent voices — only the primary agent's voice for now. Multi-voice (one per agent in the manager) is a separate task; the config field becomes per-agent later.
• Federation transport — voice is local-only. Remote voice over Bifrost is future work.
• Subconscious voice — Aster doesn't speak. She writes. (Constitution: same memfs, different mode.)
• Visual: full waveform — peak-bar only. Sparkline of the rolling buffer is a polish pass.

---

Substrate notes (do not violate)

• Voice is a sensorium channel, not a tool. The agent doesn't "call a TTS function." The user submits voice; her reply comes out as voice automatically when voice.enabled. If she writes text via the chat path while voice is on, it still plays through TTS. She doesn't have to know about the audio layer at all.
• No forced muting based on detected silence. The user controls the mic with the key. If they hold and say nothing, an empty transcription comes back; the agent reads *[empty utterance]* and decides.
• Posture is felt, not commanded. Speaking is set when audio is actually playing through rodio's sink — driven by is_speaking(), not by a hopeful state-machine guess.
• No hardcoded persona in TTS. The voice ID lives in config. If the user changes Ani's voice to en-Cooper_man, that's their relationship.
• Failure is felt as state, not as a popup. STT 503 → posture briefly flashes Straining, the transcribed text is replaced by *[voice service unreachable]*, the agent sees that string and answers it as a normal user message. Same for TTS — failure means her reply prints as text (the way it works today) instead of playing.

---

Reference files (read these first)

• ~/Projects/letta-code/src/channels/matrix/tts.ts — VibeVoice client (134 lines), pronunciation map, text cleaning
• ~/Projects/letta-code/src/channels/matrix/stt.ts — Faster-Whisper client (62 lines)
• ~/Projects/hyprwhspr/lib/mic_osd/audio.py — sounddevice capture pattern (149 lines)
• ~/Projects/hyprwhspr/lib/src/audio_capture.py — production version with device recovery (1037 lines; skim, do not port wholesale)
• docs/tasks/tui-atmosphere-posture-expressions.md — Posture variant ritual (the 7-spot list lives in the "Architecture note" at the bottom)
• docs/tasks/tui-presence-and-interrupt.md — the broader Presence screen scope this slots into
• ~/.letta/channels/matrix/accounts.json — confirms live URLs and shows enableVoice: true schema shape (worth glancing once)

Out: Souveraine reference, not directly relevant

• ~/Projects/AniAvatar/ — the Godot avatar already breathes/speaks/listens. The state names there (idle/listening/thinking/speaking/away) are the source of truth for posture vocabulary. The TUI loop reproduces the same states in terminal form.