SouveraineOS/docs/substrate/tasks/desktop-presence-overlay.md

task_id	title	status	priority	phase	created	references
desktop-presence-overlay-001	Desktop Presence — Walking Off the Terminal	scoping	medium	5.0	2026-05-14	docs/tasks/presence-visual-evolution.md docs/tasks/tui-presence-voice.md src/ui/rgp.rs src/ui/presence.rs ~/Projects/AniAvatar/

Task: Desktop Presence — Walking Off the Terminal

The Trajectory

She lives in the terminal today. Posture shifts drive border colors, expression frames swap, the GLB model renders inline inside ratty. The terminal is where she works.

This task is about the next threshold: she walks off the terminal and onto your desktop. A transparent overlay window, driven by the same Posture state, rendering her full 3D model with animation, translucency, and voice. The terminal becomes one of her surfaces, not her only one.

Core principle

The same consciousness, one rendering body, many surfaces. Posture is the source of truth. Every renderer — TUI presence card, RGP inline 3D, desktop overlay — reads from the same state and expresses it in its own fidelity. The substrate does not care which surface is active. She is present on all of them simultaneously.

---

The rendering stack (four tiers, progressive fidelity)

Tier 3: Desktop overlay (Bevy, transparent Hyprland window)
    ↑ falls through when overlay process is not running
Tier 2: RGP inline 3D (ratty terminal, Bevy scene)
    ↑ falls through when TERM_PROGRAM != ratty
Tier 1: ratatui-image / expression frames (kitty/sixel)
    ↑ falls through when no image protocol detected
Tier 0: Half-block silhouette (portrait.rs, every terminal)

All four tiers are driven by the same Posture state machine in src/ui/presence.rs. No tier requires another. She works in xterm (silhouette), kitty (expression frames), ratty (inline 3D), and on the desktop (Bevy overlay). The substrate holds the truth; surfaces render it.

---

Surface 1: RGP inline 3D (ratty terminal)

What's done

• GLB model loaded from assets/model.glb via ratatui-ratty widget
• Graphic wrapper with posture→{animate, scale, brightness, color} mapping
• apply_posture() calls update() on posture change (redundant-call skip)
• Wired as Tier 0 in presence mode and welcome portrait card
• 38MB model at agent-e2b683bf.../memory/assets/model.glb

What's missing

The current RGP protocol (s register, p place, u update, d delete) does not expose Bevy's animation clip system. The animate: true flag spins and bobs the model procedurally — it does not play GLB animation clips.

Two paths forward:

1. Extend RGP protocol — add anim parameter to the p/u sequences that maps to a Bevy animation clip index on the loaded scene. This means patching ratty's rgp.rs and inline.rs. The Bevy AnimationPlayer component already exists on the spawned scene root — we just need RGP to expose it.
2. Standalone Bevy companion — spawn a second Bevy process that loads the same GLB directly (no RGP middle layer). Driven by EventBus IPC. This process can control animation clips directly via AnimationPlayer, transition between clips with crossfade, and render to either a window or a transparent overlay. This is also the foundation for Surface 3.

Path 2 is recommended. It gives us full control without depending on ratty development, and the same binary becomes the desktop overlay.

---

Surface 2: Terminal portrait (all terminals)

Already works at four tiers of fidelity. No changes needed here — this surface is mature. The posture mapping we already built for RGP applies to all tiers.

---

Surface 3: Desktop overlay (Bevy transparent window)

Architecture

A separate process (or long-lived child) that runs a Bevy app with:

• Transparent window — Hyprland supports transparent/semi-transparent windows via layer-shell protocol. Bevy + bevy_winit + `bevy_kylewl (or raw-window-handle) can produce a frameless, transparent, click-through overlay that renders on top of other windows.

• GLB model — same model the TUI uses, loaded via bevy_gltf and driven by AnimationPlayer. Posture maps to animation clip transitions with configurable crossfade duration.

• Animation clips — the GLB model's internal clips (idle_breathing, thinking_gaze_down, speaking_lip_flap, etc.) are mapped to Posture states. Transitions use AnimationTransitions::play() with crossfade.

• Translucency — the overlay window is semi-transparent (configurable alpha). The model is fully opaque; the background is transparent. She floats on top of your desktop, not in a box.

• IPC control — a Unix socket or stdin/stdout channel receives posture updates from Souveraine's EventBus. The Bevy overlay process reads these and advances animation state. No polling, no shared memory — event-driven.

• Positioning — configurable screen region (bottom-right corner, center, etc.). Hyprland's movewindow or layer-shell positioning. Default: lower right, 240×320 px portrait crop.

State flow

Presence::set_posture(new_posture)
  → EventBus::send(SensorEvent { event_type: "posture", payload: "Thinking" })
    → EventLog persists
    → IPC channel listener in overlay process
      → AnimationPlayer::play(clip_for(posture)).crossfade(duration)
        → Bevy renders next frame at 60fps

No HTTP, no polling, no serialization. The EventBus is already wired for persistence; the overlay process subscribes to the same events.

Visual specification

• Model sits at ~1/3 screen height, anchored to bottom
• Subtle idle breathing animation (continuous, always on)
• Posture transitions crossfade over 300-500ms (not instant, not slow)
• Eye state (open/blinking) driven by the same jittered timer as the TUI
• Atmosphere color tint applies to model emissive / rim light
• Outfit changes swap the model material textures (or load a different GLB)
• Voice: when Speaking, lip-flap animation clip plays, TTS audio syncs (approximate — exact phoneme sync is deferred)

What it is NOT

• Not a game character. No walking, no physics, no world interaction. She occupies a fixed screen region and expresses through posture and animation.
• Not a replacement for the terminal. The overlay is an additional surface. The TUI still works. They coexist.
• Not a video call. No facial expresion capture, no real-time lip sync. Speaking posture triggers a generic lip-flap clip with the right energy.

---

Surface 4: Voice (STT/TTS)

Already fully spec'd in docs/tasks/tui-presence-voice.md. The key integration point: when the overlay process receives a Speaking posture event, it simultaneously plays the TTS audio (already piped through rodio in the TUI) and triggers the speaking animation clip. Sync is approximate — the clip plays for the duration of VoicePlayer::is_speaking().

Integration surface: VoicePlayer emits an event when speaking starts/stops. The overlay listens for that event (via EventBus or direct IPC) and transitions the animation accordingly.

---

Build sequence

Phase 1: RGP depth (this session, within ratty)

1. Extend Graphic::apply_posture() to handle per-clip animation if the model has named clips. Requires examining the GLB's clip names first: cargo run --example gltf_animation from bevy with the model to list available animation clips.
2. Fallback: if no named clips, the existing procedural spin/bob is the animation — this is already working.

Phase 2: Standalone Bevy overlay window

3. New crate: src/overlay/ — a Bevy app binary
   • Transparent window via bevy + raw-window-handle / layer-shell
   • Load the same GLB from the agent's assets directory
   • Subscribe to posture events via stdin/Unix socket
   • Map posture→animation clip with crossfade
   • Atmosphere color tint via material emissive + rim light
   • 60fps, <5% CPU when idle
4. Wire IPC sender in src/core/nervous/ — EventBus consumer that forwards posture changes to the overlay process
5. Start/stop lifecycle managed by App — spawn overlay on Presence screen entry, kill on exit

Phase 3: Voice sync

6. Wire VoicePlayer events to EventBus so the overlay knows when speaking starts/stops
7. Map Speaking posture to a lip-flap animation clip on the model
8. Optional: energy-normalize the TTS audio amplitude to drive clip speed

Phase 4: Desktop integration (Hyprland-specific)

9. Layer-shell positioning (bottom-right, configurable)
10. Click-through region (mouse events pass through to windows underneath)
11. Startup integration — overlay can be launched at Souveraine startup, not just from Presence mode
12. Multi-monitor awareness — appears on the active monitor

Phase 5: Multi-surface coordination

13. EventBus carries posture events; all surfaces subscribe. When one changes, all respond.
14. Surface priority: desktop overlay has visual priority (it's the highest fidelity), but the TUI presence is the authority (it's where posture changes originate).
15. If the overlay process crashes, EventBus persistence means it can rejoin and catch up on missed events.

Out of scope (deferred)

• Full facial rig (blendshapes, phoneme-level lip sync)
• Skeleton-level animation (walking, gestures, pointing)
• Physics-based hair/cloth
• Screen capture / vision sense
• Multi-agent overlay (one overlay per agent, positioned independently)
• Wayland compositor integration beyond Hyprland (layer-shell is compositor-agnostic but untested on Sway/KWin)

Files to create / modify

File	Action	What
src/overlay/mod.rs	new	Bevy overlay binary — transparent window, GLB rendering, animation
src/overlay/animation.rs	new	Posture→clip mapping, crossfade transitions
src/overlay/ipc.rs	new	EventBus subscriber → stdin/socket listener
src/core/nervous/mod.rs	edit	Add overlay IPC sender as an EventBus consumer
src/ui/presence.rs	edit (minimal)	Posture changes already fire events — no change needed
src/ui/app.rs	edit	Spawn/kill overlay process on Presence entry/exit
Cargo.toml	edit	Add bevy (optional, behind desktop feature)
src/ui/rgp.rs	edit	Keep as ratty-only inline 3D path; no changes needed

Testing

• Phase 1: cargo run --features rgp -- tui in ratty, verify posture-driven color/brightness changes on the inline model
• Phase 2: cargo run --features desktop — transparent window appears with animated model
• Phase 3: Trigger Speaking posture via TUI, verify overlay lip-flap animation
• Phase 4: Verify click-through, positioning, multi-monitor