SouveraineOS/docs/substrate/tasks/matrix-sensorium.md

task_id	title	status	assignee	priority	phase	blocked_by
matrix-sensorium-001	Matrix Sensorium — Souveraine's first non-terminal surface	scoped	TBD	medium	4.0	matrix-outbound-streaming

✅ UNBLOCKED. The Sensorium trait now matches the ChannelAdapter contract (sensorium-adapter-reckoning.md is complete and merged). The inbound path is wired end-to-end: Matrix room messages fire sensorium:input SensorEvents, the SensoriumInputHandler picks them up, injects a turn via the backend, and turn:* events flow back to the sensorium's run loop over the EventBus.

What remains:

• Phase 5 — outbound streaming. MatrixSensorium::send_message is a stub. The handle_turn_event handler accumulates buffers per-room but doesn't send Matrix messages yet. The first chunk needs to create a new Matrix message, subsequent deltas need m.replace edits at ~150ms throttle (the letta-code streaming.ts pattern).
• MatrixSensorium needs to share its matrix_client with send_message. Currently the client lives inside run()'s closure scope. Fix: store as Option<Arc<MatrixClient>> on the struct, set by run().

Task: Matrix Sensorium

Objective

Give Ani a Matrix surface — she can be reached, and can render herself, in any Matrix room (Element, etc.). Crucially: build it as a sensorium adapter, not as a "channel". A Matrix room is a surface consciousness renders to and receives input from — that is exactly what src/core/sensorium/mod.rs already defines. The Matrix work is the proof case that turns that abstraction from dead code into a live layer.

Background

Casey ran a Matrix adapter for LettaBot. The reference implementation lives in his fork of letta-code:

• Source: ~/Projects/letta-code-justino/src/channels/matrix/ — pinned at aad7fd11 (the refactor/matrix-adapter merge: the old 2264-line monolith split into 13 focused modules + a turn/ streaming model).
• 5 unpushed local WIP fixes are preserved on branch backup-local-wip-20260518 in that repo — listener/rate-limit patches against the old monolith, not yet reconciled with the refactor. Review separately if needed; not a blocker.

letta-code models this as a ChannelAdapter (alongside Telegram/Slack/ Discord). Souveraine does not have channels — it has a Sensorium. The two are the same idea; this task re-homes the concept onto Souveraine's term and architecture rather than importing letta-code's channels/ subsystem.

Why this is not a copy-paste

Gap 1 — language

letta-code is TypeScript/Bun on matrix-bot-sdk. Souveraine is Rust. There is no code reuse. The Rust equivalent is the matrix-sdk crate (matrix-rust-sdk — the same Rust core matrix-bot-sdk reaches through to for crypto). This is a behavioral port: study his logic, reimplement against matrix-sdk.

Two things that are hard-won pain in his TS code and become free in Rust:

• crossSigning.ts (216 lines) — he reaches through matrix-bot-sdk into the underlying OlmMachine because the SDK exposes no cross-signing API. matrix-sdk exposes bootstrap_cross_signing() natively.
• client.ts — the undici/fetch transport shim works around Bun's socket pooling and matrix-bot-sdk's deprecated request lib. matrix-sdk owns its own HTTP transport. The whole file is irrelevant in Rust.

Gap 2 — the Sensorium trait is the wrong shape today

pub trait Sensorium: Send + Sync {
    fn bandwidth(&self) -> BandwidthClass;
    fn discovery_level(&self) -> DiscoveryLevel;
    fn render(&self, state: &ConsciousnessState) -> RenderedOutput;   // snapshot only
    fn input_receiver(&mut self) -> &mut mpsc::Receiver<InputEvent>;
}

render() produces one RenderedOutput snapshot. A Matrix turn cannot be a snapshot — it is a sequence of message edits over time (stream segment → tool card → stream segment → final), and the output must route back to a specific room. letta-code's ChannelAdapter carries all of this: handleStreamText, handleStreamReasoning, handleStreamReset, handleTurnLifecycleEvent, and per-message chatId / threadId. Sensorium carries none of it.

So this task extends the Sensorium trait — it does not just implement it.

Gap 3 — the sensorium layer is currently dead code

Sensorium, TuiSensorium, MobileSensorium, SensoriumCoordinator are all defined and unconstructed (~15 dead-code warnings in that file). MatrixSensorium would be the first sensorium ever wired into the runtime. That is the right outcome — but it means scope includes activating SensoriumCoordinator, not slotting into a working system.

Design — the central open question

How does outbound streaming reach a sensorium?

InputEvent already carries conversation_id: Option<String> — inbound routing (Matrix room → conversation) is fine. The gap is outbound: RenderedOutput has no room id, and there is no streaming hook.

Souveraine already has a nervous system — EventBus (broadcast channel) and SensorEvent with seed_id. Proposed direction: a sensorium does not get a fatter render(). Instead it subscribes to turn-lifecycle / stream events on the EventBus and renders them itself — which matches how TuiSensorium would eventually consume the same bus. The trait gains, roughly:

pub trait Sensorium: Send + Sync {
    fn bandwidth(&self) -> BandwidthClass;
    fn discovery_level(&self) -> DiscoveryLevel;
    fn input_receiver(&mut self) -> &mut mpsc::Receiver<InputEvent>;

    /// Drive this surface from the turn-lifecycle event stream. Replaces the
    /// snapshot `render()`. The sensorium owns its own incremental rendering.
    async fn run(&mut self, events: EventBus) -> Result<()>;
}

This is a proposal, not settled — it is the first thing to confirm with Casey before any code. The alternative (explicit handle_stream_* methods like letta-code) is heavier and TUI-irrelevant. Decide here first.

What ports from his turn/ model

His turn/ directory is the genuinely valuable behavioral logic — keep its shape, reimplement in Rust:

His file	What it knows	Souveraine equivalent
ChatTurn.ts	Per-room turn coordinator; owns the blocks below	MatrixTurn struct, one per active room
StreamingMessage.ts	Throttled leading-edge message edits (250ms → 8s backoff), word-boundary trimming	streaming-edit logic — the hard part
ToolBlock.ts	Per-tool timing, tool-card rendering into a message	maps to Souveraine tool cards
ThinkingBlock.ts	Reasoning-chunk rendering	maps to subconscious/reasoning surfacing
htmlFormat.ts	Markdown → Matrix HTML, streaming-safe	Rust markdown → Matrix HTML
media.ts	Media up/download	matrix-sdk media API

Phase plan (proposed)

1. Trait extension — settle and implement the Sensorium change above. Update TuiSensorium / MobileSensorium signatures so the crate still builds. No Matrix yet.
2. SensoriumCoordinator wiring — construct it in the runtime, register sensoria, give it the EventBus. Kills the dead-code warnings honestly.
3. Transport spike — matrix-sdk login + sync + send + receive against a real homeserver. Cross-signing bootstrap. Prove the transport.
4. MatrixSensorium inbound — room message → InputEvent → conversation routing. DM policy / pairing (see his setup.ts, controlRequests.ts).
5. MatrixSensorium outbound — port the turn/ streaming model: MatrixTurn + streaming message edits + tool/thinking blocks + HTML format.
6. Setup + crypto polish — setup wizard entry, cross-signing, media.

Implementation Architecture (6 phases, ~14 files)

Phase 1: Trait extension + Coordinator wiring

Files:

• src/core/sensorium/mod.rs — Replace render() + input_receiver() with async fn run(&mut self, bus: EventBus) -> Result<()>. Redesign SensoriumCoordinator with run_all() (spawns sensorium tasks with CancellationToken for coordinated shutdown), shutdown(). TuiSensorium/MobileSensorium get stub run() with dual-channel select! (input_rx + EventBus). RenderedOutput demoted to #[allow(dead_code)].
• src/backend/local.rs — Add Arc<tokio::sync::Mutex<SensoriumCoordinator>> field, initialized in new() but empty (not populated with sensoria yet).

Phase 2: TurnEventDispatcher

Files:

• src/core/nervous/turn_dispatcher.rs — TurnEventDispatcher struct (holds EventBus + Option<String> seed_id). 8 emit methods: emit_segment, emit_reasoning, emit_tool_start, emit_tool_end, emit_tool_tick, emit_turn_finish, emit_primary_complete, emit_interrupted. Each fires a SensorEvent with documented payload schema and event_type namespaced as "turn:segment", "turn:reasoning", etc.
• src/backend/local.rs — In run_turn(), construct a TurnEventDispatcher and call its methods at 7 integration points (reasoning chunks, tool start/end, token streaming, turn done, primary complete, interrupt).

Phase 3: Matrix transport spike

Files:

• src/core/sensorium/matrix/mod.rs — MatrixSensorium struct (client, homeserver, access_token, sync_cancel token, Arc<tokio::sync::Mutex<HashMap<String, MatrixTurn>>>). run() builds client, spawns sync loop, then select! loops between EventBus and cancellation. handle_event() dispatches by event_type to the right turn.
• src/core/sensorium/matrix/client.rs — build_client() (restore session from access token), run_sync_loop() (spawns a sync stream future).
• Cargo.toml — Add matrix-sdk = { version = "0.7", default-features = false, features = ["rustls-tls", "experimental-sliding-sync"] }. No experimental-encryption until Phase 6.

Phase 4: Turn model port

Files:

• turn.rs — MatrixTurn struct. Methods mirroring ChatTurn.ts: on_reasoning_chunk, on_tool_start, on_tool_end, on_stream_text, on_interrupted, finish. Each delegates to sub-blocks.
• stream.rs — StreamingMessage. Throttled leading-edge edits: 250ms initial backoff, doubles per rate-limit to 8s cap. word_boundary_trim() on every edit. finalize() sends complete text.
• blocks.rs — ToolBlock (scheduled → timed → live ticker at 5s cadence, dedup with (xN) count, completion with duration). ThinkingBlock ("Thinking..." placeholder, 150ms flush interval, 12k char reasoning clip).

Phase 5: HTML + wire format

Files:

• html.rs — streaming_markdown_to_html() (close partial fences/backticks/emphasis, render with pulldown-cmark). word_boundary_trim(). clip_reasoning_for_matrix() (keep tail at 12k chars). close_fences() / close_emphasis() / strip_html() helpers.

Phase 6: Setup + crypto + media

Files:

• matrix/setup.rs / credentials.rs — Keyring-based credential store (homeserver, access_token, user_id). bootstrap_cross_signing() becomes a one-liner via matrix-sdk.

Key risk areas

1. matrix-sdk version API surface — 0.7 may have different signatures than documented. Verify Client::builder(), restore_session(), sync_stream() at implementation time.
2. Sync loop is spawned in its own task — so the EventBus select! loop stays responsive — no risk there.
3. Turn state machine — finished flag guards against late events; turns.remove(target) on finish drops late arrivals silently.
4. Edit race conditions — Matrix is last-writer-wins per event_id, so concurrent edits are safe. The 250ms backoff naturally prevents rapid-fire.
5. Message size — 65536 byte Matrix limit is well above the 12k char reasoning clip.

Build sequence

1. Phase 1 → cargo build must pass with 0 errors. cargo test confirms.
2. Phase 2 → No matrix-sdk dependency yet. Pure unit-testable.
3. Phase 3 → Add matrix-sdk. Compiles but not wired into coordinator yet.
4. Phase 4 + 5 → Turn model depends on HTML module, so implement together.
5. Phase 6 → Credential helpers, then cross-signing.

---

Code layout (proposed)

src/core/sensorium/
  mod.rs            # trait + coordinator (extend)
  matrix/
    mod.rs          # MatrixSensorium: impl Sensorium
    client.rs       # matrix-sdk client construction, sync loop
    turn.rs         # MatrixTurn coordinator
    stream.rs       # streaming-edit logic (StreamingMessage port)
    blocks.rs       # tool / thinking block rendering
    html.rs         # markdown -> Matrix HTML
    setup.rs        # setup wizard + cross-signing bootstrap

Dependency policy — Matrix is an optional surface (decided 2026-05-18)

letta-code does not force the cost of every channel on every install — the Matrix dependency is only pulled in if you choose that channel. Souveraine's sensorium follows the same rule. A sensorium is a surface you opt into; the substrate must not carry a homeserver SDK by default.

Therefore matrix-sdk is gated behind a Cargo feature, not an unconditional dependency:

[features]
matrix = ["dep:matrix-sdk"]

[dependencies]
matrix-sdk = { version = "0.7", optional = true, default-features = false, features = ["rustls-tls", "experimental-sliding-sync"] }

• The whole src/core/sensorium/matrix/ module is #[cfg(feature = "matrix")]. A default build never compiles matrix-sdk and never links it.
• SensoriumCoordinator registration of a MatrixSensorium is also #[cfg(feature = "matrix")] — the coordinator itself stays feature-agnostic.
• Confirm rustls (not openssl) to match the existing reqwest / sqlx rustls posture in Cargo.toml.
• Matrix account credentials → OS keyring via src/core/credentials.rs, not souveraine.toml.

This generalises: every future non-terminal surface (Slack, etc., out of scope here) is its own optional feature. The TUI and the in-process surfaces are the only always-on sensoria.

Not in scope

• Telegram / Slack / Discord. Matrix only — it is the surface Casey runs.
• Reconciling the 5 WIP commits on backup-local-wip-20260518.
• Voice/media-call surfaces.

Related

• src/core/sensorium/mod.rs — the trait this task makes real
• src/core/nervous/ — EventBus / SensorEvent, the outbound event source
• src/core/credentials.rs — keyring for Matrix creds
• docs/tasks/sensorium-visual-flair.md — expressive vocabulary, TUI-side
• Reference: ~/Projects/letta-code-justino/src/channels/matrix/ @ aad7fd11