SouveraineOS/docs/substrate/tasks/multimodal-and-input-metadata.md

task_id	title	status	assignee	priority	phase
multimodal-input-001	Multimodal + Input Metadata — richer input than plain text	landed (Sub-scope A incl. HTTP boundary, Sub-scope B); C pending		medium	4.x

2026-08-11 update — the HTTP boundary now carries images (souveraine cfad688). Sub-scope A was implemented for the TUI in May and stopped at the server door: src/api/models.rs held content: String, so no HTTP surface could send an attachment even though the session, run_turn, and the provider client were all already multimodal. MessageContent is now an untagged String | Vec<ContentBlock> — a bare string is the unchanged legacy wire; an array carries ordered text/image parts in the substrate's own ContentBlock vocabulary (the same shape ContentValue uses facing the provider). Only text and image are accepted inbound: tool_use, tool_result and reasoning are engine-written and a surface posting one gets 400 unsupported_content_block rather than a forged turn in history. The vision gate answers at the door — an image sent to a model without supports_images returns 422 image_not_supported naming the model, instead of being stripped downstream to [Image: attached by user], which reads to the human as if she looked and said nothing. run_turn's marker degrade stays as the fallback for history that already holds images when a model changes. Verified: 9 focused tests in src/api/models.rs, full suite 330 passing, cargo clippy -- -D warnings clean (checked in a detached worktree on ArchDev, not on the laptop). Not exercised: no real image has crossed the wire from a surface — nothing sends one yet. Still open: the surface half. surfaces/quickshell/services/Ai.qml::attachFile() still refuses attachments and its comment now describes a limitation that no longer exists; image intake in the UI belongs to TASK-72. Sub-scope C (user/system metadata) remains pending.

2026-05-19 update: Sub-scopes A (image support) and B (paste detection) implemented. What was built: ContentBlock::Image variant, ContentPart/ContentValue in Bifrost wire format, clipboard paste via arboard (Ctrl+Shift+V), /attach command, resize pipeline (progressive quality

• dimension scaling), model gating (supports_images flag), TUI rendering (Image chat bubbles), persistence via serde. Sub-scope C (user/system metadata) is still pending — the InputEvent changes are deferred until the Matrix sensorium exercises that path. Key files: src/core/session/mod.rs, src/bridge/bifrost.rs, src/core/image.rs, src/ui/chat/commands.rs, src/ui/chat/events.rs, src/backend/local/turn.rs, src/api/models.rs, src/core/config.rs. Tests: 172 passing (3 new resize pipeline tests).

Task: Multimodal & Input Metadata

Origin

Surfaced 2026-05-18 while scoping the Matrix sensorium. The moment a surface is not a terminal, "input" stops being a flat string. A Matrix message can carry an image; a paste is not the same gesture as typing; and every input arrives with metadata (who, from where, on what device) that the agent and the subconscious should be able to feel. InputEvent (src/core/sensorium/ mod.rs) currently carries only content: String + a thin InputMetadata.

These are not Matrix-specific — they apply to the TUI, mobile, and any future surface. Captured as one task because they share a theme: making input as rich as the surface that produced it.

Sub-scope A — Image support (vision-capable models)

When the configured model advertises image support, an InputEvent must be able to carry image data (or a reference), and the Bifrost request builder (src/bridge/bifrost.rs) must emit the OpenAI-compatible multimodal content array ({"type": "image_url", ...}) instead of a plain string.

• Capability gate: only send images to models that support them — otherwise degrade gracefully (describe / drop with a notice).
• Surfaces that can supply images: Matrix (media.ts equivalent), TUI (path paste / drag), mobile.

Sub-scope B — Paste detection

When the human pastes into the input, the agent should know it was a paste, not typed text. Other harnesses have clean logic for this — a large input arriving in a single terminal event, or a bracketed-paste escape sequence (\e[200~ ... \e[201~). The TUI input handler should:

• Enable bracketed-paste mode and detect the paste boundary.
• Tag the resulting InputEvent so the content is framed as pasted material (e.g. wrapped / labelled) rather than conversational text — the agent reads *[pasted N lines]* as information, the way she reads *[raised hand]*.

Sub-scope C — User & system metadata

Every InputEvent should be able to carry structured context about its origin, beyond today's file_path / project_path / selected_text:

• User metadata — who sent it (resolved name, not a hardcoded "casey"; see the source: human decision in the todo tool), their handle on the originating surface.
• System metadata — host, surface/device, locale, timestamp already in ambient_line(). Consolidate so ambient_line() and InputEvent draw from one source of truth.

Design direction

Extend InputEvent / InputMetadata in src/core/sensorium/mod.rs:

pub struct InputEvent {
    pub content: InputContent,          // was: String
    pub conversation_id: Option<String>,
    pub origin: InputOrigin,            // user + system metadata
    pub metadata: InputMetadata,
}

pub enum InputContent {
    Text { text: String, pasted: bool },
    Multimodal { text: String, images: Vec<ImageRef> },
}

Exact shape TBD — settle alongside the Matrix sensorium's inbound path (matrix-sensorium Phase 4), since Matrix is the first surface that exercises all three sub-scopes at once.

Related

• docs/tasks/matrix-sensorium.md — the surface that surfaced this
• src/core/sensorium/mod.rs — InputEvent, InputMetadata, ambient_line()
• src/bridge/bifrost.rs — multimodal request construction
• src/ui/chat.rs — TUI input handling, bracketed paste