Watch
1
0
Fork
You've already forked SouveraineOS
0

tasks: the message API carries images now, the surface still does not

Reconciles the two owning docs with souveraine cfad688. Sub-scope A was
always half-landed: the substrate was multimodal end to end and the HTTP
door was text-only, so the TUI could see and every other surface was
blind. The door opens; nothing walks through it yet.

TASK-72 keeps image intake — picker, chip, base64, rendering — and now
has a server to call.
This commit is contained in:
Fimeg 2026-08-11 12:31:36 -04:00
commit 9669b0c3e3
2 changed files with 34 additions and 2 deletions

View file

@ -1,12 +1,37 @@
---
task_id: multimodal-input-001
title: Multimodal + Input Metadata — richer input than plain text
status: landed (Sub-scopes A & B)
status: landed (Sub-scope A incl. HTTP boundary, Sub-scope B); C pending
assignee:
priority: medium
phase: 4.x
---
> **2026-08-11 update — the HTTP boundary now carries images (souveraine `cfad688`).**
> Sub-scope A was implemented for the TUI in May and stopped at the server door:
> `src/api/models.rs` held `content: String`, so no HTTP surface could send an
> attachment even though the session, `run_turn`, and the provider client were
> all already multimodal. `MessageContent` is now an untagged `String |
> Vec<ContentBlock>` — a bare string is the unchanged legacy wire; an array
> carries ordered text/image parts in the substrate's own `ContentBlock`
> vocabulary (the same shape `ContentValue` uses facing the provider).
> Only `text` and `image` are accepted inbound: `tool_use`, `tool_result` and
> `reasoning` are engine-written and a surface posting one gets 400
> `unsupported_content_block` rather than a forged turn in history. The vision
> gate answers at the door — an image sent to a model without `supports_images`
> returns 422 `image_not_supported` naming the model, instead of being stripped
> downstream to `[Image: attached by user]`, which reads to the human as if she
> looked and said nothing. `run_turn`'s marker degrade stays as the fallback for
> history that already holds images when a model changes.
> **Verified:** 9 focused tests in `src/api/models.rs`, full suite 330 passing,
> `cargo clippy -- -D warnings` clean (checked in a detached worktree on
> ArchDev, not on the laptop). **Not exercised:** no real image has crossed the
> wire from a surface — nothing sends one yet.
> **Still open:** the surface half. `surfaces/quickshell/services/Ai.qml::attachFile()`
> still refuses attachments and its comment now describes a limitation that no
> longer exists; image intake in the UI belongs to TASK-72. Sub-scope C
> (user/system metadata) remains pending.
>
> **2026-05-19 update:** Sub-scopes A (image support) and B (paste detection) implemented.
> **What was built:** ContentBlock::Image variant, ContentPart/ContentValue in Bifrost wire format,
> clipboard paste via arboard (Ctrl+Shift+V), /attach command, resize pipeline (progressive quality

View file

@ -155,7 +155,14 @@ call the last item complete from a clean QML startup alone.
- **Paste affordance.** The TUI flagged pasted content and wrapped it for the
agent. The GUI does not, which is why a handoff document addressed to another
instance was read as a message to me on 2026-08-10.
- **Image intake** at the UI end.
- **Image intake** at the UI end. The server side landed 2026-08-11
(souveraine `cfad688`): the message API now accepts `"content"` as an ordered
array of `text`/`image` parts, gates on the model's vision capability at the
door, and refuses engine-owned block kinds. `Ai.qml::attachFile()` still
refuses attachments and its comment now cites a limitation that no longer
exists. Owner of the inbound contract is
`docs/substrate/tasks/multimodal-and-input-metadata.md`; the picker, the
chip, the base64 read and the rendering are this task's.
- **Resume offer** (patch 0001) has no continue affordance until this lane
lands; default-fresh works, one-tap-continue does not exist.