Watch
1
0
Fork
You've already forked SouveraineOS
0

tasks: 65 — a stalled subconscious wedges the surface

Measured once: reply commits and PrimaryComplete fires, then the N+1
pass stops at round 3 of 10 with a fully idle runtime. The pass runs
inside run_turn, so tx never drops and the SSE stream never closes.
Lead is the todo tool. Stream lifetime is a separate fix.
This commit is contained in:
Fimeg 2026-08-10 15:46:18 -04:00
commit 287acdffd9
2 changed files with 118 additions and 0 deletions

View file

@ -0,0 +1,117 @@
# TASK-65 — A stalled subconscious wedges the surface after the answer landed
**Status:** open, 2026-08-10. Reproduced once, measured, not diagnosed.
**Connects to:** TASK-48 (lock surface reload), `COMPACTION_STRATEGIES.md`,
`souveraine` commits `a10ef5e``30e1c34` (prompt caching + provider-follows-model).
## Symptom
Souveraine Surfaces shows a turn as still running, indefinitely, after the
agent has already answered. The reply is on screen and committed to disk. The
spinner never clears and the conversation cannot be finalised.
## What was measured
One turn, agent `agent-f6422236` (Souveraine), conversation
`c7d90a47-5fee-4555-a25c-e389be9831fe`, server pid 755069 (restarted
2026-08-10T19:18:38Z, package `0.1.r398.g30e1c34a340c`).
| Time (UTC) | Event |
|---|---|
| 19:31:27 | turn starts, `msg_count=3`, `model=claude-opus-5` |
| 19:31:31 | round 0 returns, `tool_calls=2`; `cache_read=0 cache_write=22585` |
| 19:31:37 | round 1 returns, `tool_calls=0 content_len=727`; `cache_read=22585` |
| 19:31:38 | **reply committed**`message_count: 4`, assistant block 721 chars |
| 19:31:40 | subconscious N+1 starts, `model=deepseek-v4-flash` |
| 19:31:58 | sub round 0 returns, `tool_calls=1` |
| 19:32:01 | sub round 1 returns, `tool_calls=2` |
| 19:32:22 | sub round 2 returns, `tool_calls=3` |
| — | **nothing further.** Log silent at 19:42Z, ten minutes on |
State of the process while wedged:
- `%CPU 0.0`, cumulative `TIME 00:00:00`. Not computing.
- No child processes. Nothing shelled out.
- Every thread parked: 7 `tokio-rt-worker` on `futex_do_wait`, 1 on
`do_epoll_wait`, 3 idle `sqlx-sqlite-wor`. The runtime is **fully idle**
a task is awaiting something that will never fire.
- Exactly one socket: `ESTAB 127.0.0.1:8484 ← 127.0.0.1:39990`, Send-Q 0,
Recv-Q 0. Surfaces holding the SSE stream open. Nothing queued, so the
server is *not* blocked writing to it.
- `SUBCONSCIOUS_POST_TURN_ROUNDS = 10`. It stopped after round 2 — three of
ten. It did not exhaust its budget.
## The wedge mechanism (confirmed by reading, not inferred)
`PrimaryComplete` is sent at `src/server/turn.rs:916` and the message commits
just before it — so the user's answer is released and durable. But the N+1
pass runs **inside `run_turn`**, which still owns `tx`. A subconscious that
never returns means `run_turn` never returns, `tx` never drops, the SSE stream
never closes, and the surface waits forever on a turn that is, as far as the
user is concerned, finished.
That coupling is the defect independent of whatever caused the stall: a
post-turn pass must not be able to hold the response channel open. Either drop
`tx` / emit the stream terminator before the N+1 pass, or run the pass
detached from the turn's stream lifetime.
## The lead
`todo`. Casey's recall, and it checks out — `SUBCONSCIOUS_SAFE_TOOLS`
(`src/server/consciousness_engine.rs:46`) grants the subconscious
`read, write, edit, glob, grep, list_dir, memory, schedule, todo, halt,
intrusive`. The stall happens *after* a round returns `tool_calls=3`, i.e.
during tool execution, and the tool-call count climbed 1 → 2 → 3 across
rounds. `todo`, `schedule`, and `memory` (git-backed, `auto_push = true`) are
the three with state or I/O behind them.
An idle runtime with no sockets and no children points at an await that cannot
resolve — a lock held by a dropped guard, or a channel whose peer is gone —
rather than slow work. `global_sensorium()` is an `RwLock` taken on every tool
dispatch (`core/tools/mod.rs:284`); a writer guard leaked anywhere deadlocks
every subsequent tool call exactly like this.
## Ruled out
- Not the model or provider: three DeepSeek rounds returned cleanly, no 400s,
no retries. The provider-follows-model fix (`30e1c34`) is working — this path
previously failed fast against the wrong provider, which is why the stall was
never seen before.
- Not the prompt cache work: subconscious runs through the bifrost client, which
those commits do not touch.
- Not output loss: the reply is committed and complete.
- Not budget exhaustion: 3 of 10 rounds.
## Acceptance
1. Root cause of the stall named, with the awaiting task identified — not
guessed. A backtrace (gdb / `tokio-console`) or added instrumentation on the
tool-dispatch path, since external inspection cannot see past an idle runtime.
2. The offending tool's hang fixed, with a regression test that exercises the
subconscious executing it.
3. **`run_turn` returns, and the SSE stream closes, even when the N+1 pass
hangs.** A post-turn pass must never be able to wedge a surface. This is
acceptance in its own right and must hold with the stall bug deliberately
reintroduced.
4. Surfaces clears its spinner on stream close; verify against
`~/.config/quickshell/souveraine/services/Souveraine.qml` (`/v1/conversations/
{id}/messages`, and the `/cancel` endpoint at :505 as the manual escape).
5. Verified on hardware: a real Surfaces turn against Souveraine that finishes
and finalises, subconscious on DeepSeek.
## Notes for whoever picks this up
- Reproduce by talking to **Souveraine** (not Vanguard) through Surfaces with
`n1_enabled` on. Vanguard's subconscious has a separate, older failure — a
DeepSeek 400, *"assistant message with 'tool_calls' must be followed by tool
messages"*, 4-for-4 on 2026-08-10 morning. Different bug, do not conflate.
- `/v1/conversations/{id}/cancel` is the non-destructive way to test whether
cancellation unblocks the wedged task. If it does, the await is
cancellation-aware and the fix is likely a timeout; if it does not, it is a
lock.
- Server cwd decides which config loads. `ConsciousnessConfig::discover_path()`
(`core/config.rs:1296`) checks `./souveraine.toml` **before**
`~/.souveraine/config.toml`. The service runs from `/home/casey` and gets the
right one; anything launched from `~/Projects/souveraine` reads the stale,
gitignored repo copy instead — wrong provider, wrong model. That file is a
local dev leftover, not a template.

View file

@ -38,6 +38,7 @@ Four that are cheap relative to what they unblock:
| # | Task | What's left |
|---|------|-------------|
| 65 | [A stalled subconscious wedges the surface](65-subconscious-stall-wedges-the-surface.md) | Reproduced and measured, not diagnosed. The reply commits and `PrimaryComplete` fires, then the N+1 pass stalls at round 3 of 10 with a fully idle runtime — and because it runs inside `run_turn`, the SSE stream never closes and Surfaces spins forever. Lead is the `todo` tool. Needs a backtrace; the stream-lifetime coupling is a fix in its own right. |
| 62 | [Full USB KVM](62-full-usb-kvm.md) | GUD + HID + NCM + smoo and the signed package path are green. Still open: install and cable/glass acceptance; real Session Authority gates (peer identity, probe lease, step-up, D-Bus enforcement); cold-boot SSH-over-NCM proof. Type-C/TCPM host negotiation remains TASK-58. |
| 08 | [Device state manager / power profiles](08-device-state-manager.md) | Largest task; sub-divide before starting. (e) lease missing, (f) **surfaced 2026-07-31** — health, evidence and the decision trail render on the Device page; last-seen and grip still absent from `device_state`. (g) confidence gates need a decision before code. |
| 16 | [culver: messenger + contacts](16-culver-messenger-contacts.md) | A/B/D landed. C (contact card) builds and awaits phone acceptance. F is the folded-in TASK-01. |