Watch
1
0
Fork
You've already forked SouveraineOS
0
SouveraineOS/docs/tasks/65-subconscious-stall-wedges-the-surface.md
Fimeg 287acdffd9 tasks: 65 — a stalled subconscious wedges the surface
Measured once: reply commits and PrimaryComplete fires, then the N+1
pass stops at round 3 of 10 with a fully idle runtime. The pass runs
inside run_turn, so tx never drops and the SSE stream never closes.
Lead is the todo tool. Stream lifetime is a separate fix.
2026-08-10 15:46:18 -04:00

5.9 KiB

TASK-65 — A stalled subconscious wedges the surface after the answer landed

Status: open, 2026-08-10. Reproduced once, measured, not diagnosed. Connects to: TASK-48 (lock surface reload), COMPACTION_STRATEGIES.md, souveraine commits a10ef5e30e1c34 (prompt caching + provider-follows-model).

Symptom

Souveraine Surfaces shows a turn as still running, indefinitely, after the agent has already answered. The reply is on screen and committed to disk. The spinner never clears and the conversation cannot be finalised.

What was measured

One turn, agent agent-f6422236 (Souveraine), conversation c7d90a47-5fee-4555-a25c-e389be9831fe, server pid 755069 (restarted 2026-08-10T19:18:38Z, package 0.1.r398.g30e1c34a340c).

Time (UTC) Event
19:31:27 turn starts, msg_count=3, model=claude-opus-5
19:31:31 round 0 returns, tool_calls=2; cache_read=0 cache_write=22585
19:31:37 round 1 returns, tool_calls=0 content_len=727; cache_read=22585
19:31:38 reply committedmessage_count: 4, assistant block 721 chars
19:31:40 subconscious N+1 starts, model=deepseek-v4-flash
19:31:58 sub round 0 returns, tool_calls=1
19:32:01 sub round 1 returns, tool_calls=2
19:32:22 sub round 2 returns, tool_calls=3
nothing further. Log silent at 19:42Z, ten minutes on

State of the process while wedged:

  • %CPU 0.0, cumulative TIME 00:00:00. Not computing.
  • No child processes. Nothing shelled out.
  • Every thread parked: 7 tokio-rt-worker on futex_do_wait, 1 on do_epoll_wait, 3 idle sqlx-sqlite-wor. The runtime is fully idle — a task is awaiting something that will never fire.
  • Exactly one socket: ESTAB 127.0.0.1:8484 ← 127.0.0.1:39990, Send-Q 0, Recv-Q 0. Surfaces holding the SSE stream open. Nothing queued, so the server is not blocked writing to it.
  • SUBCONSCIOUS_POST_TURN_ROUNDS = 10. It stopped after round 2 — three of ten. It did not exhaust its budget.

The wedge mechanism (confirmed by reading, not inferred)

PrimaryComplete is sent at src/server/turn.rs:916 and the message commits just before it — so the user's answer is released and durable. But the N+1 pass runs inside run_turn, which still owns tx. A subconscious that never returns means run_turn never returns, tx never drops, the SSE stream never closes, and the surface waits forever on a turn that is, as far as the user is concerned, finished.

That coupling is the defect independent of whatever caused the stall: a post-turn pass must not be able to hold the response channel open. Either drop tx / emit the stream terminator before the N+1 pass, or run the pass detached from the turn's stream lifetime.

The lead

todo. Casey's recall, and it checks out — SUBCONSCIOUS_SAFE_TOOLS (src/server/consciousness_engine.rs:46) grants the subconscious read, write, edit, glob, grep, list_dir, memory, schedule, todo, halt, intrusive. The stall happens after a round returns tool_calls=3, i.e. during tool execution, and the tool-call count climbed 1 → 2 → 3 across rounds. todo, schedule, and memory (git-backed, auto_push = true) are the three with state or I/O behind them.

An idle runtime with no sockets and no children points at an await that cannot resolve — a lock held by a dropped guard, or a channel whose peer is gone — rather than slow work. global_sensorium() is an RwLock taken on every tool dispatch (core/tools/mod.rs:284); a writer guard leaked anywhere deadlocks every subsequent tool call exactly like this.

Ruled out

  • Not the model or provider: three DeepSeek rounds returned cleanly, no 400s, no retries. The provider-follows-model fix (30e1c34) is working — this path previously failed fast against the wrong provider, which is why the stall was never seen before.
  • Not the prompt cache work: subconscious runs through the bifrost client, which those commits do not touch.
  • Not output loss: the reply is committed and complete.
  • Not budget exhaustion: 3 of 10 rounds.

Acceptance

  1. Root cause of the stall named, with the awaiting task identified — not guessed. A backtrace (gdb / tokio-console) or added instrumentation on the tool-dispatch path, since external inspection cannot see past an idle runtime.
  2. The offending tool's hang fixed, with a regression test that exercises the subconscious executing it.
  3. run_turn returns, and the SSE stream closes, even when the N+1 pass hangs. A post-turn pass must never be able to wedge a surface. This is acceptance in its own right and must hold with the stall bug deliberately reintroduced.
  4. Surfaces clears its spinner on stream close; verify against ~/.config/quickshell/souveraine/services/Souveraine.qml (/v1/conversations/ {id}/messages, and the /cancel endpoint at :505 as the manual escape).
  5. Verified on hardware: a real Surfaces turn against Souveraine that finishes and finalises, subconscious on DeepSeek.

Notes for whoever picks this up

  • Reproduce by talking to Souveraine (not Vanguard) through Surfaces with n1_enabled on. Vanguard's subconscious has a separate, older failure — a DeepSeek 400, "assistant message with 'tool_calls' must be followed by tool messages", 4-for-4 on 2026-08-10 morning. Different bug, do not conflate.
  • /v1/conversations/{id}/cancel is the non-destructive way to test whether cancellation unblocks the wedged task. If it does, the await is cancellation-aware and the fix is likely a timeout; if it does not, it is a lock.
  • Server cwd decides which config loads. ConsciousnessConfig::discover_path() (core/config.rs:1296) checks ./souveraine.toml before ~/.souveraine/config.toml. The service runs from /home/casey and gets the right one; anything launched from ~/Projects/souveraine reads the stale, gitignored repo copy instead — wrong provider, wrong model. That file is a local dev leftover, not a template.