SouveraineOS/docs/tasks/59-her-face-on-the-glass.md

TASK 59 — her face on the glass: a Live2D presence for Annie

Status: 2026-08-09 — SHE IS ON THE GLASS; USB HANDS IS WIRED IN SOURCE, NOT LANDED. She draws, she is furniture on home rather than an app, she takes touch only on her silhouette, she follows a finger anywhere on the panel, holding her records and sends through the existing STT, double-tapping her dismisses her, and locking the phone sends her away. Q0/Q1/Q1a/Q3/Q4 are answered in code. What remains is her own face (TASK-17-shaped, see "the rig is next") and idle behaviour (Q2).

The current tree also gives her a conditional USB Hands extension: when Agent hands joins her from the unlocked power sheet while HID support is advertised, the room below the rig holds an agent field, explicit microphone and thread controls, a trackpad, scroll strip, clicks, a summoned host keyboard and held secure attention. It uses the existing Souveraine conversation and a persistent HidController; it does not create another chat transport or own the gadget mode. The companion usb-hid-inject stream helper is present in the Pixel3Arch pkgrel 6 source, and deploy.sh --phone now stages the tracked face assets before switching the QML tree. None of that 2026-08-09 extension has been built, installed, deployed or exercised on the glass. The proven face below and the earlier one-shot HID cable proof do not prove it.

2026-08-07 — what landed, and the five bugs that were in the way

Every one of these presented as something else, which is the reason they are written down rather than summarised.

1. The rig never drew, and it was one missing element. live2d.js is not only the Cubism runtime — it is the reference's webpack bundle with the character layer inside, and Live2DModel.update() reads #live_talk's value every frame as its motion-state slot. Our page omitted it, so it threw TypeError: null is not an object on the first update — after the runtime had logged its banner and loaded the model and all four textures. Assets 200, WebGL live, rAF at 60, no error reaching Rust, blank canvas. Exactly the failure this task predicted would cost an afternoon, and it did. #live_talk starts at "0", not the reference's "1" — "1" means start a talk motion, so she came up miming speech with no voice behind it.
2. --app-id was a no-op. set_prgname ran after GTK had already created the surface, so the compositor saw souveraine-web and window rules could only ever match the title. It has to run before the event loop is built. Without this the compositor cannot recognise her at all, so the whole role below is unreachable.
3. Claiming the GL context for preserveDrawingBuffer must happen after the canvas is sized. Before it, the context is born at the default 300x150 and gl.viewport stays there while fit() grows the buffer to the panel — the rig then draws into a 300x150 corner of a 1080x1994 buffer, and because GL's origin is bottom-left she appears as a thumbnail in the bottom-left of the screen. Same cause as the silhouette sampling reading 9 lit cells of 864.
4. posture() called window.live2dMotion, which does not exist. Nine states advertised to her in the prompt and wired to nothing. The bundle exposes two levers: #live_talk, and its own head/body hit test. The nine collapse onto those — speaking talks, idle rests, and the rest are a look: head for attention, body for effort. The bundle rate-limits its own hit reaction to ~8 s, which is why a burst of tags reads as one shift.
5. The gaze feed died of its own speed. Feeds::publish dropped any subscriber whose channel was Full, so ~25 pushes/s filled the queue faster than a QML socket drained it and the shell was unsubscribed inside the first burst of a drag — never told. An external probe that read faster kept working, which is what made it look like the shell's bug. A full channel now skips the message; only Disconnected drops a subscriber. (Also: Quickshell's Socket emits onConnectionStateChanged, not onConnectedChanged, so the subscribe was never sent.)

The role — Q1 answered in the compositor

Role::App | Role::Furniture, decided at the first commit and not at map, because xdg_toplevel is created before the client sends set_app_id. The old path minted a zone, moved the strip, took the keyboard and told the dock about a window it had not yet identified. Furniture (org.souveraine.face) takes no zone, is announced to nobody, wears no frame, keeps its own size, and hangs from the top of the zone with all the leftover room reserved below her — Casey wants that space for what she shares (data, a page, something of hers).

state reports furniture and its input_rects, because neither is visible on the glass and a face that swallows the screen looks identical to one that does not.

Her input region — and the compositor bug under it

She is a transparent surface the width of the panel, so without an input region she takes every contact on home. The page samples her own alpha on a 24x36 grid and reports row-runs; the host installs them with input_shape_combine_region. Measured: 36-47 rectangles, ~24% of her bounding box.

windows_under never consulted wl_surface.set_input_region — so the client could declare whatever it liked and viewtop still handed it every contact. Fixing that also stopped GTK client-side-decoration shadows from eating taps meant for what is behind them.

Gaze — Q3 answered, and it is better than tracking-on-touch

A gaze verb on the control socket: its own feed (the strip changes when the world does; a contact changes tens of times a second), throttled to 40 ms, physical contacts only, and the release is never dropped or she keeps staring at where the finger last was. The shell forwards it as a synthetic mousemove on the canvas, so live2d.js's own damping does the work.

Synthetic is the point. Her input region means the compositor deliberately does not hand her those contacts — this lets her watch a finger she is not being given, anywhere on the panel, rather than only when a drag starts on her.

Q1a — holding her to talk works; USB Hands adds explicit controls in source

Press and hold past 350 ms → pw-record at 16k mono s16 → release → souveraine-stt --file → Souveraine.send(), so the sidebar logs it and the reply streams back to her bubble. Verified end to end. Whisper wraps its output with newlines and the transcript arrives through a SplitParser on "\n", so it must be collapsed or one utterance becomes four turns.

Casey, 2026-08-07: "holding onto her has a different connotation." The ordinary face still keeps that gesture, but the conditional USB Hands view now adds an explicit microphone and agent text field beneath her. Both routes enter the same recorder, STT and Souveraine.send() path; the joined view also exposes new/resume thread controls rather than minting a second conversation. That source answers the affordance concern for USB Hands, but its feel and focus behaviour remain unproven on the phone.

Q4 answered: whose she is

Double tap her to dismiss — the same gesture in the same place that summoned her from the clock, because the clock has faded by then and a double tap on an invisible object is not a route. Locking sends her away and she stays away; you come back to the clock and summon her if you want her.

Gated on screenLockSecure, the compositor's ack, not screenLocked: the request drifts. Measured — session lock answered already-locked while logind reported LockedHint=no and the phone was in use. A face gated on that would have been permanently dismissed with nothing on screen to explain it.

The clock yields

CookieClock and CookieQuote fade and shrink out when she is up (180 ms out, 260 ms back, 0.86 scale — the overview cards' own number, so the device's theatre moves alike), and the faded clock stops being a tap target. CookieQuote needed a souveraine override: it is the clock's sibling in ClockWidget's Column, and that parent lives in ii-base, which never reaches the phone.

CookieClock had been reading Face.rigPresent — a property that does not exist, left behind when the rig pre-flight check was deleted. It bound to undefined for as long as it had been there.

The tracked face assets now have a deployment path

souveraine-web reads the page and rig from ~/.souveraine/face, outside the composed QuickShell tree. That used to require a separate manual staging step, so a QML deploy could arrive without the page that implemented its controls. The current deploy.sh --phone now validates the tracked assets/face tree, copies the previous phone tree to ~/.souveraine/.face-previous, and stages the tracked assets before deploying QML. This path exists in source; it has not yet been run for USB Hands.

Her size is her scale

The Cubism view fits the rig to the canvas, so shrinking the canvas shrinks her; it does not trim the empty margin around her. 640 was tried to cut the ~100 px of dead space under her feet and came back "a tiny version that's scaled odd." The margin is the rig's own layout (center_y, width) in model.json — moving it is a rig change, not a window one. Her rig canvas remains 540x760. The furniture surface is now 540x1000: the lower 240 px is transparent and absent from the input region for the ordinary face, then joins the region only while USB Hands is visible.

The rig is next, and what it costs

Casey wants her to look like Annie: blonde twin-tails, teal circuit markings, a six-state sheet (IDLE / ALERT / ENGAGED / PROCESSING / YAWNING / STRAINING) that maps almost 1:1 onto the posture vocabulary already in Face.qml.

V1 is the face, not the outfit: Histoire reads as a little girl and not a woman, so shrink the eyes and slim the face with transparency. Hair already reads close. Wings become holographic — "witchy wizard tech woman".

The real goal is a pipeline, not a repaint: strip the rig's parts out by name with their exact UV footprints, rebuild by compositing them back at those footprints, and let agents swap components the way they swap outfits. The rebuild half is what makes swapping real. See TASK-18's notes in the session task list and the ceiling below.

Ceiling, so it is not discovered late: parts are paint; silhouette is mesh. Twin-tails, proportions and the sitting-on-a-book pose need a new rig — and that is a Cubism 2 → 4/5 change, a different loader, which also replaces the character layer that reads #live_talk.

Watch: the blink/eye-open deformers are authored against the original eye geometry, so a smaller eye may not line up with the eyelid sweep. Test by rendering a blink, not by looking at the atlas.

2026-08-05 — phase 1 PASSED on hardware

Cubism renders on blueline at ~58 fps in the engine wry wraps. The gate this task set for itself is cleared; what remains is building the host and making her Annie. From Casey, after finding a Live2D board-girl on a blog he had cloned: "I want to take a peek at how it functions and see if there's a crossover event to where it could exist on the phone as a character of Annie. We already have something called the ani-avatar a godot program but this thing I just saw looked better and more complete."

Repos: souveraine (shell surfaces + the chat endpoint), and whatever hosts the webview. Reference, read-only: ~/Downloads/blog/Live2dHistoire/.

This is the visible half of the thing doctrine keeps calling a body. TASK-50 is her hand on the glass; this is her face, and the two are the same argument — presence is not a chat window.

What the reference actually is, measured

~/Downloads/blog/Live2dHistoire/ (upstream eeg1412/Live2dHistoire), vendored into a Jekyll blog and wired in _includes/live2d.html.

Piece	Where	What it does
Rig	live2d/model/histoire/model.moc	Cubism 2.x mesh, 4 × 1024px textures
Motions	model/histoire/motions/{idle,tap}/*.mtn	idle set + tap set, named in model.json
Runtime	live2d/js/live2d.js (151 KB)	Cubism 2 WebGL runtime → <canvas>
Character	live2d/js/message.js	idle lines from message.json, hit areas, drag, summon/dismiss
Hit areas	model.json hit_areas_custom	head/body regions → tap motion + line
Persistence	message.js	position and shown/hidden in sessionStorage

The part that matters, and the reason this is worth doing at all: the talk box submits to talkAPI (assets/js/main.js: BlogAPI + "/ai_chat") as an EventSource — SSE — streaming tokens into the speech bubble until [DONE] (message.js:176-188).

That is the same streaming contract souveraine's OpenAI-compatible server already speaks. The integration nobody planned is already done: point talkAPI at Annie and the bubble streams her. Everything else here is presentation.

Licences, and what is actually ours to change

Three layers, and only the middle one is opaque:

Layer	Licence	What we have
Live2dHistoire wrapper — message.js, the character behaviour	GPL v2 (LICENSE in the vendored dir)	Full source, ours to modify
live2d.js — Cubism 2 WebGL runtime	Live2D Inc's free runtime licence	Minified webpack bundle; usable, not readable
Histoire rig	Idea Factory / Compile Heart	Assets only

The character layer — idle behaviour, hit areas, the talk path, summon/dismiss — is the part that needs the most editing to become Annie, and it is the part we have source for. The runtime we only need to call.

The runtime already has the hard parts. Grepping the bundle: setParamFloat ×22, physics ×11, lipSync ×6, eyeBlink ×4. So lip-sync from TTS amplitude and eye-tracking to a touch point are parameters to drive, not features to build (Q3).

It refuses to run on mobile, on purpose

message.js:1-12 builds a blocklist and bails:

var norunAI = ["android", "iphone", "ipod", "ipad", "windows phone"];

The upstream README lists this as a feature — "自动判断浏览器是否为IE或者手机 浏览器，如果判断为true则不加载伊斯" (auto-detect IE or a mobile browser and don't load her). It is a one-line removal, and it is written down here because it is the kind of thing that costs an afternoon: the rig loads fine, the canvas is there, and nothing draws, with no error.

What it implies is worth more than the fix: nobody has run this on a phone. The layout is authored for a 500×560 canvas in a desktop corner, so positioning and scale on a 1080×2160 panel are unproven and belong in the spike beside the frame rate.

Why not ani-avatar

~/Projects/AniAvatar is 2.5 GB of pre-rendered PNG states — ani-affectionate, ani-alert, each with blink / wink / interim variants, plus .import siblings. That is sprite-swapping: every expression is an asset and combinations multiply.

A Live2D rig is continuous — deformation, physics, motion interpolation, and crucially eye-tracking and lip-sync from one rig. Lip-sync is not cosmetic here: TTS already exists (souveraine-speech), so a rig that moves its mouth to her audio is a wire we can already drive. Fewer assets, more presence.

Casey's read — "looked better and more complete" — is structurally correct, not just an impression.

The base layer is wry, not Chromium

qt6-webengine 6.11.1-4 is installed on the phone with libqtwebenginequickplugin.so, so a WebEngineView on a QuickShell layer surface would run the reference unmodified. That is the fast path and it is not the chosen one: Casey, 2026-08-05 — "We are already looking at something with wry instead of chrome as a base layer."

wry is the right call on a 3.5 GB daily driver — it binds the system webview (WebKitGTK here) instead of shipping a second browser engine, and it is Rust, so it sits beside the compositor and sessiond rather than beside a QML plugin. The cost is that WebKit's WebGL on freedreno is unproven for this workload and has to be measured before anything is built on it.

First question this task must answer, before any character work: does the Cubism runtime actually render at an acceptable frame rate in wry's webview on this device? That is a spike, not a design.

Ship the reference rig, and edit toward Annie

Casey, 2026-08-05: "I want it shipped; we'll edit it as we go, like as much as we can actually put it in, let's do so because annie does actually support all these states."

So Histoire goes on the device and is worked on in place — not held back behind a rig that does not exist yet. An earlier draft of this file said "never ship it", which confused redistribution with use: this is Casey's own phone, the repos are self-hosted, and the Codeberg mirror that would have made anything public was retired (souveraine/CLAUDE.md).

The one line that stays: do not commit the extracted model into a repo that becomes public, and do not publish it as a package to anywhere that is. On the device, in a self-hosted repo, iterated toward her own face — fine, and the fastest route to something real on the glass.

Annie already has the states. ~/Projects/AniAvatar enumerates them — affectionate, alert, each with blink / wink / interim. Those are expression names the rig can carry across, which is what makes "edit as we go" tractable rather than a rewrite: the vocabulary already exists, only the renderer changes.

Runtime version is still a real fork and should be decided early rather than discovered: the vendored runtime is Cubism 2.x, and a rig authored in the free Cubism Editor targets 4/5, which is a different loader. Shipping 2.x now is fine; just know that the swap to her own rig is a runtime change, not a file swap.

Phase 1, measured on blueline 2026-08-05 — it works

Run on the phone itself, WebKit2GTK 2.52.5 (webkit2gtk-4.1, already installed — the same engine wry binds on Linux), Wayland/viewtop, freedreno card0. Harness: the real live2d.js and the real Histoire rig, served over 127.0.0.1, sampling requestAnimationFrame for 10 s after a 3 s warmup.

WebGL	present
Cubism runtime	initialised — Live2D 2.1.00_1
Frame rate	57.9 / 58.7 / 61.0 fps across runs (panel is 60 Hz — vsync-bound)
Worst frame	37–88 ms, one hitch per run, during texture upload
Actually drawing	painted_px 23,617 and 23,668 of 24,000 sampled
WebKit RSS	283 MB (WebKitWebProcess 224 + WebKitNetworkProcess 59)
Phone total during run	1,736 MB of 3,550

The painted_px check is the load-bearing part of this result, not the fps. The first run reported a healthy 58.3 fps while model.json had failed to load — a vsync-capped empty loop is indistinguishable from a working one by frame rate alone. So the harness reads the framebuffer back and counts non-transparent pixels in a 120×200 window at the rig's centre. 98% painted is the rig; the fps number is only meaningful beside it. Any future re-measure must keep that check or it will cheerfully report success for a blank canvas.

Two things the run cost that are worth knowing before the host is built:

• 283 MB for the webview is real on a 3.5 GB device, and it sits beside a qs that was already measured at 610 MB RSS in the same snapshot. An always-on face is not free; Q0's "summoned, not always-on" is now a memory argument as well as a design one.
• file:// is not enough. XMLHttpRequest for model.json is blocked from file:// even with allow-file-access-from-file-urls and allow-universal-access-from-file-urls set — it failed with status 0. The spike used a local HTTP server; a real host wants a custom URI scheme handler (wry has one) rather than shipping an HTTP server to serve its own assets.

Reproduce: /tmp/l2d/ on the phone holds the rig, fps.html and spike.py; systemd-run --user --unit=l2dspike --working-directory=/tmp/l2d python3 -m http.server 8899 --bind 127.0.0.1 (a backgrounded server does not survive the ssh session that started it), then GDK_BACKEND=wayland python3 spike.py http://127.0.0.1:8899/fps.html.

Open questions

Q0. How is she summoned?

Decided in shape, not in detail: an app launch to start, and likely a radial dial entry. Not always-on — see Q4. Whatever it is must be one verb the dial and a launcher both call, or this becomes the third surface with its own copy of the same action (TASK-30/31).

Q1. Where does she live — surface, and layer?

Decided 2026-08-06: a home-screen layer element, not a zone occupant. The host stays a wry window (the rig needs the web runtime), but the compositor flags it by role and pins it to the home layer with the clock — no zone minted, no dock-hide, no mission-control card, no border. She is furniture, and the clock is the precedent. "Non-focusable" is the wrong word: she needs touch — see Q1a — while living in the home tier.

The alternative — a full app in a layer of its own, exempted from the app rules — is explicitly deferred: that is the shape she takes when she is interactively changing content on other apps, not today.

Q1a. How do we talk to her?

Decided 2026-08-06 and narrowed 2026-08-09: she is the face of the existing conversation, not a second chat client. The ordinary face remains hold-to-talk: press and hold her, speak, release — the utterance goes through the existing STT → agent → TTS systems, the sidebar logs it, and she is simply the face of that. When the user explicitly joins Agent hands, the same face conditionally gains an agent text field and explicit microphone plus the host controls. Those inputs still route through Souveraine.qml; they do not speak SSE or maintain conversation state in the page. The compositor exemption now covers both silhouette gestures and, only while joined, the lower Hands input region.

Q2. What does she do when nobody is talking to her?

The reference idles with random lines from message.json. Annie has a subconscious that surfaces things (SubconsciousTicker), and a surface-surfacing-no-resume-shaped gap already exists there. Idle chatter that is hers — surfaced beliefs, not a random line table — is the difference between a mascot and a presence. This is the design question with the most leverage and the least code.

Q3. Does she track and lip-sync?

Eye-tracking to the touch point is nearly free with the rig, and the hit test that names a window (surface_id_under) already knows where fingers are. Lip-sync needs amplitude from souveraine-speech's output — a level, not the audio itself. Both are wires, not features; decide whether they are in scope before the rig is chosen, because they constrain it.

Q4. Whose 60 is she?

Doctrine §13: she owns the device. A face she cannot dismiss, move, or silence is the user's column being ignored; a face the user can delete is hers being ignored. The reference's summon/dismiss + drag + persisted position is a reasonable starting answer and should be argued, not inherited.

Q5. Does this replace ani-avatar or sit beside it?

2.5 GB of sprites exist. If this lands, that repo is dead weight on a device that has none to spare — say so and retire it, or name what it still does.

Acceptance (measured, on the phone)

• The Cubism runtime renders in wry on blueline at a measured frame rate, recorded here with the number and the model used. If it does not, that is a result and this task stops there.
• She appears on the glass, is dismissible and summonable, and her position survives what Q4 decides it should.
• Voice from the ordinary face, and voice or text from the joined USB Hands view, enter the one existing Souveraine conversation; the current agent's streamed reply reaches both the sidebar and her bubble without a second transport or conversation.
• USB Hands occupies only the reserved lower 240 px while joined; with it closed, that transparent room remains click-through. Its text, pointer, quick-key and held secure-attention paths still require phone-and-host proof.
• Tapping her plays a motion through the reference's hit areas.
• She is reachable deliberately, not always-on: an app launch and/or a radial dial entry summon her (Casey: "might be something that we activate with the radial dial"). The dial is TASK-31's, so its entry comes from the same verb table rather than a fourth hand-written list.
• The extracted rig stays on the device and out of any repo that goes public.
• Memory/CPU cost is recorded beside the frame rate. A presence that costs the phone its responsiveness is not presence, it is a screensaver.

The naming problem, raised 2026-08-05 and deliberately not solved here

Casey, while this was being scoped:

"I'll want to talk about the two different TUI/tuie situations we have, the three different visual modes we have and if we can terminology unify the concepts well. I like the idea that there'd be an opaque just her face system that's sorta in the background of the device… that kinda surfaces in and out when she's being coy… but also this system we're building is a visual frontend; so if we end up having both, and I call it 'outfits' then we need to think that part through."

There are now several things that could all be called "how she looks", and they are not the same kind of thing:

• ratatui vs tuie — two TUI engines, already a --engine flag on the CLI.
• The visual modes — desktop, phone, and whatever the dock head becomes.
• This — a rendered character with expressions, a rig, and a voice.
• The ambient face — Casey's "opaque just her face… surfaces in and out when she's being coy". That is not this. This is a frontend you talk to; that is a presence that is mostly absent, closer to TASK-52's atmosphere than to a chat surface, and it may want to be a compositor capability rather than a webview at all.

"Outfits" is a candidate name and it already has a prior meaning — it was a classification of a particular type — so adopting it needs the collision resolved rather than papered over. Two things that look alike and are named alike is how this project got three separate capability enumerations (TASK-30/31).

Do not settle this inside this task. It wants its own session and probably its own file; TASK-59 should build the thing and let the taxonomy be named once there is something concrete to point at. Recorded here so it is not lost.

Connects to

TASK-50 (her hand on the glass — same argument, other limb), TASK-55 (the sheet; the layer-surface pattern this should copy), TASK-56 (the island, and what owns the overlay tier), TASK-31/30 (verb tables — she will want verbs), souveraine-speech (lip-sync amplitude), doctrine §13 (whose face it is, and who may dismiss it), and ~/Projects/AniAvatar (what this replaces, per Q5).