Fimeg/SouveraineOS
Watch
1
0
Fork
You've already forked SouveraineOS
 0
Code Releases Activity

tools: read CI outcomes from the API, and do it automatically

Browse source 
A rotted action pin skipped the publish job rather than failing it, so
edge stopped publishing and the phone sat on an old build for a working
day with nothing saying why. The runner journal logs task pickup and
never outcome, which is what was being read.

ci-status.sh asks Gitea instead, drilling run -> job -> log tail on a
failure. claude-hook-push-ci.sh runs it after every git push so the
answer arrives without anyone choosing to look. TASK-42 has the
three-layer plan; this is layer 1.
This commit is contained in:
Fimeg 2026-07-28 17:38:44 -04:00
commit f7b4b04525
5 changed files with 360 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

118
docs/tasks/42-build-outcomes-are-invisible.md Normal file
View file

@ -0,0 +1,118 @@
# TASK 42 — Build outcomes are invisible, and the silence looks like success
**Status:** open. Raised 2026-07-28 out of the grip session, where a broken
publish went unnoticed for the length of a working day. **Size:** small for the
detection, larger for the surfacing. **Repos:** `souveraine` (updater, CI),
`Pixel3Arch`, tooling.
## What happened, because the shape matters
`dtolnay/rust-toolchain` was pinned to a SHA that was the tip of that repo's
`stable` branch. dtolnay force-moves `stable` on every Rust release, so the
commit was orphaned — still an object on GitHub, reachable from no ref. act
cannot resolve it, `rust-test` dies before compiling anything, and
`aarch64-artifact` is therefore **skipped**.
Not failed. *Skipped.* The run finishes, the phone keeps installing whatever was
last published, and nothing anywhere says the pipeline stopped delivering. The
device sat on `0.1.r260` while `primary` and `public` both carried newer code,
and the only reason it was caught is that someone went looking for an unrelated
reason.
**The silence is the bug.** A pipeline that fails loudly is a nuisance; one that
stops publishing quietly is indistinguishable from one with nothing to publish.
## The audit that followed (2026-07-28)
Queried once, by API, across every repo of ours that has Actions:
| Repo | Recent runs |
|---|---|
| `souveraine` | 2 failures on `public`, green after the repin |
| `Pixel3Arch` | runs 929 and 911 failed on `main` — kernel.yml, which TASK-25 records as never once green |
| `culver` | three consecutive failures before the last success |
| `souveraine-viewtop` | 2 failures |
| `hexagonrpc` | green |
| `souveraine-updater` | green |
None of those failures had been seen by anyone. This is not one broken pin; it
is that **nothing reads build outcomes at all.**
## Gitea already reports everything
This is not a Gitea limitation and no new CI feature is needed. Everything was
in the API the whole time:
GET /api/v1/repos/{owner}/{repo}/actions/runs?limit=N -> status, conclusion, head_sha, branch
GET /api/v1/repos/{owner}/{repo}/actions/runs/{id}/jobs -> per-job conclusion
GET /api/v1/repos/{owner}/{repo}/actions/jobs/{id}/logs -> the actual error
Authenticates with the same Gitea token already in `~/.git-credentials` on
archdev and in the Pixel3Arch remote URL.
What was being read instead was `journalctl --user -u gitea-runner`, which logs
task *pickup* and never *outcome* — so it shows activity for a run that failed
thirty seconds later. Three wrong diagnoses in a row came out of that one habit.
**The runner journal is not a status source.** Use the API.
## Three layers, and they are separable
### 1. After a push, know what happened (built 2026-07-28)
`tools/ci-status.sh` in Pixel3Arch — takes a repo name, prints the latest run's
conclusion, and on failure prints the failing job and the tail of its log. Wired
as a Claude Code `PostToolUse` hook on `git push` so it runs without anyone
choosing to run it. That closes the immediate loop and is the cheap 80%.
Limitation: it reports the run that exists *at that moment*. A cross build takes
~20 minutes, so the hook usually reports "in_progress" and the real answer needs
a second look. Good enough to catch "failed instantly", which is what pin rot,
lint, and attribution failures all look like.
### 2. Notice that publishing has stopped (TASK-29's natural job)
The layer that would actually have caught this. `souveraine-updater` already
talks to the repo the phone installs from; it is the right place to ask **"how
old is the newest thing in `edge`, and how does that compare to the branch head
it should be tracking?"**
A pipeline that has published nothing in N days while `public` has moved is the
signal. That is one comparison, it needs no new infrastructure, and it does not
care *why* publishing stopped — which is the property that makes it robust
against the next failure mode nobody predicted.
### 3. Surface it on the device
The phone is the thing that suffers a stale build, and it currently has no way
to say so. This is the same shape as TASK-08(f) (`sensors_degraded` reaches no
surface) and belongs with it in TASK-19's health readout: *last successful
publish*, *installed vs available*, *how far behind*.
## Do not
- **Do not treat a green run as a published artifact.** A `skipped` publish job
lives inside a run whose overall conclusion can still read success. Check the
job, or better, check the artifact.
- **Do not pin an action to a branch tip's SHA.** That is what rotted. A tag's
SHA is immutable and safe; `actions/checkout@34e1148` (a v4 tag) survived the
same event untouched. If an action must track a branch, pin to a commit on a
branch that is append-only, and say in a comment why that branch was chosen.
- **Do not read the runner journal for status.** See above.
## Acceptance
- A push whose CI fails produces a visible failure without anyone deciding to
look.
- Something notices when `edge` has not moved while its branch has, and says so
before the phone is a week behind.
- The device can answer "am I running the newest build, and if not, why not".
- The pin-rot class is prevented, not just fixed once: a check that flags any
`uses:` SHA that is no longer reachable from a ref would have caught this the
day the branch moved.
## Connects to
TASK-29 (`souveraine-updater` — layer 2 is its job), TASK-25 (one repo all
packages; kernel.yml has never been green and nobody knew that either),
TASK-27 (pipeline audit), TASK-19 + TASK-08(f) (the device-side readout),
TASK-28 (half-upgrades move authority silently — the same disease one layer up).

1
docs/tasks/README.md
View file

@ -46,6 +46,7 @@ plus open threads from the 2026-07-17 session.
| 40 | [Inference tiering for the sensor surfaces](40-inference-tiering-sensors.md) | **open, gating** | P1 applied to sensord/grip/dial; blocks TASK-13's raw stream and gates TASK-31's IPC |
| 41 | [Producers must be attested; one authority per input](41-attested-producers.md) | open | doctrine §10 pointed at a real surface; demonstrated live 07-28 — a squeeze and a dial-open both accepted from an SSH shell |
| 42 | [viewtop: the Souveraine compositor](42-viewtop-compositor.md) | **foundation 07-28** | lock/gate/layers/back + intent table, 123 tests, CI green for the first time; no Wayland/DRM yet. Engine in-process is settled; the two-client lock handoff is the thing not to get wrong |
| 42 | [Build outcomes are invisible](42-build-outcomes-are-invisible.md) | detection built 07-28 | a rotted action pin skipped the publish job for a working day and nothing said so; `tools/ci-status.sh` + push hook close layer 1, layer 2 is TASK-29's |
| — | [Unify the shell trees](unify-shell-trees-laptop-phone.md) | open | ~900-file `ii` drift; blocks durable shell fixes reaching the phone |
Archived (see `archive/`): 05 crash reporter — done 2026-07-21,

69
tools/README.md Normal file
View file

@ -0,0 +1,69 @@
# tools
Cross-repo operational tooling. Lives here rather than in a component repo
because it reaches across all of them.
## `ci-status.sh` — what did CI actually do
./ci-status.sh # infer the repo from the cwd's git remote
./ci-status.sh souveraine # name it
./ci-status.sh souveraine 10 # last 10 runs instead of 3
./ci-status.sh --all # newest run for every repo we own
Exits 0 when the newest run is green or still going, 1 when it failed, so it
works as a gate as well as a report. On a failure it drills down by itself: the
run, then which job, then the tail of that job's log with the docker noise
stripped.
Token comes from `$GITEA_TOKEN`, else `~/.git-credentials`, else the Pixel3Arch
remote URL. No new copy of a secret in the tree.
### Read this before trusting anything else for build status
**`journalctl -u gitea-runner` is not a status source.** It logs task *pickup*
and never *outcome*, so a run that died thirty seconds in still shows as
activity. That habit produced three confident wrong diagnoses in a row on
2026-07-28 while the correct answer sat in the API untouched.
The failure that prompted all this: a pinned action SHA rotted, `rust-test` died
before compiling, and so `aarch64-artifact` was **skipped** — not failed. The run
looked unremarkable, `edge` silently stopped publishing, and the phone kept
installing a build from days earlier for a working day. Full write-up and the
three-layer fix in `../docs/tasks/42-build-outcomes-are-invisible.md`.
## `claude-hook-push-ci.sh` — the same thing, without having to remember
A Claude Code `PostToolUse` hook on `Bash`. It ignores everything that is not a
`git push`, and after one it prints that repo's CI status.
Wired in `~/.claude/settings.json`:
```json
"hooks": {
"PostToolUse": [
{ "matcher": "Bash",
"hooks": [ { "type": "command",
"command": "/home/casey/Projects/SouveraineOS/tools/claude-hook-push-ci.sh" } ] }
]
}
```
It always exits 0 — a status reporter that can block a push is a worse problem
than the one it solves.
**What it cannot do.** A cross build takes ~20 minutes, so straight after a push
the honest answer is usually "running". It reliably catches the fail-fast class
— pin rot, lint, attribution, a broken workflow edit — which is what actually
bites. Catching "publishing quietly stopped" needs the staleness comparison in
TASK-42 layer 2, which belongs to `souveraine-updater`, not to a longer sleep
in a hook.
## Pinning actions, since this is where it cost us
- A **tag's** SHA is immutable. `actions/checkout@34e1148` (v4) survived
untouched.
- A **branch tip's** SHA is not. `dtolnay/rust-toolchain` force-moves `stable`
every Rust release, which orphaned the pin — the commit still exists as a
GitHub object but is reachable from no ref, so act cannot resolve it.
- If an action must track a branch, pin to a commit on an **append-only**
branch and say in a comment why that branch was chosen.

122
tools/ci-status.sh Executable file
View file

@ -0,0 +1,122 @@
#!/usr/bin/env bash
# What did CI actually do? Ask Gitea, not the runner journal.
#
# ci-status.sh # infer the repo from the cwd's git remote
# ci-status.sh souveraine # name it
# ci-status.sh souveraine 10 # last 10 runs instead of 3
# ci-status.sh --all # newest run for every repo we own
#
# Exit 0 when the newest run is green or still going, 1 when it failed, so this
# is usable as a gate as well as a report.
#
# WHY THIS EXISTS. On 2026-07-28 a pinned action SHA rotted, `rust-test` died
# before compiling anything, and `aarch64-artifact` was therefore SKIPPED rather
# than failed — so the run looked unremarkable while the phone quietly kept
# installing a build from days earlier. It went unnoticed for a working day.
# The whole answer was in this API the entire time. What was being read instead
# was `journalctl -u gitea-runner`, which logs task PICKUP and never OUTCOME,
# and which produced three confident wrong diagnoses in a row.
#
# THE RUNNER JOURNAL IS NOT A STATUS SOURCE. This is.
#
# See docs/tasks/42-build-outcomes-are-invisible.md.
set -uo pipefail
GITEA="${GITEA_URL:-http://10.10.20.120:4455}"
OWNER="${GITEA_OWNER:-Fimeg}"
# Token, in order of preference: the environment, then the credential store
# git already uses for this host. No third copy of a secret in the tree.
if [ -z "${GITEA_TOKEN:-}" ] && [ -r "$HOME/.git-credentials" ]; then
GITEA_TOKEN=$(sed -n 's|^http://\([0-9a-f]\{40\}\)@10\.10\.20\.120.*|\1|p' \
"$HOME/.git-credentials" | head -1)
fi
if [ -z "${GITEA_TOKEN:-}" ]; then
GITEA_TOKEN=$(git -C "$HOME/Projects/Pixel3Arch" remote -v 2>/dev/null \
| sed -n 's|.*http://\([0-9a-f]\{40\}\)@.*|\1|p' | head -1)
fi
[ -n "${GITEA_TOKEN:-}" ] || { echo "ci-status: no token (set GITEA_TOKEN)" >&2; exit 2; }
api() { curl -sS --max-time 20 -H "Authorization: token $GITEA_TOKEN" "$GITEA/api/v1$1"; }
# Gitea ignores ?limit on this endpoint and returns the full history, so the
# slice happens here. Found the hard way: a "last 3 runs" call printed 120.
summarize() {
python3 -c '
import json, sys
limit = int(sys.argv[1])
try:
d = json.load(sys.stdin)
except Exception:
print("ERR"); raise SystemExit
runs = d.get("workflow_runs", d) if isinstance(d, dict) else d
if not runs:
print("NONE"); raise SystemExit
for r in runs[:limit]:
c = r.get("conclusion")
mark = "ok" if c == "success" else ("FAILED" if c == "failure" else (c or "running"))
print("DETAIL", r.get("id"), r.get("head_branch"), (r.get("head_sha") or "")[:8], mark)
n = runs[0]
print("TOP", n.get("id") if n.get("conclusion") == "failure" else "-")
' "$1"
}
check_repo() {
local repo="$1" limit="${2:-3}" out failed_run jobs failed_id
out=$(api "/repos/$OWNER/$repo/actions/runs" 2>/dev/null | summarize "$limit")
case "$out" in
ERR|NONE|"") printf 'ci-status: %-20s no Actions\n' "$repo"; return 0 ;;
esac
printf '%s\n' "$out" | awk -v r="$repo" '$1=="DETAIL"{
printf "ci-status: %-20s run %-5s [%-8s] %s -> %s\n", r, $2, $3, $4, $5 }'
failed_run=$(printf '%s\n' "$out" | awk '$1=="TOP" && $2!="-"{print $2}')
[ -n "$failed_run" ] || return 0
echo "ci-status: --- run $failed_run: which job ---"
jobs=$(api "/repos/$OWNER/$repo/actions/runs/$failed_run/jobs")
printf '%s' "$jobs" | python3 -c '
import json, sys
d = json.load(sys.stdin)
for j in d.get("jobs", d):
print(" %-22s %s" % (j.get("name"), j.get("conclusion")))
' 2>/dev/null
failed_id=$(printf '%s' "$jobs" | python3 -c '
import json, sys
d = json.load(sys.stdin)
for j in d.get("jobs", d):
if j.get("conclusion") == "failure":
print(j.get("id")); break
' 2>/dev/null)
if [ -n "$failed_id" ]; then
echo "ci-status: --- why ---"
api "/repos/$OWNER/$repo/actions/jobs/$failed_id/logs" 2>/dev/null \
| grep -viE "docker (pull|create|run|volume)|Writing entry|Extracting|cloning|Cleaning up|Created container|Start(ed|ing) container|Removed container|ENV ==>|Image exists" \
| tail -12 | sed 's/^/ /'
fi
return 1
}
# Listed, not discovered: a new repo should be a deliberate addition here, and
# a repo that disappears from the list should be noticed rather than silently
# stop being checked.
ALL_REPOS="souveraine Pixel3Arch culver souveraine-viewtop souveraine-player hexagonrpc souveraine-updater"
if [ "${1:-}" = "--all" ]; then
worst=0
for r in $ALL_REPOS; do check_repo "$r" "${2:-1}" || worst=1; done
exit $worst
fi
repo="${1:-}"
if [ -z "$repo" ]; then
url=$(git remote -v 2>/dev/null | awk 'NR==1{print $2}')
repo=$(basename "${url%.git}" 2>/dev/null)
fi
[ -n "$repo" ] || { echo "ci-status: no repo (pass a name)" >&2; exit 2; }
check_repo "$repo" "${2:-3}"

50
tools/claude-hook-push-ci.sh Executable file
View file

@ -0,0 +1,50 @@
#!/usr/bin/env bash
# Claude Code PostToolUse hook: after a `git push`, say what CI did.
#
# Wired in ~/.claude/settings.json as a PostToolUse hook on Bash. Reads the
# tool-call JSON on stdin, does nothing unless the command was a push, and
# otherwise prints the pushed repo's CI status so the answer arrives without
# anyone deciding to ask for it.
#
# WHY. A pinned action SHA rotted on 2026-07-28 and stopped `edge` publishing
# for a working day. Nothing failed loudly — the publish job was *skipped* —
# and the habit in place was to read the runner journal, which reports task
# pickup and never outcome. Remembering to check is not a mechanism; this is.
#
# WHAT IT CANNOT DO. A cross build takes ~20 minutes, so immediately after a
# push the honest answer is usually "running". That still catches the whole
# fail-fast class — pin rot, lint, attribution, a bad workflow edit — which is
# what actually bites. Anything slower needs the layer-2 staleness check in
# TASK-42, not a longer sleep here.
#
# Never fails the tool call: exit 0 always. A status reporter that can block a
# push is a worse problem than the one it solves.
set -uo pipefail
SELF_DIR="$(dirname "$(realpath "$0")")"
payload=$(cat 2>/dev/null || true)
cmd=$(printf '%s' "$payload" | python3 -c '
import json, sys
try:
print(json.load(sys.stdin).get("tool_input", {}).get("command", ""))
except Exception:
pass
' 2>/dev/null)
case "$cmd" in
*"git push"*) ;;
*) exit 0 ;;
esac
# Which repo. The push may have run in any directory, so prefer the cwd's
# remote and fall back to whatever the command named.
repo=""
url=$(git remote -v 2>/dev/null | awk 'NR==1{print $2}')
[ -n "$url" ] && repo=$(basename "${url%.git}")
[ -n "$repo" ] || exit 0
echo "--- CI (auto, after git push) ---"
"$SELF_DIR/ci-status.sh" "$repo" 2 2>&1 | head -30
echo "--- a build in progress needs a second look; see TASK-42 ---"
exit 0