SouveraineOS/docs/substrate/NET_KERNEL_GAPS.md

SouveraineOS Net Layer — Kernel Gaps & Bugs to Fix

Status: for review. Every kernel-config gap and known ceiling/bug blocking the networking layer, in one place. Fix these in linux-blueline (Pixel3Arch), rebuild, reflash.

Verified live on the Pixel 3 (blueline), kernel 7.1.1-sdm845-g5199244ee0ee, 2026-07-17. Config snapshot: /tmp/blueline-kernel-config.txt (205 lines).

Context: see docs/NET_LAYER_PLAN.md for the layer design this enables.

---

Why these exist (not a bug, a lean build)

The sdm845 defconfig and pmOS APKBUILDs build the kernel lean — debug info stripped, optional netfilter/tc classifiers off. Same class of cause for all of them. One rebuild fixes the cluster.

---

Config gaps — flip these to =y

BTF / CO-RE (the one that started this)

The BPF subsystem is fully enabled (CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT, CONFIG_CGROUP_BPF, CONFIG_BPF_LSM all =y). But /sys/kernel/btf/vmlinux is absent, which blocks CO-RE BPF (compile-once programs that relocate against the running kernel's structs).

BTF is a generated artifact, not a switch. pahole (pkg dwarves) must run at build time over DWARF debug info. The chain is currently broken in two places:

Symbol	Current	Set to	Why
CONFIG_DEBUG_INFO	=y ✓	keep	base requirement — already on
CONFIG_DEBUG_INFO_REDUCED	=y	=n	reduced debug info strips what pahole needs to emit BTF — this is the hidden second blocker
CONFIG_DEBUG_INFO_BTF	(absent — option not selectable)	=y	the actual BTF emitter; only becomes selectable once the two above are satisfied
CONFIG_DEBUG_INFO_DWARF5	not set	=y (preferred)	modern; matches pahole expectations

Build-system side (Pixel3Arch linux-blueline APKBUILD / makepkg):

• Add dwarves to makedepends. Without pahole in the build chroot, the BTF step silently no-ops even with the config correct.
• Verify pahole --version ≥ 1.16 in the chroot.

Decision (Casey, 2026-07-17): rebuild kernel with BTF rather than write CO-RE-free programs. Unlocks full CO-RE — richer BPF, struct walks, kprobes.

Netfilter mark + BPF-match (the enforcement edge)

nft the binary is installed, but the kernel modules it would load for our "stamp fwmark → nft enforces" path are not built:

Symbol	Current	Set to	Why
CONFIG_NETFILTER_XT_TARGET_MARK	not set	=y	set fwmark on packets in netfilter (the nft meta mark set path needs this)
CONFIG_NETFILTER_XT_MATCH_BPF	not set	=y	match packets by BPF verdict in netfilter
CONFIG_NETFILTER_XT_TARGET_CONNMARK	not set	=y	carry consumer identity on the conntrack flow (CLAT/NAT survival)

tc classifier / action (optional second enforcement path)

Symbol	Current	Set to	Why
CONFIG_NET_CLS_BPF	not set	=y	classify traffic by BPF at the tc layer (egress Qdisc)
CONFIG_NET_CLS_ACT	not set	=y	tc actions (needed for cls_bpf to deny/redirect)
CONFIG_NET_ACT_BPF	not set	=y	BPF tc action
CONFIG_NFT_SOCKET	not set	=y	nft socket match — match on the originating socket's cgroup/mark
CONFIG_NFT_TUNNEL	not set	=y	if we ever need nft-side tunnel matching for CLAT

conntrack zones — the CLAT ceiling (severity TBD, see Bugs §1)

Symbol	Current	Set to	Why
CONFIG_NF_CONNTRACK_ZONES	not set	=y	run multiple independent conntrack tables — required to carry a consumer's policy identity through CLAT (v4→v6 translation re-originates packets). NF_CONNTRACK_MARK=y is on but mark-without-zones caps the multi-flow translation case.

---

The spine IS present (no fix needed)

These are on, and they're the load-bearing part of per-consumer routing:

CONFIG_IP_ADVANCED_ROUTER=y       ← ip rule + policy routing
CONFIG_IP_MULTIPLE_TABLES=y       ← per-link routing tables (v4)
CONFIG_IPV6_MULTIPLE_TABLES=y     ← per-link routing tables (v6) — critical for LTE
CONFIG_NF_CONNTRACK_MARK=y        ← marks on conntrack flows
CONFIG_BPF_LSM=y  CONFIG_CGROUP_BPF=y   ← attach points
CONFIG_NFT_CT=m  CONFIG_NFT_FIB_IPV4=m  CONFIG_NFT_FIB_IPV6=m

So fwmark → ip rule → per-link table steering works for both IPv4 and IPv6 as-is. The gaps are the mark-stamping and matching edges, not the steering.

---

Ceilings — verdicts (sourced)

The "is cgroup-bpf+fwmark+ip-rule actually sufficient?" questions, resolved. Sources inline. Severity is honest, not optimistic.

1. CLAT (464XLAT) translation obscures consumer identity — REAL, MANAGEABLE

LTE on this device is IPv6-only; clat (tun) synthesizes the IPv4 path by re-packaging v4 packets as v6. The concern: does a consumer's fwmark survive the v4→v6 translation?

Verdict — the mark survives for routing, if we set it the right way:

• CGROUP_SOCK_ADDR (attach cgroup/connect4/connect6) fires at connect() and at UDP sendto() (implicit connect) — before the packet enters the network stack. The program calls bpf_setsockopt(SO_MARK) on the socket. This is the Android per-app-routing pattern, confirmed across docs. (ebpf.io CGROUP_SOCK_ADDR)
• A SO_MARK set this way applies to the original socket, so the routing decision (ip rule fwmark → per-link table) is made at egress before CLAT commits to a path. The kernel's policy routing runs on the skb mark, which translation at the IP layer does not strip. This is the same mechanism WireGuard uses to keep envelope traffic out of its own routing loop. (Server Fault — FwMark routing)
• Where CLAT still bites: the translated IPv6 flow enters conntrack as a new flow. For per-consumer accounting/metering (byte budgets) across the translation, the mark must reach the conntrack entry — that needs NF_CONNTRACK_MARK (on) and NF_CONNTRACK_ZONES (off — see config gap) so the translated flow can be associated with the consumer's zone. (LWN — conntrack zones)

So: steering (wifi-only consumer's packets don't egress LTE) works without zones, because the mark is set on the socket pre-translation and policy routing honors it. Metering across CLAT needs zones. Both are in the rebuild list. NetworkManager itself now uses eBPF to support CLAT/IPv6-mostly (Red Hat), so we're on a well-trodden path, not inventing.

2. cgroup namespace + Waydroid/LXC — host attach WORKS — NON-ISSUE if done right

Verdict — host-side attach at the container's parent cgroup classifies all container sockets, namespace or not:

• cgroup-attached BPF programs are inherited by descendant cgroups and fire for every process in the cgroup subtree regardless of the process's other namespace memberships. A host-side attach at the cgroup containing the LXC container applies to Waydroid processes inside it. (kernel-internals.org — cgroup BPF effective programs, eunomia — cgroup policy control)
• The gotcha is the reverse — attaching from inside an unprivileged LXC container needs CAP_BPF/CAP_SYS_ADMIN and the host's cgroup view, which unprivileged containers lack. We don't do that. We attach from souveraine-netd on the host at the Waydroid container's cgroup. Whole- container policy works; per-Android-app needs Waydroid's per-app cgroups surfaced to the host (deferred per Casey, 2026-07-17). (bpfman — attaching eBPF in containers)

3. CGROUP_SOCK_ADDR scope — outbound + bind, NOT inbound accept — KNOWN LIMITATION

Verdict — covers everything we route, by design:

Syscall	Hook	Use
connect(2) TCP	connect4/connect6	outbound ✓
sendto(2) unconnected UDP	connect4/connect6 (implicit)	outbound ✓
bind(2)	bind4/bind6	servers binding a port
getpeername/getsockname	respective hooks	either

There is no TCP accept() hook in CGROUP_SOCK_ADDR. Inbound socket routing (steering an incoming connection to a listener) needs the separate BPF_SK_LOOKUP program — and only fires for traffic with no established socket, so it does not help connected flows. (Medium — sk_lookup)

So what: per-consumer routing is outbound-only, which is exactly the product (we decide where our traffic egresses). Inbound federation is not applicable on LTE/CGNAT anyway (no inbound path). Servers binding ports get bind hooks. This is not a blocker — it's the correct scope. Raw sockets and AF_UNIX are out of scope (we don't route them).

4. Process-tree propagation — WORKS — NON-ISSUE

If the substrate's bash tool runs curl, curl inherits bash's cgroup on fork/exec. cgroup-bpf fires on every socket any process in the cgroup subtree creates. The model is inheritance-based, so shelled-out children are classified by descent, not by re-enrollment. Confirmed by the same descendant-inheritance docs as §2.

---

Runtime knobs to set (not config, just sysctl)

• net.ipv4.ip_forward = 0 → likely 1 (CLAT + host-side rerouting may need forwarding on). Verify before flipping — security implication.
• rp_filter = 0 is fine (loose); actually helps multi-link. Leave it.
• bpffs is mounted (/sys/fs/bpf, mode 700) — good. Pin programs there so they survive loader exit (relevant to the standing-vs-short-lived-daemon decision).

---

The consolidated rebuild (one kernel, one flash)

All of the above collapses into a single linux-blueline rebuild:

defconfig additions:

CONFIG_DEBUG_INFO_REDUCED=n          # was =y — hidden BTF blocker
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_INFO_BTF=y              # was absent
CONFIG_NETFILTER_XT_TARGET_MARK=y    # was not set
CONFIG_NETFILTER_XT_MATCH_BPF=y      # was not set
CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
CONFIG_NET_CLS_BPF=y                 # was not set
CONFIG_NET_CLS_ACT=y                 # was not set
CONFIG_NET_ACT_BPF=y                 # was not set
CONFIG_NF_CONNTRACK_ZONES=y          # was not set — CLAT ceiling
CONFIG_NFT_SOCKET=y
CONFIG_NFT_TUNNEL=y

APKBUILD/makepkg: add dwarves to makedepends; verify pahole ≥ 1.16.

Floor doctrine (from RedFlag): document this kernel version + config as the net layer's required floor. souveraine-netd refuses to attach if the floor isn't met (BTF absent, mark target missing), rather than silently running unenforced. Same discipline as RedFlag's "don't silently add CAP_BPF; document the kernel version floor."

Ignore-list (present but useless — don't be fooled):

• CONFIG_NET_CLS_CGROUP=m — the old cgroup-v1 net classifier. v1-only; does nothing on this cgroup-v2-only kernel. Not a path.

---

Prior art — are we reinventing?

Partially. The mechanisms exist; the product (per-app routing as a user-facing, agent-controllable policy layer on mobile Linux) does not.

• Android bpfloader / NETD — Android's own cgroup-bpf + fwmark + per-UID routing. Our mechanism is essentially the Android pattern lifted onto plain Linux. The difference: Android expresses it through ConnectivityManager / NetworkCapabilities inside the framework; we expose it as a declarative policy surface the agent and user can mutate. The mechanism is proven at billion-device scale — strong validation that cgroup-bpf+fwmark+ip-rule is the right substrate.
• Cilium — full BPF datapath for cluster network policy. Far heavier than we need (Kubernetes-oriented, XDP/TC datapath, per-endpoint identity via kube-apiserver). Conceptually adjacent; not a dependency, not something to copy wholesale. Useful as reference for BPF map designs and conntrack handling.
• bpfd / bpfman — a BPF program daemon for loading/attaching eBPF on a host. Closest in shape to what souveraine-netd would be (privileged daemon managing BPF attach lifecycle). Worth studying for the daemon-design question (standing vs short-lived — bpfman is itself a standing daemon, which is against RedFlag's doctrine; tension noted).
• NetworkManager eBPF for CLAT/IPv6-mostly — NM now uses eBPF to support CLAT. Confirms the path we're on is mainstream, and that NM is a peer component, not something to replace.

Verdict: we are not reinventing the mechanism. We're productizing it for the mobile-Linux sovereign-agent context — declarative policy, agent step-up mutation, consumer model — none of which the existing tools provide. Reuse the kernel mechanisms and the Android pattern; build the policy/product layer.

Honest ceiling statement

What cgroup-bpf + fwmark + ip-rule does NOT give us, and what's required beyond it for the real per-app-routing product:

1. Inbound connection routing — out of scope for cgroup-bpf (no accept hook; sk_lookup only for no-socket traffic). Acceptable: the product is outbound egress policy, and LTE/CGNAT has no inbound path anyway.
2. Cross-translation metering without conntrack zones — steering works pre-translation; byte accounting across CLAT needs NF_CONNTRACK_ZONES. In the rebuild list. Until then, metering is accurate for native-v6/v4 flows and approximate for CLAT-translated v4-over-v6.
3. Per-Android-app granularity inside Waydroid — whole-container works; per-app needs Waydroid's per-app cgroups surfaced to the host. Deferred until Waydroid is installed (Casey, 2026-07-17). The cgroup-subtree layout is designed so a per-app subtree can be carved later without rework.
4. Anything requiring BPF-LSM rather than cgroup-bpf — none identified. CONFIG_BPF_LSM=y is on and available if a future policy needs an LSM hook (e.g., deny-socket-creation by cgroup), but the current design doesn't require it. Not a ceiling today; a reserved option.

Net: with the kernel rebuild (BTF + netfilter mark/match + conntrack zones + tc-cls), cgroup-bpf+fwmark+ip-rule is sufficient to deliver the per-app-routing product for native and whole-container consumers, outbound, with accurate-enough metering. The remaining ceilings are either correctly out of scope (inbound), deferred by external dependency (Waydroid per-app), or non-issues done the documented way (container namespace, process trees, CLAT steering).