SouveraineOS/docs/substrate/NET_KERNEL_GAPS.md

# SouveraineOS Net Layer — Kernel Gaps & Bugs to Fix

Status: for review. Every kernel-config gap and known ceiling/bug blocking the
networking layer, in one place. Fix these in `linux-blueline` (Pixel3Arch),
rebuild, reflash.

Verified live on the Pixel 3 (blueline), kernel `7.1.1-sdm845-g5199244ee0ee`,
2026-07-17. Config snapshot: `/tmp/blueline-kernel-config.txt` (205 lines).

Context: see `docs/NET_LAYER_PLAN.md` for the layer design this enables.

---

## Why these exist (not a bug, a lean build)

The sdm845 defconfig and pmOS APKBUILDs build the kernel lean — debug info
stripped, optional netfilter/tc classifiers off. Same class of cause for all
of them. One rebuild fixes the cluster.

---

## Config gaps — flip these to =y

### BTF / CO-RE (the one that started this)

The BPF subsystem is fully enabled (`CONFIG_BPF`, `CONFIG_BPF_SYSCALL`,
`CONFIG_BPF_JIT`, `CONFIG_CGROUP_BPF`, `CONFIG_BPF_LSM` all =y). But
`/sys/kernel/btf/vmlinux` is **absent**, which blocks CO-RE BPF (compile-once
programs that relocate against the running kernel's structs).

BTF is a *generated artifact*, not a switch. `pahole` (pkg `dwarves`) must
run at build time over DWARF debug info. The chain is currently broken in
*two* places:

| Symbol | Current | Set to | Why |
|---|---|---|---|
| `CONFIG_DEBUG_INFO` | `=y` ✓ | keep | base requirement — already on |
| `CONFIG_DEBUG_INFO_REDUCED` | `=y` | **`=n`** | reduced debug info strips what `pahole` needs to emit BTF — this is the hidden second blocker |
| `CONFIG_DEBUG_INFO_BTF` | (absent — option not selectable) | **`=y`** | the actual BTF emitter; only becomes selectable once the two above are satisfied |
| `CONFIG_DEBUG_INFO_DWARF5` | not set | `=y` (preferred) | modern; matches pahole expectations |

**Build-system side** (Pixel3Arch `linux-blueline` APKBUILD / makepkg):
- Add `dwarves` to `makedepends`. Without `pahole` in the build chroot, the
  BTF step silently no-ops even with the config correct.
- Verify `pahole --version` ≥ 1.16 in the chroot.

Decision (Casey, 2026-07-17): rebuild kernel with BTF rather than write
CO-RE-free programs. Unlocks full CO-RE — richer BPF, struct walks, kprobes.

### Netfilter mark + BPF-match (the enforcement edge)

`nft` the binary is installed, but the kernel modules it would load for our
"stamp fwmark → nft enforces" path are **not built**:

| Symbol | Current | Set to | Why |
|---|---|---|---|
| `CONFIG_NETFILTER_XT_TARGET_MARK` | not set | **`=y`** | set fwmark on packets in netfilter (the nft `meta mark set` path needs this) |
| `CONFIG_NETFILTER_XT_MATCH_BPF` | not set | **`=y`** | match packets by BPF verdict in netfilter |
| `CONFIG_NETFILTER_XT_TARGET_CONNMARK` | not set | **`=y`** | carry consumer identity on the conntrack flow (CLAT/NAT survival) |

### tc classifier / action (optional second enforcement path)

| Symbol | Current | Set to | Why |
|---|---|---|---|
| `CONFIG_NET_CLS_BPF` | not set | **`=y`** | classify traffic by BPF at the tc layer (egress Qdisc) |
| `CONFIG_NET_CLS_ACT` | not set | **`=y`** | tc actions (needed for cls_bpf to deny/redirect) |
| `CONFIG_NET_ACT_BPF` | not set | **`=y`** | BPF tc action |
| `CONFIG_NFT_SOCKET` | not set | `=y` | nft `socket` match — match on the originating socket's cgroup/mark |
| `CONFIG_NFT_TUNNEL` | not set | `=y` | if we ever need nft-side tunnel matching for CLAT |

### conntrack zones — the CLAT ceiling (severity TBD, see Bugs §1)

| Symbol | Current | Set to | Why |
|---|---|---|---|
| `CONFIG_NF_CONNTRACK_ZONES` | not set | **`=y`** | run multiple independent conntrack tables — required to carry a consumer's policy identity through CLAT (v4→v6 translation re-originates packets). `NF_CONNTRACK_MARK=y` is on but mark-without-zones caps the multi-flow translation case. |

---

## The spine IS present (no fix needed)

These are on, and they're the load-bearing part of per-consumer routing:

```
CONFIG_IP_ADVANCED_ROUTER=y       ← ip rule + policy routing
CONFIG_IP_MULTIPLE_TABLES=y       ← per-link routing tables (v4)
CONFIG_IPV6_MULTIPLE_TABLES=y     ← per-link routing tables (v6) — critical for LTE
CONFIG_NF_CONNTRACK_MARK=y        ← marks on conntrack flows
CONFIG_BPF_LSM=y  CONFIG_CGROUP_BPF=y   ← attach points
CONFIG_NFT_CT=m  CONFIG_NFT_FIB_IPV4=m  CONFIG_NFT_FIB_IPV6=m
```

So fwmark → `ip rule` → per-link table steering works for both IPv4 and IPv6
as-is. The gaps are the *mark-stamping and matching edges*, not the steering.

---

## Ceilings — verdicts (sourced)

> The "is cgroup-bpf+fwmark+ip-rule actually sufficient?" questions, resolved.
> Sources inline. Severity is honest, not optimistic.

### 1. CLAT (464XLAT) translation obscures consumer identity — REAL, MANAGEABLE

LTE on this device is **IPv6-only**; `clat` (tun) synthesizes the IPv4 path by
re-packaging v4 packets as v6. The concern: does a consumer's fwmark survive
the v4→v6 translation?

**Verdict — the mark survives for routing, if we set it the right way:**

- `CGROUP_SOCK_ADDR` (attach `cgroup/connect4`/`connect6`) fires at `connect()`
  and at UDP `sendto()` (implicit connect) — *before* the packet enters the
  network stack. The program calls `bpf_setsockopt(SO_MARK)` on the socket.
  This is the Android per-app-routing pattern, confirmed across docs.
  ([ebpf.io CGROUP_SOCK_ADDR](https://docs.ebpf.io/linux/program-type/BPF_PROG_TYPE_CGROUP_SOCK_ADDR/))
- A `SO_MARK` set this way applies to the **original socket**, so the routing
  decision (`ip rule fwmark` → per-link table) is made at egress **before**
  CLAT commits to a path. The kernel's policy routing runs on the skb mark,
  which translation at the IP layer does not strip. This is the same mechanism
  WireGuard uses to keep envelope traffic out of its own routing loop.
  ([Server Fault — FwMark routing](https://serverfault.com/questions/1144215/prevent-routing-loop-with-fwmark-in-wireguard))
- **Where CLAT still bites:** the translated IPv6 flow enters conntrack as a
  *new* flow. For per-consumer *accounting/metering* (byte budgets) across the
  translation, the mark must reach the conntrack entry — that needs
  `NF_CONNTRACK_MARK` (on) **and `NF_CONNTRACK_ZONES` (off — see config gap)**
  so the translated flow can be associated with the consumer's zone.
  ([LWN — conntrack zones](https://lwn.net/Articles/371028/))

**So:** steering (wifi-only consumer's packets don't egress LTE) works without
zones, because the mark is set on the socket pre-translation and policy routing
honors it. *Metering* across CLAT needs zones. Both are in the rebuild list.
NetworkManager itself now uses eBPF to support CLAT/IPv6-mostly
([Red Hat](https://developers.redhat.com/articles/2026/07/08/networkmanager-supports-ipv6-mostly)),
so we're on a well-trodden path, not inventing.

### 2. cgroup namespace + Waydroid/LXC — host attach WORKS — NON-ISSUE if done right

**Verdict — host-side attach at the container's parent cgroup classifies all
container sockets, namespace or not:**

- cgroup-attached BPF programs are **inherited by descendant cgroups** and fire
  for every process in the cgroup subtree regardless of the process's other
  namespace memberships. A host-side attach at the cgroup containing the LXC
  container applies to Waydroid processes inside it.
  ([kernel-internals.org — cgroup BPF effective programs](https://kernel-internals.org/cgroups/cgroup-bpf/),
  [eunomia — cgroup policy control](https://eunomia.dev/tutorials/cgroup/))
- The gotcha is the reverse — *attaching from inside* an unprivileged LXC
  container needs `CAP_BPF`/`CAP_SYS_ADMIN` and the host's cgroup view, which
  unprivileged containers lack. **We don't do that.** We attach from
  `souveraine-netd` on the host at the Waydroid container's cgroup. Whole-
  container policy works; per-Android-app needs Waydroid's per-app cgroups
  surfaced to the host (deferred per Casey, 2026-07-17).
  ([bpfman — attaching eBPF in containers](https://bpfman.io/v0.5.4/blog/2024/02/26/technical-challenges-for-attaching-ebpf-programs-in-containers/))

### 3. CGROUP_SOCK_ADDR scope — outbound + bind, NOT inbound accept — KNOWN LIMITATION

**Verdict — covers everything we route, by design:**

| Syscall | Hook | Use |
|---|---|---|
| `connect(2)` TCP | `connect4`/`connect6` | outbound ✓ |
| `sendto(2)` unconnected UDP | `connect4`/`connect6` (implicit) | outbound ✓ |
| `bind(2)` | `bind4`/`bind6` | servers binding a port |
| `getpeername`/`getsockname` | respective hooks | either |

There is **no TCP `accept()` hook** in `CGROUP_SOCK_ADDR`. Inbound socket
routing (steering an *incoming* connection to a listener) needs the separate
`BPF_SK_LOOKUP` program — and only fires for traffic with no established
socket, so it does not help connected flows.
([Medium — sk_lookup](https://medium.com/all-things-ebpf/ebpf-sk-lookup-socket-lookup-and-redirection-08643062fab2))

**So what:** per-consumer routing is **outbound-only**, which is exactly the
product (we decide where *our* traffic egresses). Inbound federation is not
applicable on LTE/CGNAT anyway (no inbound path). Servers binding ports get
`bind` hooks. This is not a blocker — it's the correct scope. Raw sockets and
`AF_UNIX` are out of scope (we don't route them).

### 4. Process-tree propagation — WORKS — NON-ISSUE

If the substrate's bash tool runs `curl`, curl inherits bash's cgroup on
fork/exec. cgroup-bpf fires on every socket any process **in the cgroup
subtree** creates. The model is inheritance-based, so shelled-out children
are classified by descent, not by re-enrollment. Confirmed by the same
descendant-inheritance docs as §2.

---

## Runtime knobs to set (not config, just sysctl)

- `net.ipv4.ip_forward = 0` → likely **`1`** (CLAT + host-side rerouting may
  need forwarding on). Verify before flipping — security implication.
- `rp_filter = 0` is fine (loose); actually *helps* multi-link. Leave it.
- bpffs is mounted (`/sys/fs/bpf`, mode 700) — good. Pin programs there so
  they survive loader exit (relevant to the standing-vs-short-lived-daemon
  decision).

---

## The consolidated rebuild (one kernel, one flash)

All of the above collapses into a single `linux-blueline` rebuild:

**defconfig additions:**
```
CONFIG_DEBUG_INFO_REDUCED=n          # was =y — hidden BTF blocker
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_DEBUG_INFO_BTF=y              # was absent
CONFIG_NETFILTER_XT_TARGET_MARK=y    # was not set
CONFIG_NETFILTER_XT_MATCH_BPF=y      # was not set
CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
CONFIG_NET_CLS_BPF=y                 # was not set
CONFIG_NET_CLS_ACT=y                 # was not set
CONFIG_NET_ACT_BPF=y                 # was not set
CONFIG_NF_CONNTRACK_ZONES=y          # was not set — CLAT ceiling
CONFIG_NFT_SOCKET=y
CONFIG_NFT_TUNNEL=y
```

**APKBUILD/makepkg:** add `dwarves` to makedepends; verify `pahole` ≥ 1.16.

**Floor doctrine (from RedFlag):** document this kernel version + config as
the net layer's required floor. `souveraine-netd` refuses to attach if the
floor isn't met (BTF absent, mark target missing), rather than silently
running unenforced. Same discipline as RedFlag's "don't silently add CAP_BPF;
document the kernel version floor."

**Ignore-list (present but useless — don't be fooled):**
- `CONFIG_NET_CLS_CGROUP=m` — the old cgroup-v1 net classifier. v1-only;
  does nothing on this cgroup-v2-only kernel. Not a path.

---

## Prior art — are we reinventing?

Partially. The *mechanisms* exist; the *product* (per-app routing as a
user-facing, agent-controllable policy layer on mobile Linux) does not.

- **Android bpfloader / NETD** — Android's own cgroup-bpf + fwmark + per-UID
  routing. Our mechanism is essentially the Android pattern lifted onto plain
  Linux. The difference: Android expresses it through `ConnectivityManager` /
  `NetworkCapabilities` inside the framework; we expose it as a declarative
  policy surface the agent and user can mutate. The mechanism is proven at
  billion-device scale — strong validation that cgroup-bpf+fwmark+ip-rule is
  the right substrate.
- **Cilium** — full BPF datapath for cluster network policy. Far heavier than
  we need (Kubernetes-oriented, XDP/TC datapath, per-endpoint identity via
  kube-apiserver). Conceptually adjacent; not a dependency, not something to
  copy wholesale. Useful as reference for BPF map designs and conntrack
  handling.
- **bpfd / bpfman** — a BPF program daemon for loading/attaching eBPF on a
  host. Closest in *shape* to what `souveraine-netd` would be (privileged
  daemon managing BPF attach lifecycle). Worth studying for the daemon-design
  question (standing vs short-lived — bpfman is itself a standing daemon,
  which is *against* RedFlag's doctrine; tension noted).
- **NetworkManager eBPF for CLAT/IPv6-mostly** — NM now uses eBPF to support
  CLAT. Confirms the path we're on is mainstream, and that NM is a peer
  component, not something to replace.

**Verdict:** we are not reinventing the mechanism. We're productizing it for
the mobile-Linux sovereign-agent context — declarative policy, agent step-up
mutation, consumer model — none of which the existing tools provide. Reuse the
kernel mechanisms and the Android pattern; build the policy/product layer.

## Honest ceiling statement

What cgroup-bpf + fwmark + ip-rule does NOT give us, and what's required
beyond it for the real per-app-routing product:

1. **Inbound connection routing** — out of scope for cgroup-bpf (no `accept`
   hook; `sk_lookup` only for no-socket traffic). Acceptable: the product is
   outbound egress policy, and LTE/CGNAT has no inbound path anyway.
2. **Cross-translation metering without conntrack zones** — steering works
   pre-translation; *byte accounting* across CLAT needs `NF_CONNTRACK_ZONES`.
   In the rebuild list. Until then, metering is accurate for native-v6/v4
   flows and approximate for CLAT-translated v4-over-v6.
3. **Per-Android-app granularity inside Waydroid** — whole-container works;
   per-app needs Waydroid's per-app cgroups surfaced to the host. Deferred
   until Waydroid is installed (Casey, 2026-07-17). The cgroup-subtree layout
   is designed so a per-app subtree can be carved later without rework.
4. **Anything requiring BPF-LSM rather than cgroup-bpf** — none identified.
   `CONFIG_BPF_LSM=y` is on and available if a future policy needs an LSM hook
   (e.g., deny-socket-creation by cgroup), but the current design doesn't
   require it. Not a ceiling today; a reserved option.

**Net:** with the kernel rebuild (BTF + netfilter mark/match + conntrack
zones + tc-cls), cgroup-bpf+fwmark+ip-rule is **sufficient** to deliver the
per-app-routing product for native and whole-container consumers, outbound,
with accurate-enough metering. The remaining ceilings are either correctly
out of scope (inbound), deferred by external dependency (Waydroid per-app),
or non-issues done the documented way (container namespace, process trees,
CLAT steering).