SouveraineOS/docs/substrate/NET_LAYER_PLAN.md

SouveraineOS Networking Layer

Status: design, under discussion. Not yet building. Target: the product, not a v1. May not compile on first run; that's fine.

The product

A machine — laptop or phone — where every consumer of the network (the substrate's own federation/sync/voice sockets; system services; a Waydroid Android instance and its apps; an arbitrary shelled-out process) is governed by an explicit per-consumer network policy, enforced at the right kernel/ userspace layer for what it is. Default baseline is "behave like a normal Linux host"; policy is an override. The user and the agent both express it; the substrate enforces it.

Mobile Linux has not solved this. We are.

What's live on the device (verified 2026-07-17)

Two data links, NM-tracked, correct metrics:

• wlan0 wifi — IPv4, default route metric 600
• qrtr0 gsm "Fido LTE" — IPv6-only, global v6 on qmapmux0.0, no native v4
• clat tun — 464XLAT IPv4-over-IPv6, MTU 1260

NetworkManager + ModemManager active and correct. Kernel prefers WiFi via metric — we READ priority, never implement it.

Kernel facts that drove the design (not assumptions)

• cgroup v2 only (cgroup2fs). Mounted controllers: cpuset cpu io memory hugetlb pids. No net_cls, no net_prio — those are v1-only, deliberately absent in v2.
• nft, iptables, ip present. No libcgroup userspace (not needed on v2).
• Conclusion: the textbook "net_cls.classid + nftables match" recipe is dead on this kernel. Per-process classification must be cgroup-bpf (CGROUP_SKB / CGROUP_SOCK_ADDR) stamping a socket mark, with ip rule
  • per-link routing tables (and/or nft mark-match) doing the enforcement.
• Waydroid not yet installed — it's future alongside-software, governed as a classified consumer class, not integrated via binder.

Architecture — two regimes, one model

                    Consumer  (who)
                       │  has a  LinkPolicy  (which links, with what constraints)
                       │  enforced via an  Enforcement  (how)
                       ▼
   ┌───────────────────────────────┬───────────────────────────────┐
   │  Substrate-socket regime      │  Kernel-classification regime  │
   │  (userspace, no privilege)    │  (privileged: cgroup-bpf+nft)  │
   │                               │                                │
   │  federation WS                │  Waydroid instance + its apps  │
   │  memfs git sync               │  system services (systemd units)│
   │  voice STT/TTS HTTP           │  arbitrary shelled-out procs   │
   │  any socket WE open           │  any socket we DIDN'T open     │
   │                               │                                │
   │  SO_BINDTODEVICE / bind()     │  cgroup dir + pid write +      │
   │  before connect()             │  BPF mark + ip rule per link   │
   │  + re-resolve on handoff      │  + nft allow/deny/mark-match   │
   └───────────────────────────────┴───────────────────────────────┘
                       ▲
              LinkState (authoritative, from NM/MM D-Bus)
              owned by machined::net, same as before

Why two regimes, not a choice: the substrate's own connections NEED userspace control — only the substrate can re-resolve peer DNS on egress change and reconnect its WS with the right semantics; a kernel policy layer can't express that. Arbitrary processes NEED kernel classification — only the kernel sees sockets the substrate never opened. These aren't competing approaches; they're the correct tools for two disjoint categories of traffic, unified by one Consumer/LinkPolicy model.

The shared model (the spine)

/// A physical or virtual link, as the kernel + NM know it.
pub struct Link {
    pub device: String,            // "wlan0", "qrtr0", "clat"
    pub kind: LinkKind,            // Wifi | Cellular | Wired | Tun
    pub metered: bool,
    pub v4: bool,
    pub v6: bool,
    pub default: bool,             // holds the lowest-metric default route
    pub source_v4: Option<Ipv4Addr>,  // bind address for SO_BINDTODEVICE fallback
    pub source_v6: Option<Ipv6Addr>,
    pub signal: Option<u8>,
    pub roaming: bool,             // MM AccessTechnologies → guess
}

/// Which links a consumer may use, and how. The primary type.
pub struct LinkPolicy {
    pub allow: LinkSet,            // explicit allow-list of link constraints
    pub deny: LinkSet,             // explicit deny-list (wins over allow)
    pub max_metered: Option<DataBudget>, // cap cellular bytes per period
}

pub enum LinkSet {
    /// Match by capability, not by device name — survives interface rename.
    AnyMatching { kind: Option<LinkKind>, metered: Option<bool>,
                  v4: Option<bool>, v6: Option<bool>, roaming: Option<bool> },
    /// Pin to a named device (rare; for diagnostics).
    Device(String),
    /// The kernel default route — the "no policy" baseline.
    DefaultRoute,
}

/// Who the policy applies to.
pub struct Consumer {
    pub id: ConsumerId,
    pub policy: LinkPolicy,
    pub enforcement: Enforcement,
}

pub enum ConsumerId {
    /// A named substrate component we author.
    Component(&'static str),       // "federation", "memfs-sync", "voice-stt"
    /// A cgroup path — matches any process placed there.
    Cgroup(String),                // "/souveraine/waydroid", "/souveraine/web"
    /// A systemd unit we'll enroll.
    Unit(String),
}

pub enum Enforcement {
    /// Substrate opens the socket itself and binds it. No privilege.
    SubstrateSocket,
    /// Kernel enforces via cgroup-bpf mark + ip rule + nft. Privileged.
    KernelClassify { cgroup: String, fwmark: u32 },
}

LinkClass (Any/Validated/Unmetered/Bulk) survives as a convenience derived from LinkPolicy + live LinkState — it's how a call-site asks "does my policy permit a Bulk operation right now" without re-stating constraints. It is NOT the primary type; the primary type is the explicit LinkPolicy.

Tier 1 — machined::net (link truth)

Unchanged from the earlier sketch: NM/MM D-Bus reads via the existing zbus dep, builds LinkState, serves it on the guarded ok/reason socket as NetState. std-only, synchronous, audited, SO_PEERCRED-logged — same discipline as the rest of machined. LinkState is the input to everything.

Add: a LinkPolicy store (where per-consumer policies live) also served via machined, since policy is machine-level truth, not agent-level.

Tier 2 — core/net (the policy engine)

In-process. Holds the current LinkState (from machined, or direct NM D-Bus on desktop). The engine:

1. Resolves a Consumer's LinkPolicy against live LinkState → the set of currently-usable Links (or: "deny, no matching link" / "deferred, only metered available and budget exhausted").
2. Exposes tokio::sync::watch of (LinkState, resolved-policies) so consumers react to handoff.
3. Provides the substrate-socket bind helper: bind_socket_to(link: &Link) — SO_BINDTODEVICE, falling back to bind(source_addr) — used by federation/sync/voice before connect().
4. Provides enroll(consumer) for KernelClassify consumers: create the cgroup dir, attach the BPF program, install the ip rules + nft rules.

Tier 3 — the kernel-classification regime (the hard, real part)

This is what makes it a product instead of a library.

Mechanism (cgroup v2 + BPF, since net_cls is gone)

1. cgroup layout under /sys/fs/cgroup/souveraine.net/<consumer>/. Each KernelClassify consumer gets a dir; processes are enrolled by writing their pid to cgroup.procs. Waydroid's container init goes into /souveraine.net/waydroid/; its apps inherit.
2. BPF classifier: a BPF_PROG_TYPE_CGROUP_SKB (egress) and/or CGROUP_SOCK_ADDR (connect) program attached to each consumer cgroup. It stamps the socket's skb / socket with a fwmark identifying the consumer + its current policy verdict. Verdict recomputed on LinkState change (userspace pushes a new map; BPF reads the map).
3. Routing: per-link routing tables (wifi in table 100, lte in table 200, clat in table 300) + ip rule entries keyed on fwmark → table. A socket marked "wifi-only" routes through table 100 regardless of the default. A consumer with no policy gets the default route (baseline behavior).
4. Allow/deny: nftables matches fwmark + egress interface → accept or drop. A wifi-only consumer attempting to egress on qrtr0 is dropped.
5. Metered budget: a BPF map counter per consumer; once the period budget is hit, the verdict flips to deny-cellular. Reset by userspace on the period boundary.

Privilege + trust model (inherits RedFlag's binary-trust pattern)

This is the load-bearing security design, lifted directly from RedFlag's redflag-helper (capability-token executor, verified in helper/src/main.rs).

The standing-daemon question is resolved: netd is NOT standing. The BPF programs are long-lived kernel state, pinned to /sys/fs/bpf/souveraine/ (bpffs is already mounted, mode 700). They survive netd exiting. So the privileged process and the privileged kernel state are decoupled — a standing daemon is not needed to hold the programs, and RedFlag's "no standing elevated daemon" doctrine holds.

souveraine-netd — transient, per-operation, capability-token executor.

Modeled 1:1 on redflag-helper:

• Invoked via one systemd-run --wait ... souveraine-netd --token-file ... line (the agent's/machined's only sudo for net ops). Scoped by polkit to transient units, exactly as RedFlag does.
• Reads one Ed25519-signed capability token on stdin, performs exactly one mutation, exits. No shell, no inherited env, fail-closed on every path.
• Exit codes double as deny taxonomy (copy RedFlag's 10–26 scheme).
• Token operations (fixed vocabulary — this is the "models won't rewrite code" guarantee): attach-program, set-policy <consumer>, enroll-cgroup <pid>, set-ip-rule, set-metered-budget, detach-program.

Signing authority (decided 2026-07-17): machined signs netd tokens.

machined already holds the machine Ed25519 seed and the souveraine-machined:v1 signing context. netd tokens are signed by machined under a new domain netd-policy. One root of trust per machine, already provisioned and audited (SO_PEERCRED, guarded ok/reason). No second key, no off-host authority. The agent never signs; it step-up-requests, machined signs on policy authority, netd verifies against the machine pubkey.

The agent step-up path (the product surface you want):

agent decides "consumer X should be wifi-only"
  → step-up IPC to machined: {operation: set-policy, consumer: X,
                              policy: {allow: wifi-only}, reason: ...}
  → machined verifies agent's step-up auth (ambient/personal/stepUp tiers,
    same machinery as the session capability work — gated, acked, verified)
  → machined signs netd-policy capability token:
      closure_hash = sha256(canonical policy mutation)
      signed = machined.sign("souveraine-machined:v1:netd-policy:{token}")
  → machined invokes netd via the one systemd-run line, token on stdin
  → netd: verify token vs machine pubkey → validate trust paths →
          hash-check → perform the one set-policy → write verdict map → exit
  → result back to machined → back to agent

The verbs (set-policy, enroll-cgroup, ...) and the policy schema are fixed. A model expresses intent over this stable vocabulary; it can never ask netd to do something outside the enumerated operations, because netd refuses unknown ops (EXIT_UNSUPPORTED_OP). Extending the engine adds verbs; it never requires the model to emit different logic for existing ones.

Inherited RedFlag mechanisms (all verified in helper/src/main.rs):

1. Pinned keyring, verify-keys-not-servers — netd trusts the machine pubkey by fingerprint. /etc/souveraine/trusted-keys (or derive from machined's seed pub). No URL trust.
2. Trust-path self-validation (SEC-021) — every file netd relies on (keyring, the BPF ELF being attached, policy files) must be root-owned, not a symlink, not group/other-writable. Violation = hard denial (EXIT_TRUST_PATH). Defends its own inputs.
3. Hash-pinned BPF objects — the BPF program ELF is part of the signed closure; netd re-computes sha256 constant-time before attach. A swapped .o is denied. stage_and_verify into root-only staging before verify, so the caller can't swap bytes mid-flight.
4. Atomic replace — programs/maps pinned atomically; failed attach leaves no half-state (rename, never write-in-place).
5. Replay guard — consumed tokens recorded in netd state file.
6. Self-upgrade via the same gate — netd replaces itself using a netd-self token verifying the new binary's hash. Trusted path never widens for updates.

Reactivity without a standing daemon (the key trick):

Link-change reactions (flip the verdict map when wifi drops) are handled by a small unprivileged watcher — NM-D-Bus / netlink listener with write access only to the bpffs map file, not to netd's privilege and not to CAP_NET_ADMIN. bpffs map permissions gate who may write the verdict. So the privileged surface stays transient (netd runs per-op, exits), and the reactive surface is unprivileged (map writes only). This is how you get live handoff behavior without a standing elevated process.

netd systemd unit (inherits RedFlag's hardening template):

• AmbientCapabilities=CAP_NET_ADMIN ONLY. No CapabilityBoundingSet (RedFlag rule: strips setuid caps from sudo inside the unit, kills the invocation path).
• ProtectSystem=strict, ProtectHome=true, PrivateTmp=true, ProtectKernelTunables=true, RestrictSUIDSGID=true.
• ProtectControlGroups left default — netd needs cgroup dir creation.
• Documents the kernel-version floor (see NET_KERNEL_GAPS.md) and refuses to attach if the floor isn't met (BTF absent, mark target missing) — fail-closed, never silently unenforced. RedFlag's floor doctrine.

Waydroid as a consumer class

Not a binder integration. Waydroid's LXC container init is enrolled into /souveraine.net/waydroid/ (or a subtree with per-app subgroups if we want per-Android-app granularity). Its sockets are classified and routed like any other process. The Android ConnectivityManager inside the container keeps working against whatever egress the kernel gives it; we govern that egress from outside. binder stays out of scope.

The handoff (the thing you feel leaving the house)

Two halves, both required:

1. Substrate connections re-resolve + reconnect on link change. peer_outbound_task today pins its resolved peer across backoff. Fix: hold endpoint as URL, re-resolve via tokio::net::lookup_host on every reconnect, subscribe to core/net's watch and force-reconnect on primary change. Cellular-v6-only + CLAT case: log the CLAT ride, don't pointlessly retry an IPv4 peer.
2. Kernel-classified consumers follow the link set their policy allows. When wifi drops, a wifi-only consumer's ip-rule + nft deny drops its egress; when wifi returns, it flows again. No userspace action needed — the BPF verdict map flips with LinkState.

Surfaces

• health panel: active link, metered, cellular signal, connectivity, AND the resolved verdict per enrolled consumer ("waydroid: wifi-only (denied on lte)"). The consumer view is the new part.
• (later) a policy UI: assign consumers to policies. Read-only first.

Dependencies & privilege

• zbus (present) for NM/MM D-Bus.
• New: aya (pure-Rust BPF loader). Chosen over libbpf-rs to keep C out of netd's privileged TCB. Requires kernel BTF (the rebuild adds it).
• New: nftables — shell nft first (invoked by netd per-op); bind nftnl-rs later only if a hot path demands it. Per-op shelling fits the transient- executor model fine.
• netd carries CAP_NET_ADMIN (transient, per-op via systemd-run), NOT machined. machined signs tokens; netd enforces. The agent process stays unprivileged throughout. See Privilege + trust model above.
• Cross-build on archdev: the Rust binaries cross-compile as today; BPF programs are ELF-for-BPF (arch-independent), host-built, not cross- compiled to aarch64.

Decisions (resolved 2026-07-17)

1. Daemon shape: transient executor, NOT standing. netd runs per- operation (one signed token, one mutation, exit), modeled on redflag-helper. BPF programs persist as kernel state pinned to bpffs, decoupled from the privileged process. Reactivity via an unprivileged map-writer. RedFlag's "no standing elevated daemon" doctrine holds. Detail in the Privilege + trust model section above.

2. Loader: aya. Pure Rust, no C libbpf in the privileged TCB (RedFlag "small auditable privileged surface"), cross-friendly for archdev. Requires kernel BTF — the rebuild adds it (see NET_KERNEL_GAPS.md).

3. Signing authority: machined signs netd tokens under domain netd-policy. Single root of trust, already provisioned. Agent step-up- requests; machined signs; netd verifies.

4. Agent may change policy, via step-up. Tied to the ambient/personal/ stepUp capability tiers — a policy mutation is a step-up request (gated, acked, verified), same machinery as the session capability work.

5. Metered budget: soft warn at N + hard deny at M (Android NetworkPolicyManagerService model). Honest about enforcement precision.

Still open (genuinely, not deferred for safety)

• Per-Android-app granularity inside Waydroid. Whole-container works now; per-app needs Waydroid's per-app cgroups surfaced to the host. Defer until Waydroid is installed. The cgroup-subtree layout is designed so a per-app subtree carves later without rework. (Separately interesting as its own thread: running Android apps semi-natively — noted, not this build.)

What's deliberately NOT deferred to a "v2"

• The Consumer/LinkPolicy/Enforcement model is built whole, not phased in.
• Kernel classification (BPF + ip rule + nft) is part of this build, not a follow-on — it's the entire reason the product is worth making.
• Per-consumer metered budget is part of this build.
• The capability-token trust model (machined-signed, RedFlag-inherited) is part of this build, not bolted on later.
• The health surface shows consumer verdicts, not just link state.

What IS staged by necessity (not by safety): Waydroid per-app granularity (needs Waydroid installed); the policy-editing UI (needs the engine live); the kernel rebuild (needs the flash). These are gated on external facts, not on us playing it safe.