# Program Sandbox **Page:** kit/program-sandbox [Download Raw Markdown](./kit/program-sandbox.md) --- # Program Sandbox This page covers the optional `sandbox` block on a [daemon program](/kit/daemons/): what each field does, how the daemon normalizes and validates the block, what changes for a program that carries one, what the container needs for it, and how to read the sandbox state back. A sandbox confines one program with kernel-enforced mechanisms: bubblewrap in a user namespace for the filesystem, Landlock for TCP bind and connect, an optional block on UDP sockets, a transient systemd scope for memory, task and CPU limits, in-process limits on open files and file size, and one nftables table for inbound connections. You set it from `programs.json` or from the add and edit endpoints, like every other program field. Programs without an effective sandbox keep the ordinary rendering path, including the serialization of their environment values. No program's environment values are escaped: a value that contains a double quote is written in single quotes. Which quoted values a request may send is described under [required fields and defaults](/kit/daemons/#required-fields-and-defaults). A sandboxed program's values may contain no quote of either kind, `%`, `\` or control character; such a value in its own `environment` is refused, and one taken from the container's environment is dropped. There is no periodic enforcement loop. ## What the sandbox does not do A sandbox applies the restrictions you configure, and nothing else. Read this before relying on one against hostile code. - **It restricts writes, not reads.** A read-only root still lets the program read every file its account can read, including secrets in trees that cannot be hidden. The container's `/proc` is the program's own, so other processes in the container stay visible. - **`restricted` limits TCP ports, not hosts.** The listed ports are allowed to any destination. UDP, ICMP and raw sockets are outside the policy, and so is a socket that starts listening without binding. To stop UDP as well, set `udp: "deny"`, which works under every mode. Unix sockets reachable by path stay reachable, under every mode. - **`restricted` has no MPTCP, no io_uring and no TCP Fast Open send.** So that the port lists hold, a `restricted` program cannot create an MPTCP socket (the request answers `EPROTONOSUPPORT`) or an SMC socket, cannot set up an io_uring ring (`io_uring_setup` answers `ENOSYS`), and cannot send with `MSG_FASTOPEN` (the send answers `EOPNOTSUPP`). Software that falls back to plain TCP, to epoll, or to connect-then-send keeps working. A program that requires io_uring does not run in this mode. No other mode is affected. - **Every sandbox hides `/run/user`.** A sandboxed program sees an empty `/run/user` in place of the real one, so `systemctl --user`, the account's session bus and anything else in its runtime directory are out of reach from inside the sandbox. - **Ingress rules cover new IPv4 TCP connections on eth0**, on the ports the policy declares. Established connections are never cut, and packets whose SOURCE is loopback or the gateway are exempt by default. That exemption is by address, not by caller: requests Hoody Proxy forwards from the internet arrive with the client's own address and are filtered and metered like any other source. - **The rate limit is best effort.** Budgets reset whenever the table is rebuilt, and once its 65535-source set is full, new sources are not metered at all. The allowlist keeps applying. - **Running programs are not re-checked.** A launch refuses when the firewall table does not match the committed state, but a program already running keeps its listener unfiltered until a firewall transaction or a daemon start reinstalls the rules. - **Hidden paths can be moved.** An actor who can rename a validated directory, or a directory above it, can change where a hidden mount lands. Writable mounts are pinned by descriptor, but their destination is still resolved by path, and nothing is re-checked while the program runs. - **Confining a program does not clean up what ran before it.** A detached child of an earlier unsandboxed run is outside everything the daemon observes. - **There are no disk-space quotas.** `max_file_size` caps each file and `tmp_size` caps the private temporary directories, but nothing limits the total space a program writes to its `writable` paths. - **It needs a non-root account**, and it is per program: creating a container confines nothing by itself. ## API Endpoints Summary All endpoints are relative to your Daemon Manager service URL: ``` https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com ``` - [`POST /api/v1/daemon/programs/add`](/api/daemon/management/) - Create a program, optionally with a `sandbox` block - [`POST /api/v1/daemon/programs/edit/{id}`](/api/daemon/management/) - Replace, clear, or keep a program's `sandbox` block - [`GET /api/v1/daemon/programs/{id}/sandbox`](/api/daemon/management/) - Stored policy, effective argument vectors, and live firewall state - [`GET /api/v1/daemon/health`](/api/daemon/monitoring/) - Sandbox mechanism probe and container-wide firewall state The examples on this page show the CLI, the SDK and HTTP. The SDK examples use a `containerClient` created as on the [Daemons](/kit/daemons/) page. The CLI adds one guard of its own. An edit that sets any `--sandbox-*` flag first reads the stored block (`configured`). When the new block would omit or null a stored setting, it sends nothing and lists each dropped setting with the flag that keeps it. A stored `false` counts as a setting; stored nulls and empty lists do not. If that first read fails, the edit is refused too. `--sandbox-replace` skips the check entirely, and `--sandbox null` clears the block without it. The guard only looks for dropped settings: it does not judge whether the values you send are weaker. HTTP and SDK edits have no such guard. ## When to use a sandbox Use a sandbox for a program that serves traffic or runs code you do not fully control: a web app that should write only to its data directory, a worker that should reach only selected TCP destination ports, or an internet-facing service that should accept connections from a known network at a bounded rate. Each concern maps to one sub-object, and you can set any one of them on its own. ## The sandbox block The block has three sub-objects, `filesystem`, `network` and `process`. Every one is optional, and an optional field set to `null` means absent. Unknown keys are rejected at the sandbox root and inside each sub-object, so a misspelt field is a 400 rather than a quietly weaker policy, and a field a newer client sends is a 400 against an older daemon. `private_pid`, `no_new_privileges` and capability-grant fields are unsupported. A resolved policy larger than 64 KiB is refused with `sandbox policy is too large`. That size is measured on the spec as it will be transported, carrying the program's own name, the widest port of its instance range and the widest id the daemon can assign, so a policy accepted here is never refused afterwards by the stage that receives it. ```json { "name": "my-api", "command": "node server.js", "user": "api-svc", "directory": "/hoody/storage/apps/my-api", "port_range": { "start": 8000, "end": 8003 }, "sandbox": { "filesystem": { "read_only_root": true, "writable": ["/hoody/storage/apps/my-api"], "hidden": ["/root"] }, "network": { "mode": "restricted", "bind_ports": ["8000-8003"], "connect_ports": [443, 53], "ingress_allow_from": ["203.0.113.0/24"], "ingress_allow_platform": true, "ingress_rate_limit": "50/s" }, "process": { "max_memory": "512M", "max_pids": 256, "max_open_files": 1024, "private_tmp": true } } } ``` ### Filesystem fields `read_only_root` stops writes to the root tree; it does not restrict reading, and `hidden` is what removes a directory from view. Protected trees such as `/etc`, `/run` and `/hoody` cannot be hidden, so secrets a program's account can read there stay readable. `/proc` is the container's, bound read-write, so other processes in the container remain listed; `/dev`, a private `/tmp` and the hidden tmpfs mounts are separate mounts and are not read-only. `/run/user` is hidden in every sandbox, whatever the policy, including a runtime directory that appears after the program started; a sandboxed program whose `directory` is under `/run/user` is rejected with 400. | Field | Accepts | Default | Effect | |-------|---------|---------|--------| | `read_only_root` | boolean | `false` | Mounts the root tree read-only. Three mounts are outside that: `/proc` is bound read-write, so per-process files such as `/proc/self/oom_score_adj` stay writable; `/dev` is a fresh device mount; and `/sys` is read-only for every sandboxed program regardless of this flag. Private temporary directories and hidden mounts have their own behaviour, and ordinary account permissions still apply. | | `writable` | array of absolute directory strings | none | Bound read-write inside the read-only root. A non-empty list requires `read_only_root`: `writable` on its own is a 400, not a restriction. An empty or null list is inert. | | `hidden` | array of absolute directory strings | none | Replaced by an empty, unreadable tmpfs. Follows the same overlap rule as `writable`, and additionally may not overlap `/usr`, `/bin`, `/sbin`, `/lib`, `/lib64`, `/lib32`, `/libx32`, `/hoody`, `/hoody/storage`, the program's own `directory`, or any `writable` entry. An empty or null list is inert. | Both lists require existing absolute directories with no `..` component, using only ASCII letters and digits, `/`, `_`, `.`, `-`, space and tab; each is canonicalized, and `/` itself is forbidden. A path may neither equal, contain, nor be contained by a protected root: the storage and supervisord configuration trees, the log base directory, `/etc`, `/run`, `/var/run`, `/hoody/plugins`, `/proc`, `/sys`, `/dev`, `/boot`, and the daemon executable's directory. `/hoody` and `/hoody/storage` are therefore refused because they contain the daemon's own tree, while a directory such as `/hoody/storage/apps/my-api` is accepted as `writable` when its canonical path avoids this daemon's configured protected roots. Each `writable` and `hidden` path is checked again at every launch, and a path that changes identity during the launch, or that is no longer a directory, is refused. The identity the API saw when the policy was stored is not pinned. The directory mounted writable is the directory that was checked. The DESTINATION of every mount is still resolved by pathname: an actor who can rename a validated directory, or any directory above it, can move where a hidden mount lands and leave the contents it was meant to hide visible under another name. Nothing rechecks these paths while the program runs, so do not rely on `hidden` where untrusted code can rename those directories. ### Network fields | Field | Accepts | Default | Effect | |-------|---------|---------|--------| | `mode` | `full`, `restricted` or `none` | `full` | `full` adds no Landlock bind or connect restriction; ingress allowlist and rate settings still apply. `restricted` confines TCP bind and connect with Landlock to `bind_ports` and `connect_ports`. `none` runs the program in an empty network namespace, which closes IP sockets and abstract unix sockets but not a unix socket reachable by a filesystem path. | | `bind_ports` | ports and `"a-b"` ranges | union of `port_range` and `ready_port` when omitted or null | TCP ports the program may bind under `restricted`. Also the listening set for ingress rules and port ownership. An explicitly empty list receives no defaults. | | `connect_ports` | ports and `"a-b"` ranges | none | TCP destination ports the program may connect to, on any host. Valid only with `mode: restricted`; an empty list under `restricted` denies every TCP connect, and a non-empty list with any other mode is a 400. | | `ingress_allow_from` | up to 64 CIDRs, each a strict IPv4 dotted quad `a.b.c.d/n` with octets 0 to 255 and a prefix from 0 to 32 in plain decimal (`/24`; `/024` and `/+24` are refused) | none | Source networks allowed to open new connections to the listening set on eth0. Requires a non-empty listening set. An empty list, a null, or the field omitted applies no source restriction rather than denying everyone; a rate limit still applies. | | `ingress_allow_platform` | boolean | `true` | Returns traffic from `127.0.0.0/8`, and from the platform address, before the rate meter and the allowlist. The platform address is the eth0 default-route gateway, and when there is no default route it is the first host address of the first connected eth0 route. It is looked up on the first firewall transaction and remembered once found; a lookup that fails is retried by the next transaction, and the table rendered in between covers loopback only. | | `ingress_rate_limit` | `"N/s"` or `"N/m"`, N from 1 to 100000 | none | Per-source budget for new connections, evaluated before the allowlist. The rule meters matching SYN packets and allows a burst of N packets on top of its replenishment rate, so it is not a fixed-window count of completed connections. Also requires a non-empty listening set. | | `udp` | `allow` or `deny` | `allow` | `deny` stops the program from creating UDP sockets, under every `mode`: the request fails with `EACCES`. ICMP ping sockets and SCTP sockets are refused too, and the program cannot set up an io_uring ring. Name resolution through the system's standard (glibc) resolver switches to TCP, so under `mode: restricted` add TCP port 53 to `connect_ports`. | `restricted` allows the listed TCP destination ports on any host: it is a port policy, not a host policy. It does not cover UDP (set `udp: "deny"` for that), ICMP, raw sockets or Unix sockets, and it restricts `bind` and `connect`, so a socket that starts listening without binding is outside it and has no ingress rule. `full` and `restricted` share the container's abstract Unix-socket namespace; `none` gives the program its own. Under `mode: none` the filesystem policy is what decides who the program can still talk to, because a unix socket it can open by path stays reachable. A port item is an integer from 1 to 65535 or an inclusive range `"a-b"` with `1 <= a <= b <= 65535`; zero, reversed ranges and out-of-range endpoints are rejected, and overlapping or adjacent items are merged. Landlock restricts TCP destination ports, not hosts or protocols: under `restricted`, UDP, ICMP and raw sockets are outside the policy. It covers `bind` and `connect`, not a socket that begins listening without binding, so the declared listening set is what the ingress rules cover. A bind or connect that Landlock denies fails inside the program with `EACCES` (`Permission denied`), and the restriction is inherited by every process the program starts. Ingress filtering behaves differently: it silently drops denied packets. A source the allowlist refuses normally sees a connection timeout, while a rate-limited connection can still succeed when a retransmitted SYN arrives after the budget refills. For an effective sandbox, `bind_ports` must cover `port_range` and `ready_port` when those are set; when neither is set, `bind_ports` is required for `mode: restricted`. `none` rejects `port_range`, `ready_port`, and any non-null `bind_ports` value including `[]`; it rejects a non-empty `connect_ports` or `ingress_allow_from` and any rate limit, while empty connect and allowlist arrays are inert and `ingress_allow_platform` is accepted and inert. Firewall records committed by an older daemon with a leading-zero prefix such as `/024` are read as the decimal `/24` that was meant. The program's stored `sandbox` block is not rewritten, so correct such a spelling with an edit before the policy is applied again; a committed allowlist entry that cannot be read at all makes loading the firewall state fail. With `udp: "deny"`, software that brings its own resolver, or a C library other than glibc, has to be configured for DNS over TCP itself, or its lookups fail. Under every `mode` except `none`, the launch is refused when the container's `/etc/resolv.conf` is missing or cannot be read, is not a regular file, is larger than 64 KiB, or resolves into a hidden, writable or private temporary path or under `/run/user` or `/dev`, which every sandbox mounts over. Under `mode: none` the program has no network to ask, so `/etc/resolv.conf` is not checked. A 32-bit x86 program that creates sockets through the legacy `socketcall` interface cannot create any socket that way under `deny`, Unix sockets included. A socket created outside the sandbox and passed in over a Unix socket is not covered. The Hoody Kit ports `4`, `5`, `40`, `44`, `45`, `46`, `48`, `50`, `55`, `59`, `60`, `61`, `75`, `76`, `77`, `78`, `3971`, `3998`, `3999`, `4000` and `23333`, plus the daemon's own effective port, are reserved against sandboxed listening sets. ### Process fields | Field | Accepts | Default | Effect | |-------|---------|---------|--------| | `max_memory` | `[KMG]`, or a bare decimal byte count, from 16M to 64G | none | Written to the scope as `MemoryMax=` together with `MemorySwapMax=0` and `OOMPolicy=continue`. | | `max_pids` | integer from 4 to 65536 | none | `TasksMax=` on the scope. The count includes the two wrapper helpers, so budget at least 2 above the program's own process and thread count. | | `max_open_files` | integer from 8 to 1048576 | none | `RLIMIT_NOFILE`, soft and hard, set inside the sandbox after the helpers are set up and immediately before the program starts, so the helpers' descriptors are not charged to it. | | `max_cpu` | `"%"`, n from 1 to 102400 | none | CPU time cap, measured against one CPU: `"50%"` is half of one CPU and `"200%"` is two full CPUs. A port-range program gets the cap once per instance, so its instances can use that many times the cap together. | | `max_file_size` | `[KMG]`, or a bare decimal byte count, from 1K to 1024G | none | The largest single file the program may write. A write that would grow a file past it fails with `EFBIG` (`File too large`), and the program handles that error; it is normally not killed. It caps each file, not total disk use, and does not cover the program's log files. | | `private_tmp` | boolean | `false` | Gives the program its own empty `/tmp` and `/var/tmp`. A 400 when any canonical `filesystem.writable` path equals, contains, or lies inside `/tmp` or `/var/tmp`: the private mounts go on first, so a writable bind that meets one of them would put the container's shared directory back. Also a 400 when the program's `directory` lies strictly below either path, or when the daemon itself is installed at or below either: the fresh mounts hide them, and the launch would fail without reaching the program. | | `tmp_size` | `[KMG]`, or a bare decimal byte count, from 1M to 64G | none | Size of each private `/tmp` and `/var/tmp`. Requires `private_tmp: true`, and is a 400 without it. Each of the two directories gets the full size, so together they can hold twice the value. Files there are held in memory and count against `max_memory`. A write past the size fails with `ENOSPC`. | K, M and G are binary units. A quoted decimal byte count within the same limits, such as `"16777216"`, is accepted too. The swap cap is what makes the memory limit enforceable: on a container with swap, `MemoryMax` alone lets a program exceed its cap by swapping. With `OOMPolicy=continue` the kernel kills only the offending process; when that process is the program itself, supervisord sees exit code 137 and the wrapper then tears down the rest of the scope. The program sees `/sys` read-only, so it cannot raise its own limits. Creating the scope is not taken as proof that the limits exist: systemd creates a scope even when a controller is unavailable and drops the limit. Before the program runs, the wrapper reads back each limit the policy requested from the scope's cgroup, `memory.max` and `memory.swap.max` for `max_memory`, `pids.max` for `max_pids`, and the CPU quota for `max_cpu`. A file that is missing or unreadable, a value of `max`, or a value other than the request refuses the launch with exit 78 and `sandbox: the scope was created but its limits were not installed: ...`. `memory.max` may equal the request rounded down to a page boundary, and the CPU quota may differ from the request by up to 1%. Limits the policy did not request are not checked, and `systemd_run: true` on health only says that the executable exists. ## Normalization and edits Defaults and merged port ranges determine the resolved policy. The block is effective only if a restriction remains: `read_only_root`, a non-empty `hidden`, a `mode` other than `full`, a non-empty `ingress_allow_from`, `ingress_rate_limit`, `udp: "deny"`, `max_memory`, `max_pids`, `max_cpu`, `max_open_files`, `max_file_size` or `private_tmp`. A block that restricts nothing is stored as absent and claims no ports; `ingress_allow_platform` alone, at either value, and `bind_ports` alone are both ineffective. The API removes an entirely ineffective block but does not expand an effective block into its resolved defaults: effective blocks retain their configured values, and optional null fields are treated as absent and may be omitted when read back. A no-op block can survive only in a hand-edited `programs.json`. On `programs/edit/{id}` the block is three-state: - **Absent** leaves the current block unchanged. - **`null`** clears it. - **An object** replaces the stored block as a unit. There is no field-level merge into a sandbox block, so an edit must resend every field it wants kept. On `programs/add`, absent and `null` mean the same thing. Quick-start does not support sandboxed execution: any non-null `sandbox` value, including `{}`, is a 400, and `sandbox: null` is accepted as absent. A top-level key that looks like a misspelt `sandbox`, such as `sandbx` or `Sandbox`, is a 400 there too, which catches common typos. A quick-start program always runs without confinement, so use a persistent program when you need it. ## What changes for a sandboxed program - **The pid supervisord tracks is the wrapper**, not the program. The rendered configuration keeps the `environment=` line, drops `user=` because the uid drop happens inside the wrapper chain, and adds `killasgroup=true`. - **Exit codes shift.** Only the program is signalled. bubblewrap reports a signal death as `128+n` and the wrapper exits with that code, so a `SIGKILL` reaches supervisord as 137 rather than as a signal. An OOM kill under `max_memory` is the same 137. - **A policy change replaces the loaded definition.** The 12-hex policy revision is part of the supervisord command line, so a change to the resolved policy changes the rendered configuration and supervisord reloads it. The reload stops whatever was running under the old definition; whether a new instance starts follows `boot` and `lazy_load`, so a boot program comes back and a lazy or non-boot one stays down until something starts it. Reordering or merging entries in `bind_ports` and `connect_ports` preserves the revision when the resolved ranges are unchanged. `writable`, `hidden` and `ingress_allow_from` keep their submitted order, so reordering one of them changes the revision and replaces the loaded definition, with the same start rules as any other policy change. An unchanged configuration skips the reload only when no earlier reload of that program is still unconfirmed. A disabled program's configuration is removed instead, and a lazy or non-boot program is rendered with `autostart=false`, so an edit does not by itself start an instance. - **Some environment names are refused.** For an effective block, the glibc-unsecure names `LD_*`, `GCONV_PATH`, `GETCONF_DIR`, `HOSTALIASES`, `LOCALDOMAIN`, `LOCPATH`, `MALLOC_*`, `NIS_PATH`, `NLSPATH`, `RESOLV_HOST_CONF`, `RES_OPTIONS`, `TMPDIR`, `TZDIR` and `GLIBC_TUNABLES` are a 400 in `environment`. They are also filtered out of the container environment the daemon injects, with a warning. - **Supervisord's own variables are removed.** The `SUPERVISOR_*` names, including the address of its control socket, are stripped from the program's environment. An empty network namespace does not close a unix socket opened by path, so a confined program must not be handed the address of the process that supervises it. - **A sandboxed program's environment values must be literal.** Supervisord expands `%(...)s` and `%(...)c` across the whole `environment=` line and then splits it with a shell-style lexer, and released versions build that lexer in a mode where a backslash is an ordinary character rather than an escape. A value carrying `%`, a quote of either kind, a backslash or a control character can therefore reappear as further assignments in the environment of the root wrapper process, and a value wrapped in single quotes would arrive with those quotes stripped rather than as written, which runs before any of this daemon's checks. No escaping is correct for every supported parser version, so an effective block refuses such values outright, with a 400. A stored `display` is validated with the environment, so an unsafe one refuses the policy rather than quietly disappearing; the render path additionally drops an unsafe value from the sources the API does not own, the container environment the daemon injects and daemon variables a command references. Environment names are refused the same way when they belong to the dynamic loader or to the sandbox wrapper's own transport, which is every name beginning `HOODY_SANDBOX_`, and both kinds are dropped from the container environment the daemon injects. Environment NAMES are held to the same split: a sandboxed program's names must be C identifiers, while an ordinary program keeps whatever names it always had, minus any that would break the configuration file itself. Programs without an effective block follow the ordinary rule instead: a value may carry one kind of quote but not both, and no quote at either end, and a backslash is an ordinary character; see [required fields and defaults](/kit/daemons/#required-fields-and-defaults). Where the configuration is written, `command` and `environment` are stripped of line feeds, carriage returns and NULs for every program, so a stored program cannot open a second supervisord section. - **A program name may not be `.` or `..`.** The name becomes a supervisord section header and a path component of the default log directory, so those two would put the daemon's log writes outside the log base. The API refuses them, and so does the render path for a hand-edited or restored `programs.json`. - **`user: root` is refused.** An effective block on a program running as uid 0 is an HTTP 400 during API validation, and the wrapper refuses it again at launch. Root inside the sandbox is still uid 0 to the system bus and to supervisord's socket, so the confinement would not hold. - **`terminal_id` is refused.** An effective block combined with `terminal_id` is a 400. - **Launches fail closed; running programs are not re-checked.** If anything flushes or changes the table under a program that is already running, that program keeps running without the filtering those rules gave it, and an identical edit or start does no firewall work, so it repairs nothing: a successful firewall transaction or a daemon start is what reinstalls the rules. Any stored `sandbox` block is validated when the program is rendered, not only one that restricts something, so a block that breaks a rule refuses the render instead of running the program unconfined. A program whose sandbox cannot be established never starts. An ingress policy must have matching installed rules, and the whole live `ip hoody_daemon` table must match what the last firewall change applied, so an external firewall reset is caught even though the stored policy is unchanged. A table that differs anywhere refuses the launch, and that includes a chain whose rules were flushed while its name survived. So does a live-table check that cannot complete within 5 seconds, or a table listing larger than 8 MiB; the listing grows with the rules and allowlists, not with the traffic a program receives. A firewall change already in progress delays a launch, and refuses it only when the change outlasts that 5-second check. A launch whose sandboxed stage does not start in time fails with `handshake failed: payload never announced itself` or `outer never released the payload`. That check covers launches; it does not repair a table that disappears under a program that is already running, which keeps its listener unfiltered until the next firewall transaction or daemon start reinstalls the rules. A removal or disable that completed before the launch refuses it; one that lands during the launch lets this instance start under the draining rules and drain with them, and a drain never removes the rules of an instance that is still starting. - **Every launch rereads the stored program.** The wrapper reads `config.json` and the whole `programs.json` at each launch. A file it cannot read or parse, including a malformed entry for another program, a row that is missing or disabled, a block that is no longer effective, and a policy whose revision differs from the one in the supervisord command line (`revision mismatch: ... (re-apply the program)`) all refuse the launch with exit 78. A hand edit of `programs.json` is read at the next launch but does not change the revision in the existing supervisord command. If the edit changes the resolved policy revision, that command refuses to launch. Use the edit endpoint, which stores the policy and republishes the configuration. - **The command is resolved inside the sandbox.** A first word containing `/` is used as written. Any other name is looked up on the program's own `PATH`, where an empty component is ignored instead of meaning the working directory, so write `.` explicitly to search it. The sandbox tries each candidate as the program's own user, so it skips a file that user may not execute and goes on to the next directory, as a shell's lookup does. It also skips a path it cannot read for another reason, such as a symbolic-link loop or an over-long name. A file that is found but is not in a runnable format stops the search. A command that is missing, not executable by the program's user anywhere on `PATH`, or cannot be executed refuses with `sandbox: inner: ...` and exit 78, and there is no shell fallback for a script without a valid interpreter line. - **A refusal is loud.** A configuration refusal prints `sandbox: ` on stderr and exits with code 78, so the program's own error log names the reason it never started. A launch cancellation or a handshake failure also prints a diagnostic, but can exit with a child's failure code or with 143. Inside the sandbox, `--cap-drop ALL`, `--unshare-pid`, `--unshare-user`, `--unshare-ipc` and `no_new_privs` are unconditional, so there is one wrapper shape for every policy. The IPC namespace is the program's own, so System V shared memory and message queues do not reach other processes in the container. `/proc` is the container's, bound rather than mounted fresh, because a fresh proc mount is refused by the container's locked mounts; `/sys` is bound read-only. Runtime scopes live under `system.slice` and are named `hd----.scope`; the diagnostic endpoint renders the scope arguments with placeholders rather than identifying a running instance. TCP restriction uses Landlock, not systemd's `IPAddressDeny`. When a program that was running unconfined gains a sandbox the daemon cannot install, it tries to stop the old process, and it decides what to stop from what supervisord reports. That identification is best effort: a configuration file is not proof of what supervisord loaded, ownership of a group name can change between the check and the stop, and a detached child of the old run is outside everything the daemon observes. If the daemon cannot even record that an unconfined process may remain, a later drain can remove that program's ingress rules; it logs that failure, and it should be treated as exposure until you have checked the container yourself. ## Ingress firewall Ingress rules live in one nftables table, `ip hoody_daemon`, with one chain per program and policy revision. The table is built from the active and draining ingress records and is rebuilt as a single atomic transaction; it remains until both sets are empty and their removal is successfully applied, so it can outlive the last ingress policy until the drain finishes. An identical admission performs no nft update and does not reset meter budgets. A lazy-start can still regenerate the table when it finalizes a completed drain. Startup reconciles the recorded state, and a container with no prior state and no desired ingress policy skips firewall work. The table coexists with the iptables-nft tables that carry the localhost DNAT rule. Everything here is IPv4 TCP on eth0. Rules match by destination port only, never by destination address, so they hold whether or not the DNAT rule is installed. Two consequences follow: - **Port ownership is container-wide.** A port in a program's listening set is reserved across every local address. A create or edit whose declared ports overlap another program's declared ports or a still-draining policy is rejected with 400, for sandboxed and unsandboxed programs alike, so an unsandboxed program cannot take a sandboxed one's port either. Two unsandboxed programs keep exactly the pre-sandbox rule between themselves: overlapping `port_range` values are refused and `ready_port` is not checked. A port that a firewall record still reserves is refused to every applicant, sandboxed or not. Disabling or removing a program persists the change first and then attempts to retire its firewall record, and its ports stay reserved while any active or draining record for it exists. A retirement that fails does not fail the call: it is logged, the record and its port reservation stay, and the next daemon start reconciles it. Retiring a record moves it to draining; it does not release the ports. Only an existing program owns the records under its id, so an add that reuses a removed program's explicit id is checked against those records like any other applicant. Once its firewall records are gone, a disabled program stops conflicting in the sandbox-specific ownership checks. The legacy `port_range` overlap check still includes disabled programs. Enabling a program re-checks port ownership, whether or not it carries a sandbox block, so a port taken while it was disabled refuses the enable. - **An undeclared service on the same port is filtered too.** Ports that an unsandboxed command opens without declaring them cannot be reserved, but traffic to such a service is still filtered when its destination port matches installed ingress rules. Inside a program's chain the order is: established and related connections return first, invalid packets drop, then the platform exception, then the rate meter, then the allowlist, then drop. A chain with only a rate limit has no terminal drop. Three things follow: - Only new connections are filtered, so the packet rules themselves never cut live ones. Editing the policy is a separate matter: it reloads the program definition and can stop the listener, which closes what was open. - With `ingress_allow_platform` at its default of `true`, `127.0.0.0/8` and the platform address return before the meter and the allowlist, so callers whose packets carry one of those two sources are neither filtered nor throttled. The exemption matches the packet's source address, not the caller's identity. Hoody Proxy forwards ordinary public traffic transparently, with the client's own address as the source, so that traffic meets the allowlist and the meter like any other source. A hairpinned request, one a container sends to a service URL on its own server, is not exempt either: the proxy binds it to the address the client dialed, which for a public URL is the host's public address rather than the gateway. The loopback half relies on the host and the container's network interface filtering to drop packets that arrive on eth0 with a spoofed loopback source, because the daemon enables `route_localnet` on eth0 for its localhost redirect, which turns off the kernel's own check for them. - The rate meter is evaluated before the allowlist. It matches SYN under the `fin|syn|rst|ack` mask, so ECN-capable SYNs are counted. The meter counts connection attempts, not requests or bytes, and one source shares a single budget across every port and instance of that revision. It is a named dynamic set, `hd___rate`, with a 60 s timeout and 65535 entries. It has two documented behaviours. Budgets reset whenever the table is regenerated, which an admission, retirement, finalize or rollback of an ingress policy causes unless it changes nothing, and which every daemon start and reset causes. And when the set is full, new sources are untracked, so the rate limit fails open while the allowlist keeps applying. ### Policy changes and draining When a policy changes, the old revision's chain stays installed until every instance supervisord is running under it is confirmed stopped. A process that escaped supervisord's tracking, a descendant an ordinary program forked and detached, is outside what any of this observes: supervisord signals the process it started, and the daemon reads the processes supervisord reports. Draining completes on its own after the old revision's processes have stopped, any launch in progress has finished, and supervisord has applied the program's latest configuration; poll `GET /api/v1/daemon/programs/{id}/sandbox` until `firewall` no longer reads `draining`. Rules and port reservations can outlive the condition that held them for a short while. If `/sys/fs/cgroup/system.slice` is missing and this daemon process has never read it successfully, every revision is treated as still alive. It also waits on a program whose earlier, unconfined definition the daemon could not establish as stopped, which is recorded in the firewall state and is independent of the scopes: that process was never wrapped, so no scope describes it. Disabling a program moves its current revision into draining without any policy change, so a drain does not always follow an edit. Chains are named `hd__`, so the two revisions are separate objects rather than an edit of one. Concurrent revisions are traversed in program-id and revision order, so on ports they share a packet must pass both allowlists, and the old listening set stays reserved. A successful add or edit returns once supervisord has applied the change; an apply that fails returns an error with the program already persisted. Its response body is unchanged and carries no firewall field. The drain continues in the background and is visible only through `GET /api/v1/daemon/programs/{id}/sandbox`, which reports `firewall: "draining"` until the old revision is gone, as long as nothing else is wrong. A damaged table, or an earlier unconfined definition that is not yet resolved, reports `firewall: "degraded"` instead, with the reason in `problems`, while the old revision remains. A removed program's endpoint answers 404 from then on, so a record retained for a deleted program is visible only in the daemon's retirement and finalization logs. The [container firewall](/foundation/networking/firewall/) runs on the host kernel and applies to the whole container. The sandbox ingress rules run inside the container, per program and per destination port. Use the container firewall for container-wide policy and the sandbox for the ports one program listens on. ## Container prerequisites and restart recovery Every effective sandbox needs an executable `/usr/bin/bwrap` and `/usr/bin/systemd-run`, usable user namespaces, working systemd scopes, and an existing `/run/user` directory. `mode: restricted` additionally needs Landlock ABI 4 or higher. Ingress policies additionally need `/usr/sbin/nft`. Memory, task and CPU limits are enforced by systemd; the file-descriptor and file-size limits are in-process rlimits. `udp: "deny"` needs a readable `/etc/resolv.conf` under every `mode` except `none`. The daemon executable and the `config.json` path that go into the wrapper command must be absolute and may contain only ASCII letters and digits, `/`, `_`, `.` and `-`; a path that would need quoting is refused, which matters for a custom installation location. A program with a `writable` entry also needs a bubblewrap that supports `--bind-fd`, and a program with `tmp_size` one that supports `--size`; without it, such a program does not start. Provisioning installs bubblewrap and nftables with the daemon. Beside `programs.json`, which describes the desired programs, the daemon keeps `sandbox-firewall-state.json` in its configuration directory as the firewall's only durable input. It records a generation number, the fingerprint of the table as applied, the digest of the snapshot it was published with, the active and draining policy records, and any unresolved unconfined installations. `sandbox-firewall.nft` is a snapshot of the rules derived from it, carrying a `# generation=N` comment. New commits increment the recorded generation, and the marker described below carries the highest generation ever committed in that directory. A failed publication puts back the previous generation together with the content that describes it, and a recovery that lost both files continues above the marker's number, so no committed generation number is ever handed out twice. A proposal whose publication failed and was put back was never committed, and its number can be used again. A restore refuses to run when the state and snapshot generations disagree. The first firewall commit also writes `sandbox-firewall.initialized` beside those files. A state file that has gone missing while the snapshot survives is lost history rather than an empty firewall: the daemon refuses to regenerate, its startup reconciliation fails, and the programs that carry an ingress policy are skipped. When the state file and the snapshot are both missing while the marker survives, the daemon first asks whether anything could still be relying on the rules they described: a sandbox scope holding processes, a launch in progress, or an installed ingress table. If any of the three is present, or cannot be read, it refuses to regenerate for the same reason, and you recover by restoring the committed state and its matching snapshot. If none of them is present the container is verifiably idle, so the daemon logs the loss as a warning and publishes an EMPTY state, and a container that legitimately cleared its files recovers instead of failing every sandboxed create. That happens at daemon start, where the reconcile that follows re-admits every enabled program's policy, and again inside the next mutation that needs the state, where only that mutation's own program is re-admitted and the others come back at their next admission or at the next daemon start. Losing both files while the daemon is running therefore does not wedge it until a restart. Read-only paths never rebuild: health and the sandbox endpoint report the loss and write nothing. With both files absent and the marker present, the restore hook exits 11 without checking whether the container is idle, and the provisioning guard likewise treats that as unknown history. Only the daemon makes the idle check, and only the daemon can write the recovered empty state before a later supervisor start. Remove the marker by hand only when no sandboxed instance can still be running. Provisioning installs `/etc/systemd/system/supervisor.service.d/hoody-daemon-firewall.conf` with `ExecStartPre=/hoody/plugins/daemon/sandbox-firewall-restore.sh`. The restore runs before every supervisor start, including one that happens while the daemon is down, and a failing restore blocks supervisord rather than letting listeners come up unfiltered. Because the rules match by port on eth0, they are effective before the daemon has installed its DNAT rule. When `nft` or the restore script is missing, provisioning asks whether rules are owed: a state file with records, a surviving snapshot, or an enabled program with an ingress policy keeps the guard, and so does a file it cannot parse. The drop-in is removed only on a verified "nothing owed", so a container without nftables and without policies keeps booting. Keeping the drop-in is not by itself a guarantee that startup is blocked: the hook decides that, and a state file that holds no active or draining record with no snapshot beside it is accepted as "nothing to restore" before the hook looks at `programs.json`, even when an enabled program now declares an ingress policy. Restore needs Python 3 and `flock`. It never runs at the same time as a daemon firewall change, so it cannot install an older generation over a newer one. Without `flock`, or when it cannot get exclusive access, it refuses with exit 12. A surviving snapshot is checked against the state file before it is applied. The state records name the chains the kernel must end up with, one per policy, and the snapshot has to define every one of them inside `table ip hoody_daemon`, with a body, and jumped to from a chain that actually carries an input hook, since a jump in a chain the kernel never calls filters nothing; a snapshot that declares chains and leaves them empty, puts them in another table, or never jumps to them is refused with exit 13. The requirement comes from the records, never from the snapshot's own `# chains=` comment, so a truncated or hand-edited file cannot describe its own contents into acceptance. Before any of that, the digest decides. Each commit records the SHA-256 of the snapshot file in the state file it writes first, and the hook refuses a snapshot whose bytes do not hash to that value, and refuses a state that carries no digest at all. That applies to every snapshot, including one belonging to a state with no policies: an empty record list is not a reason to install a file nothing vouches for. Structure alone could not do this: a snapshot can keep every chain the records name, defined and reachable, while its terminal deny is gone, its destination port is different, or a second `destroy table` follows it. A generation mismatch blocks it too, and so does a snapshot with no state file beside it, since there are then no records to check it against. What survives those checks is validated with `nft -c` and applied. Those checks establish that the snapshot is the file the state vouches for and that it defines the chains the records name; the hook does not read the table back from the kernel afterwards, so a self-consistent state and snapshot pair that describes less than the desired policies is installed without complaint. The next launch does not catch that either: it checks that the live table matches the fingerprint recorded in the state file, not that the recorded policies are the ones you intended, so check a state and snapshot pair yourself before restoring it by hand. Recorded policies with no snapshot block startup. With neither state nor snapshot, a missing `programs.json` is a first boot and the restore succeeds, while a programs file that the hook finds and cannot read or evaluate, and an enabled program that declares an ingress policy with no snapshot to supply it, both block startup. Both the guard and the hook start from the default `config.json` under `/hoody/storage/hoody-daemon/config` and follow its `storage.config_path` to the directory that actually holds the state file, the snapshot and the marker. A `config.json` that exists and cannot be parsed is "cannot tell", so the guard keeps its drop-in. The guard distinguishes a missing file from a stat error throughout its inspection. The hook does so for the state file, the snapshot and the marker, but uses shell existence tests for `config.json` and `programs.json`. A `programs.json` that exists and cannot be read stops startup with exit 8, while a path whose directory cannot be searched reads as absent for either file. The hook also accepts a configuration path as its first argument or in `HOODY_DAEMON_CONFIG`; the guard reads the default location only, so a relocated `config.json` is outside what it inspects. Every non-zero exit of the restore hook blocks supervisord: | Exit | Meaning | |------|---------| | 0 | Nothing to restore, or the snapshot applied cleanly | | 2 | `config.json` cannot be evaluated, or the state file, the snapshot or the evidence files exist but cannot be read or evaluated | | 3 | Rules must be applied and `nft` is not installed | | 4 | The state file lists active or draining policies and the snapshot is gone | | 5 | The state and snapshot generations disagree, or the snapshot's first line is not a `# generation=` header | | 6 | `nft -c` rejects the snapshot | | 7 | `nft -c` accepts the snapshot and applying it fails | | 8 | No state and no snapshot, and `programs.json` exists but cannot be read or evaluated | | 9 | No state and no snapshot, and an enabled program declares an ingress policy | | 10 | Python 3 is missing | | 11 | History lost: the marker exists and both files are gone, or a snapshot exists without its state file | | 12 | The state directory or the firewall lock cannot be used, including a missing `flock` | | 13 | The state records no snapshot digest, the digest cannot be computed or does not match, the required-chain inspection fails, or the snapshot does not enforce every chain the records require | When a program declares a sandbox the daemon cannot install, it tries to stop any unconfined process running under it. That applies at startup, when the policy is refused, its wrapped configuration cannot be published, or the firewall reconcile failed and its ingress rules cannot be shown installed. It also applies to an add, edit, enable or start whose apply failed. The daemon decides from what supervisord is actually running under the program's current and earlier names, not from the configuration file, since a failed reload can leave a wrapped file in front of an unwrapped process. When the wrapped configuration is published but supervisord has not acknowledged it, the daemon stops every group the program may be running under and keeps the configuration, so the next apply installs the confinement. Otherwise it withdraws the configuration. In both cases it checks again afterwards, and the quarantine ends only on a measurement that finds nothing running or finds the wrapper; if the process cannot be shown to have stopped, the daemon stops it directly and logs an outcome it still cannot confirm as possibly unconfined. It also records that obligation in `sandbox-firewall-state.json`, and while it stands this program's firewall records are kept instead of drained. The retention depends on that write: a record the daemon cannot write is logged as one it could not write, and the next drain can then release the rules of a listener that may still be up. A drain is finalized on the absence of processes in the policy's scopes, and a process that was never wrapped has no scope, so without that record a later disable would release the rules of a listener that is still up. The same record is written when a supervisord configuration cannot be removed, because an unconfirmed removal says nothing about what is still loaded. It is recorded only while the program still owns an active or draining record, since there are otherwise no rules to retain, and it lists the supervisord group names that have to be measured. Two cases record nothing: a program whose first apply never reached supervisord, because nothing was ever acknowledged for it and no definition of it can be loaded, and a program that owns no record at all. Two events clear it without a measurement: a removal of the program's configuration that supervisord confirmed by reloading, and an enabled program that no longer declares confinement and whose own ordinary configuration supervisord has acknowledged. Otherwise it takes a measurement. An apply whose publication supervisord acknowledged and a daemon start each trigger one; neither is evidence on its own, because each rests on a record that can be lost. A measurement clears the obligation when the recorded group list is complete, every group in it was measured, no unwrapped configuration of the program remains for supervisord to revive, and supervisord reports either nothing running or only this program's own wrapper. A quarantine that measures the program itself and finds nothing running, or finds its own wrapper, clears it directly, so a daemon start can resolve a hold while the wrapped program keeps running. Until then the program's listening set stays reserved and `GET /api/v1/daemon/programs/{id}/sandbox` carries the reason in `problems`. A running wrapper is left alone. A program with nothing running is left alone too, unless the definition supervisord still holds for it is unwrapped: that definition is withdrawn, because supervisord would start it unconfined. If supervisord cannot be asked, the daemon leaves a program alone only when every configuration that may still be loaded for it was wrapped. A program renamed by the failing edit is stopped under its old name too. A name another program has taken since is left alone only once supervisord has applied that program's own configuration. Every daemon start republishes every enabled program with an effective block, so a program whose configuration was withdrawn is re-evaluated at the next start. This matters most for the firewall case: the daemon's unit starts after supervisord, so a program already in supervisord's configuration is already running by the time the daemon looks, and declining to apply a new configuration would leave it listening under rules nobody could establish. That is deliberate: the alternative is a process running with none of the confinement its stored policy declares. The program row is untouched, so repairing the cause and enabling or editing the program brings it back. Programs without a sandbox are never quarantined; see [errors and status codes](#errors-and-status-codes) for how their failed apply is reported. Adding an ingress policy to a program that is already running unconfined stores the policy and its firewall rules before the sandboxed configuration reaches supervisord. Until supervisord has taken that configuration, the old process may still be listening. If the daemon restarts or a reset runs in that window, the daemon keeps the rules in place for as long as the old configuration is still on disk, its replacement has not been confirmed, or the configuration cannot be read. Once the sandboxed configuration is confirmed or the old one is withdrawn, the rules drain like any others and their ports are free for other programs again. A program that could not start at boot because such rules still held its port is retried while the boot replay still finds programs that can now start. If the rules clear only after the replay has finished, the program stays stopped: start it, or edit or re-enable it, once they have drained. At startup the daemon also republishes the supervisord configuration of every enabled program that carries an effective block, including one with `boot: false` and no lazy loading. Adding confinement to a program that is already running is otherwise durable only once supervisord has acknowledged its wrapped configuration, and a daemon that stopped between persisting the policy and publishing that configuration would leave the old unwrapped configuration and its process in service for the life of the container. When the bytes already match, the configuration is not rewritten, and the reload is skipped unless an earlier publication of that program is still unacknowledged; when they do not match, the reload replaces the unconfined process. A configuration change that supervisord has not confirmed is applied again by a supervisord reload at the next daemon start. If the daemon cannot record a change as pending, the apply fails before the configuration is touched. Supervisord's own inherited environment is a provisioning invariant of its systemd unit. The daemon filters unsecure names from program environments and from the container environment it injects, but it does not sanitize what supervisord itself inherits. ## Examples Each example below is an add or edit request body, not a `programs.json` file. Before running one: - the named non-root account, working directory, application files and writable directories must already exist, with suitable ownership and permissions; - a program with a `port_range` receives its port as `--port=` (or `=` when `port_param` is set), appended to the command unless it already names a port, so the command must accept that equals form; - the ports must be free and outside the reserved Kit ports; - replace `PROJECT_ID`, `CONTAINER_ID`, `SERVER`, `PROGRAM_ID` and the example network `203.0.113.0/24` with your own, and note that the HTTP and URL tabs still go through the proxy, so its permissions apply. The add calls omit `boot`, whose default is `false`, so they register the program without starting it; start it as shown on the [Daemons](/kit/daemons/) page (a new program is enabled by default, so enabling is only needed for one you disabled). ### Read-only root with a writable data directory The program sees a read-only root, with its data directory bound writable and `/root` hidden. ```bash # Register web-app with a read-only root and one writable data directory hoody daemon programs create -c "$CONTAINER_ID" \ --name web-app \ --command "/usr/bin/node /hoody/storage/apps/web-app/server.js" \ --user web \ --directory /hoody/storage/apps/web-app \ --port-range-start 8080 --port-range-end 8080 \ --sandbox-filesystem-read-only-root \ --sandbox-filesystem-writable /hoody/storage/apps/web-app/data \ --sandbox-filesystem-hidden /root ``` ```typescript await containerClient.daemon.programs.create({ name: 'web-app', command: '/usr/bin/node /hoody/storage/apps/web-app/server.js', user: 'web', directory: '/hoody/storage/apps/web-app', port_range: { start: 8080, end: 8080 }, sandbox: { filesystem: { read_only_root: true, writable: ['/hoody/storage/apps/web-app/data'], hidden: ['/root'] }, }, }); ``` ```bash curl -X POST "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com/api/v1/daemon/programs/add" \ -H "Content-Type: application/json" \ -d '{ "name": "web-app", "command": "/usr/bin/node /hoody/storage/apps/web-app/server.js", "user": "web", "directory": "/hoody/storage/apps/web-app", "port_range": { "start": 8080, "end": 8080 }, "sandbox": { "filesystem": { "read_only_root": true, "writable": ["/hoody/storage/apps/web-app/data"], "hidden": ["/root"] } } }' ``` Registers the program with a read-only root and one writable data directory. The writable directory must already exist. This restricts writes and hides `/root`; like every sandbox, it also hides `/run/user`. It adds no network policy or resource limits and hides nothing else the account can read, and `/proc` and `/dev` keep the exceptions described above. ### A restricted-network worker The worker has no `port_range`, so `bind_ports` is required under `mode: restricted`. It may make TCP connections to destination ports 5432 and 53 on any host; UDP, ICMP and raw sockets are outside this Landlock policy. It is capped at 256 MiB and 64 tasks. ```bash # Register a worker limited to TCP destination ports 5432 and 53 hoody daemon programs create -c "$CONTAINER_ID" \ --name queue-worker \ --command "/usr/bin/python3 /hoody/storage/apps/worker/main.py" \ --user worker \ --directory /hoody/storage/apps/worker \ --sandbox-network-mode restricted \ --sandbox-network-bind-ports 9100 \ --sandbox-network-connect-ports 5432,53 \ --sandbox-process-max-memory 256M \ --sandbox-process-max-pids 64 ``` ```typescript await containerClient.daemon.programs.create({ name: 'queue-worker', command: '/usr/bin/python3 /hoody/storage/apps/worker/main.py', user: 'worker', directory: '/hoody/storage/apps/worker', sandbox: { network: { mode: 'restricted', bind_ports: [9100], connect_ports: [5432, 53] }, process: { max_memory: '256M', max_pids: 64 }, }, }); ``` ```bash curl -X POST "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com/api/v1/daemon/programs/add" \ -H "Content-Type: application/json" \ -d '{ "name": "queue-worker", "command": "/usr/bin/python3 /hoody/storage/apps/worker/main.py", "user": "worker", "directory": "/hoody/storage/apps/worker", "sandbox": { "network": { "mode": "restricted", "bind_ports": [9100], "connect_ports": [5432, 53] }, "process": { "max_memory": "256M", "max_pids": 64 } } }' ``` Registers a worker that can bind TCP port 9100 and connect only to TCP destination ports 5432 and 53. This allows TCP binds on 9100 and TCP connections to ports 5432 and 53 on any host, and caps memory and tasks. UDP, other non-TCP protocols and Unix sockets reachable by path are outside the network policy, although the mode refuses MPTCP, io_uring and TCP Fast Open sends, which could reach a TCP port around it; the account keeps its ordinary filesystem access, and no ingress rules are installed. ### Allowlist and rate limit on a public service The edit targets an existing non-root program without `terminal_id` whose declared ports are compatible with the listening set. It replaces the sandbox block as a unit: new connections to ports 8000 to 8003 are accepted from one network, metered at 50 SYN packets per second per source with a burst of 50. With `ingress_allow_platform: true`, loopback sources and the gateway address bypass both checks. Public requests forwarded by Hoody Proxy carry the client's address, so they do not. ```bash # Refused, listing each setting, if the stored block has settings this edit drops. # Add them to the command, or add --sandbox-replace to drop them. hoody daemon programs update "$PROGRAM_ID" -c "$CONTAINER_ID" \ --sandbox-network-bind-ports 8000-8003 \ --sandbox-network-ingress-allow-from 203.0.113.0/24 \ --sandbox-network-ingress-allow-platform \ --sandbox-network-ingress-rate-limit 50/s ``` ```typescript // The object replaces the stored block; the SDK does not check what it drops. await containerClient.daemon.programs.update(programId, { sandbox: { network: { bind_ports: ['8000-8003'], ingress_allow_from: ['203.0.113.0/24'], ingress_allow_platform: true, ingress_rate_limit: '50/s', }, }, }); ``` ```bash curl -X POST "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com/api/v1/daemon/programs/edit/PROGRAM_ID" \ -H "Content-Type: application/json" \ -d '{ "sandbox": { "network": { "bind_ports": ["8000-8003"], "ingress_allow_from": ["203.0.113.0/24"], "ingress_allow_platform": true, "ingress_rate_limit": "50/s" } } }' ``` Replaces the sandbox block. The loaded definition is replaced, so an instance running under the old policy stops and a boot program comes back under the new one, while the previous revision's rules drain in the background. Because this edit replaces the whole block, it also removes any filesystem and process restrictions the program had and puts its network mode back to `full`. It filters new IPv4 TCP connections to ports 8000 to 8003 arriving on eth0, with the platform bypass described above. It does not restrict outbound connections or cover other ports, and the meter can reset or fill. The rules do not cut established connections, though applying this edit reloads the program and can stop its listener. ### A program with no IP networking `mode: none` puts the program in an empty network namespace. Declare no ports with it: `port_range`, `ready_port` and any `bind_ports` list, an empty one included, are refused. ```json { "name": "offline-worker", "command": "/usr/bin/python3 /hoody/storage/apps/worker/main.py", "user": "worker", "directory": "/hoody/storage/apps/worker", "sandbox": { "filesystem": { "read_only_root": true }, "network": { "mode": "none" }, "process": { "private_tmp": true } } } ``` Unix sockets it can open by path and files its account can read stay reachable, except under the directories the sandbox hides: `private_tmp` gives it empty `/tmp` and `/var/tmp`, and every sandbox hides `/run/user`. ### A batch job with CPU, file and UDP limits The job may use half of one CPU and 1 GiB of memory, which includes whatever it keeps in its private `/tmp` and `/var/tmp` of 256 MiB each. It cannot write a file larger than 2 GiB, and it cannot create UDP sockets, so its name lookups go over TCP. ```json { "name": "batch-job", "command": "/usr/bin/python3 /hoody/storage/apps/batch/run.py", "user": "worker", "directory": "/hoody/storage/apps/batch", "sandbox": { "network": { "udp": "deny" }, "process": { "max_cpu": "50%", "max_memory": "1G", "max_file_size": "2G", "private_tmp": true, "tmp_size": "256M" } } } ``` The network mode stays `full`, so TCP connections are not restricted. Under `mode: restricted`, list TCP port 53 in `connect_ports` as well, or name resolution fails. ### Remove a sandbox ```bash # Clear the block, then read back what the daemon still holds hoody daemon programs update "$PROGRAM_ID" -c "$CONTAINER_ID" --sandbox null hoody daemon programs sandbox get "$PROGRAM_ID" -c "$CONTAINER_ID" ``` ```typescript await containerClient.daemon.programs.update(programId, { sandbox: null }); const { data } = await containerClient.daemon.programs.getSandbox(programId); console.log(data.firewall, data.problems); ``` ```bash curl -X POST "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com/api/v1/daemon/programs/edit/PROGRAM_ID" \ -H "Content-Type: application/json" -d '{"sandbox": null}' ``` Clearing the block replaces the loaded definition, so the instance running under the old policy stops and `boot` and `lazy_load` decide whether one starts again. The old revision's ingress rules stay installed until every instance under them is confirmed stopped, so `configured: null` is not proof that the rules are gone: watch `firewall` until it reads `none`. ### Reinstall rules that something else removed If a tool inside the container flushed the `ip hoody_daemon` table, programs that are already running keep listening unfiltered. A disable followed by an enable retires and re-admits the policy, which rebuilds the table; it interrupts the program. ```bash # Interrupts the program. Check each response before the next command. hoody daemon programs disable "$PROGRAM_ID" -c "$CONTAINER_ID" hoody daemon programs enable "$PROGRAM_ID" -c "$CONTAINER_ID" hoody daemon programs start "$PROGRAM_ID" -c "$CONTAINER_ID" --port 8080 --if-not-running hoody daemon programs sandbox get "$PROGRAM_ID" -c "$CONTAINER_ID" ``` Expect `firewall: "ok"` once any drain finishes, and read `problems` if it says `degraded`. Rebuilding the table also rewrites the snapshot from the state records, so while the state file is intact a missing snapshot is replaced, and so is a damaged one that can still be read as text. A corrupt state file, or a missing state file beside a surviving snapshot, is not repaired: the daemon refuses to regenerate it. If both files are missing and the marker remains, the daemon rebuilds an empty state only when no sandbox scope, launch in progress or ingress table remains. For the cases it refuses, and when the boot restore hook blocks a restart, restore the committed pair from backup and run the restore hook inside the container, as described under [container prerequisites and restart recovery](#container-prerequisites-and-restart-recovery). To remove a sandbox, send `"sandbox": null` on the edit endpoint, `sandbox: null` from the SDK, or `--sandbox null` from the CLI. ## Inspect a sandbox `GET /api/v1/daemon/programs/{id}/sandbox` reports what the daemon would enforce beside what the kernel is holding. It is diagnostic only; nothing in it is settable. An existing program always answers 200, and an unknown id is a 404. A program with no effective sandbox, or one whose stored policy no longer validates, answers with `rev: null` and `effective: null`. | Field | Meaning | |-------|---------| | `configured` | The stored sandbox block, exactly as persisted, or null when the program has none. A no-op block appears here only when it was loaded from a hand-edited file. | | `rev` | Policy revision: the first 12 hex digits of the SHA-256 of the canonical spec. It is the suffix of the program's chain name, and its scope names carry the first six digits. Null when the program has no effective sandbox. | | `effective.bwrap_argv` | The bubblewrap command line built from the stored policy: namespace flags, the root bind, the `writable` and `hidden` mounts, and the working directory. It is a diagnostic template, not a runnable command or a reading of a running process: `` stands for each writable mount's descriptor and `` for the command. Landlock and `RLIMIT_NOFILE` are applied inside the wrapper, so they do not appear. | | `effective.scope_argv` | The `systemd-run --scope` command line carrying the process limits, with `` and `` placeholders in the scope unit name. | | `live.table_present` | Whether the `ip hoody_daemon` table was read successfully. `false` covers both an absent table and a failed query. | | `live.chain` | The expected chain name for the current revision, `hd__`, when table inspection succeeds; null when the policy owes no ingress rules or the inspection failed. It does not by itself prove the chain exists. | | `live.rules` | An array of nftables JSON rule objects for the current policy chain, in evaluation order. Empty when the chain does not exist or the inspection failed. | | `problems` | Strings for each diagnostic collected: the stored policy no longer validates, the firewall state file cannot be read, an enabled program's ingress policy has neither an active record matching its current revision nor any draining record for this program (a disabled program owes no rules, so it reports no problem for that), an active record belongs to a revision other than the current one, the live table does not match the committed fingerprint, or an earlier definition of this program may still be running without the confinement its policy declares, which names the groups that have to be measured before its records can drain and appears only while the program still owns a record. Empty when no diagnostic was collected. | | `firewall` | One of `ok`, `degraded`, `draining` or `none`. | The four `firewall` values: - **`ok`**: this program's current revision has a matching active ingress record, no drain is in progress, and the live table matches the fingerprint recorded at the last successful apply. - **`degraded`**: `problems` is non-empty, which includes a live table that is missing, unreadable, or different from the committed fingerprint. The fingerprint excludes handles, metadata and named-set elements, so rate-meter churn never causes this; static allowlist expressions are included. `degraded` takes precedence over `draining`. - **`draining`**: a draining record exists for this program, its current revision after a disable or an older revision after an edit, because instances running under it have not been confirmed stopped. This is reported even after the block was cleared. - **`none`**: verified that no active or draining record exists for this program under any revision and that nothing about it owes rules. A disabled program with an ingress policy and no remaining record reads `none`; its `configured`, `rev` and `effective` are still reported. An active record left behind by another revision, which is what a retirement that did not complete looks like, is reported as a problem rather than passed over. Another program may still own a live table. Read `firewall` and `problems` together: `problems` names the reason whenever `firewall` is `degraded`. ```bash # Read the stored policy, its argument vectors and the live firewall state hoody daemon programs sandbox get "$PROGRAM_ID" -c "$CONTAINER_ID" ``` ```typescript const state = await containerClient.daemon.programs.getSandbox(programId); ``` ```bash curl "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com/api/v1/daemon/programs/PROGRAM_ID/sandbox" ``` Reads the stored policy, the argument vectors it resolves to, and the live nftables state for one program. **Response** for an unsandboxed program in a container with no prior firewall history, no records and no table: ```json { "success": true, "configured": null, "rev": null, "effective": null, "live": { "table_present": false, "chain": null, "rules": [] }, "problems": [], "firewall": "none" } ``` ## Health `GET /api/v1/daemon/health` carries a `sandbox` object that probes the mechanisms this host has. A sandboxed program fails closed when a mechanism its selected policy requires is missing. Capability probes are cached after their first evaluation, firewall integrity is queried separately, and sandbox problems do not change the endpoint's HTTP 200 or its top-level `status: "ok"`. | Field | Type | Meaning | |-------|------|---------| | `bwrap` | boolean | Whether `/usr/bin/bwrap` is present and executable. | | `nft` | boolean | Whether `/usr/sbin/nft` is present and executable. Without it no ingress policy can be applied. | | `systemd_run` | boolean | Whether `/usr/bin/systemd-run`, which owns the memory and task limits, is present and executable. | | `landlock_abi` | integer | Landlock ABI level reported by the kernel. `mode: restricted` needs 4 or higher. 0 when Landlock is unavailable. | | `firewall` | string | Container-wide state of the ingress table: `ok` when the live table matches the recorded fingerprint, `degraded` when it does not, when the table cannot be read while a fingerprint is recorded, or when the firewall state file cannot be read, `none` when the committed state holds no active and no draining record. It never says `draining`; per-program drains are visible only on the sandbox endpoint. | This value is computed from committed records alone. It never validates the desired policies in `programs.json`, and it reads `ok` while records are still draining after the last ingress policy was removed. It also returns `none` when its first existence checks find no state file, no snapshot and no marker, so an error those checks conceal is not diagnosed here. ## Errors and status codes - **400 with the JSON envelope** is every controller policy rejection: `user: root`, `terminal_id`, `mode: restricted` without a resolvable listening set, a `writable` or `hidden` path overlapping a protected root, an unsecure environment name, a port already owned in either direction, and an explicit program `id` of 2147483647, since the accepted range stops one below it. These carry `{ "success": false, "error": "..." }`. Validating a sandbox object that restricts something reads the firewall state, and an unreadable state there returns 500 with `INTERNAL: `. A block that normalizes away restricts nothing and is treated as an ordinary program, so on a program with no effective sandbox it does not enter that read and an unreadable state cannot make it fail. An edit validates the merged program, so omitting `sandbox` does not bypass the checks for a block that is already stored. An ordinary unsandboxed request can also read the firewall state for its port-ownership check, but an unreadable state on that path is logged and tolerated. Enabling or replacing an effective block can fail on the firewall state or on admission. A `sandbox: null` value skips the validation read, but an edit that clears or replaces a program's existing effective sandbox still reads the firewall state first: if that state cannot be read, the edit returns 500 with `INTERNAL: ` and the stored sandbox is unchanged. An earlier validation failure can return before it. - **400 with `invalid program body: ...`** is a body the add or edit handler cannot parse: malformed JSON, a field of the wrong type, or an unknown key inside `sandbox`. It carries the same JSON envelope. A repeated top-level key answers `duplicate field "": ...`, and a near-miss of `sandbox` such as `sandbx` or `Sandbox` is refused when the exact `sandbox` key is absent; other unknown top-level keys are ignored. - **Plain-text 413** is a body over the 1 MiB limit, refused before the handler runs. - **400 with the program persisted** is a create, edit or enable of a program that has or had an effective block whose supervisord apply failed, for example a reload timeout or a validation refusal. The response says so, the firewall rules are kept, and the next apply retries the reload. A sandboxed start whose apply fails refuses to start the program unconfined. A program without an effective block also answers 400 with the program saved (`saved but NOT applied`) when supervisord refuses its configuration, cannot confirm it can load it, or the configuration cannot be written; other apply failures are logged. - **500** is a failed write of `programs.json`, with an error string starting `PERSIST: `, for every program, sandboxed or not; the change is not kept. It is also, for a request that carries or replaces an effective block, any firewall-state read failure, nft failure, or applied table that could not be read back, starting `INTERNAL: `. Not every endpoint can produce both halves, and the ones that cannot do not document them: add, edit and enable can produce either; start admits a policy but persists nothing, so only `INTERNAL: `; disable, remove and reset persist but make no admission of their own, so only `PERSIST: `, and a retirement that fails is logged rather than returned; stop neither persists nor enters a firewall transaction, so it has no 500 at all. If an applied table cannot be read back, the previous rules are put back and nothing changes. For `PERSIST`, the firewall admission made for that request is reverted for that program alone; if the revert itself fails, the rules stay in place and the failure is logged. If a firewall change is applied and then cannot be saved, the daemon puts the previous rules and both state files back. If that restoration also fails, or if an apply times out and its retry fails too, the installed rules no longer match the saved state. The daemon then refuses to start programs that need ingress rules. A program that owns no rules, including one with a filesystem-only or process-only policy, is unaffected. The refusal lifts by itself as soon as any firewall change commits cleanly; repair the committed state, its matching snapshot and the kernel rules, then make a change that commits. The refusal lives in the running daemon and gates admission and boot application only. It does not stop processes that are already running, and neither health nor the sandbox endpoint reports it, so an `ok` there is a statement about records and table integrity rather than proof that the refusal has lifted. - **Retirement failures never block cleanup.** On remove, disable, or a disabling edit of a program with an effective block, a failure to retire its firewall record is logged and does not stop the daemon from attempting the supervisord cleanup. The record stays, with its rules and port reservation, until a later successful daemon start reconciles it. The supervisord cleanup itself is reported. Remove, disable and a disabling edit all answer with the JSON error envelope, not with success, when the configuration could not be removed or the reload failed after the change was persisted; the file may already be gone, so read the daemon log for which of the two it was. At startup the daemon sweeps configurations belonging to program ids that no longer exist. A disabled program keeps its id, so its configuration is not swept: fix the cause and call disable again. That covers the retirement step of remove, disable and a disabling edit; it is not an account of every firewall-state access. A disabling edit can also fail earlier, in the validation of the merged policy or in the firewall checkpoint. - **Reset** replaces the program definitions with the configured defaults. A delayed apply from an earlier reset generation is withdrawn before publication, so it cannot republish a stale configuration afterwards. Firewall reconciliation errors are logged, and a reset can return success while the programs that carry an ingress policy are skipped at boot. ## Known limitations - The launch compares each writable and hidden path by device and inode, and passes each writable directory to bubblewrap as an open descriptor, so the directory mounted writable cannot be swapped. bubblewrap still resolves the destination of every mount by pathname, for writable binds and hidden mounts alike. A tenant who can replace a validated directory entry, or any mutable directory above it, can therefore move where a hidden mount lands, so the content it was meant to hide stays visible under the name it was moved to, or move where the pinned writable directory appears. A hidden mount only covers what is at its destination, so neither case reaches anything the account could not otherwise reach, but the first defeats a restriction the sandbox was asked to add. Nothing rechecks identity while the program runs. - The provisioning guard reads the default `config.json` only, and the restore hook needs Python 3 and `flock`. See [container prerequisites and restart recovery](#container-prerequisites-and-restart-recovery). - `/proc` is the container's, so processes outside the sandbox are listed. See [what changes for a sandboxed program](#what-changes-for-a-sandboxed-program). - Landlock is TCP-only, and it covers `bind` and `connect`, not a socket that starts listening without binding. Ingress rules are per destination port on eth0, IPv4 only. See [ingress firewall](#ingress-firewall). - `udp: "deny"` refuses every `socket()` call made through the legacy 32-bit `socketcall` interface (a `socketpair()` is still allowed), and moves DNS to TCP only for the standard glibc resolver. See [network fields](#network-fields). - There is no disk-space quota for `writable` paths and no limit on disk bandwidth. `max_file_size` caps one file at a time, and `max_cpu` applies per instance of a port-range program. - `mode: restricted` has no MPTCP, no SMC, no io_uring and no TCP Fast Open send. A program that requires io_uring does not run under `restricted`. A 32-bit x86 program that uses the legacy `socketcall` interface cannot create any socket through that interface there (Unix sockets included) or send through it; `socketpair` through it, and the direct 32-bit socket calls, still work. A socket or a ring created outside the sandbox and passed in over a Unix socket is not covered. A program that was already running under `restricted` gets this at its next start. - A sandboxed program with a `display` can lose sound, because the display's audio socket normally lives under `/run/user`. - Hiding `/run/user` covers the standard location of the user manager and the session bus. One that is reachable through another mount path, a runtime directory configured elsewhere, or a session bus on an abstract socket (reachable under `full` and `restricted`) is not covered. - Ownership rules are enforced by the API and by the boot path, which both see every program. The renderer and the launch wrapper validate the policy's own shape only, so neither re-checks ownership against neighbours it cannot see. - A signal death reaches supervisord as exit code `128+n`, and a policy change replaces the loaded definition, stopping the instance that ran under the old one. See [what changes for a sandboxed program](#what-changes-for-a-sandboxed-program). - Rate-limit budgets reset on every table regeneration, and the meter fails open when its set is full. See [ingress firewall](#ingress-firewall). - A firewall drain can outlive the API call that started it. See [policy changes and draining](#policy-changes-and-draining). - A read-only root is not a confidentiality boundary. Every file the program's account can read stays readable unless a `hidden` entry covers it, and protected trees such as `/etc`, `/run` and `/hoody` cannot be hidden. A program with `inject_container_env: true` also receives the container's environment, from which only specific unsafe names are filtered, never your application's own secrets; see [container environment](/kit/daemons/#container-environment). Run untrusted programs without ambient credentials, and give their account only the file permissions they need. - Quarantine ownership is best effort. When two programs have both used one group name, the program being quarantined keeps the name and can stop the other program's group. The ownership check and the stop are separate steps, so a name can change hands between them. The fallback reads configuration files and markers, which do not certify what supervisord has actually loaded. See [container prerequisites and restart recovery](#container-prerequisites-and-restart-recovery). - For an effective sandbox, `user: root`, `terminal_id`, capability grants, and `private_tmp` together with a writable path that meets `/tmp` or `/var/tmp`, a working directory below either, or a daemon installed below either, are all rejected. Quick-start rejects any non-null `sandbox` value. See [the sandbox block](#the-sandbox-block) for the exact overlap rule. ## Use Cases ### Application servers Give a Node.js, Python or Go server a read-only root with one writable data directory, hide `/root`, and declare its port in `bind_ports` so the port is reserved. Reserving a port does not itself filter traffic; add `ingress_allow_from` or `ingress_rate_limit` for that. ### Background workers Run a queue processor under `mode: restricted` with `connect_ports` limited to selected TCP destination ports, and cap memory and tasks to reduce the worker's impact on other programs. ### Public services Front an internet-facing service with `ingress_allow_from` and `ingress_rate_limit`. With `ingress_allow_platform: true`, loopback sources and the gateway address bypass both. A public request through the proxy keeps the client's address, so the allowlist has to include the networks you expect clients to come from. For when to filter here rather than at the proxy or the container firewall, see [allow only certain client IPs](/foundation/networking/firewall/#allow-only-certain-client-ips). ## Best Practices ### Start with the filesystem `read_only_root` with one `writable` directory is the cheapest policy to reason about. Create the writable directory before the call; the daemon refuses a path that does not exist. ### Declare every port Under `mode: restricted`, list every port the program binds for TCP. A port outside the listening set cannot be bound for TCP: that set is `bind_ports`, or `port_range` and `ready_port` when `bind_ports` is omitted. Undeclared ports cannot be reserved from the command itself, but traffic to an undeclared service is still filtered when its destination port matches installed ingress rules. ### Resend the whole block on edit An edit replaces the sandbox as a unit, so read the program first, change the fields you need, and send the complete block back. See [normalization and edits](#normalization-and-edits). ### Watch the drain after a change Poll `GET /api/v1/daemon/programs/{id}/sandbox` until the drain ends. Expect `ok` when the current policy has an active ingress record, or `none` when this program has no remaining ingress record. Investigate `degraded`, its `problems` entries, or an HTTP error. ## Useful Questions **Q: Does adding a sandbox change anything for my other programs?** Other programs keep their existing rendering. Sandbox port reservations can reject conflicting changes, and ingress rules also affect other listeners on the same destination port. **Q: Why did my program restart after an edit?** The resolved policy changed, so its revision and rendered configuration changed. See [what changes for a sandboxed program](#what-changes-for-a-sandboxed-program). **Q: Why does my program show exit code 137?** A signal death is reported as `128+n`, so a `SIGKILL` and an OOM kill are both 137. See [what changes for a sandboxed program](#what-changes-for-a-sandboxed-program). **Q: I sent a sandbox block but the program does not show one.** The block restricted nothing, so it was stored as absent. See [normalization and edits](#normalization-and-edits). **Q: Can I sandbox a program that runs as root?** No. See [what changes for a sandboxed program](#what-changes-for-a-sandboxed-program). **Q: Does the rate limit apply to traffic through Hoody Proxy?** Yes, for ordinary public requests: the proxy forwards them with the client's own source address, which the platform exemption does not cover. It applies to hairpinned requests too, since the proxy binds those to the address the client dialed rather than to the gateway. The exemption covers only traffic whose source is loopback or the gateway. See [ingress firewall](#ingress-firewall). ## Troubleshooting ### 400 `writable` without `read_only_root` **Cause**: `filesystem.writable` was set on its own. **Solution**: Set `read_only_root: true`, or drop `writable`. ### 400 on a path under `/hoody` **Cause**: `/hoody` and `/hoody/storage` contain the daemon's own tree, so they are protected roots, and overlap is checked in both directions. **Solution**: For `writable`, use a subdirectory such as `/hoody/storage/apps/`. A `hidden` path may not overlap the `/hoody` tree at all. ### 400 `mode: restricted` without a listening set **Cause**: The program has neither `port_range` nor `ready_port`, and `bind_ports` is empty or absent. **Solution**: Add `bind_ports`. Under `restricted` the program cannot bind any TCP port that is not listed. ### 400 on a port another program declares **Cause**: The port is in an enabled program's `port_range`, `ready_port` or `bind_ports` where a sandbox is involved on either side, or a firewall record, active or draining, still exists for a program that declared it, including a disabled or removed one. The legacy `port_range` overlap check counts disabled programs too. **Solution**: For a declaration conflict, change or remove the declaration, or choose another port. Waiting helps only with a record conflict: `GET /api/v1/daemon/programs/{id}/sandbox` on the other program shows the drain, and a removed program answers 404. ### 400 on an environment variable name **Cause**: The name is one of the glibc-unsecure names, such as `LD_PRELOAD` or `TMPDIR`, and the program has an effective sandbox. **Solution**: Remove the variable. ### Name resolution fails with `udp: "deny"` **Cause**: Under `mode: restricted`, TCP port 53 is not in `connect_ports`, or the program resolves names with its own resolver or a C library other than glibc, which keeps using UDP. **Solution**: Add 53 to `connect_ports`, or configure the program's resolver for DNS over TCP. If neither is possible, keep `udp` at `allow`. ### Program never starts and health shows a missing mechanism **Cause**: A sandboxed program fails closed when a mechanism its policy requires is missing: `bwrap` or `systemd_run` is false, `nft` is false with an ingress policy, or `landlock_abi` is below 4 with `mode: restricted`. **Solution**: Read the `sandbox` object on `GET /api/v1/daemon/health` and install the missing mechanism. The wrapper's own `sandbox:` line in the program's error log names the refusal. ### `firewall` reports `degraded` **Cause**: `problems` names it: the live table is missing, unreadable, or differs from the committed fingerprint; the stored policy no longer validates; the state file cannot be read; an enabled program's ingress policy holds no record; or an earlier definition of the program may still be running unconfined, which keeps its records and its port reservation. **Solution**: Inspect `problems` and the daemon's nft and state errors. Once those are resolved, an ingress-policy change that commits a new table can restore `ok`; an identical edit or an unrelated sandbox change may do nothing. For the unconfined-definition entry, repair what stopped supervisord from applying or removing the configuration and then apply or remove it again, or restart the daemon, which measures the recorded group names. That measurement clears the record when the recorded group list is complete and fully measured, no unwrapped configuration remains for supervisord to revive, and supervisord reports nothing running or only this program's own wrapper. A confirmed removal of the configuration, or removing the confinement from an enabled program whose ordinary configuration supervisord then acknowledges, clears it without a measurement. If the daemon is also refusing to start programs that need ingress rules, see [errors and status codes](#errors-and-status-codes) for what lifts that refusal. ### `firewall` stays `draining` **Cause**: Processes from the previous revision remain, or the daemon cannot confirm their absence. **Solution**: Stop the old processes and resolve any scope-observation errors. Finalization requires no processes in the old revision's scopes, no launch in progress, no supervisord change of the program still being applied, and no unconfined hold; see [policy changes and draining](#policy-changes-and-draining) and the hold conditions under [container prerequisites and restart recovery](#container-prerequisites-and-restart-recovery). ## What's Next