Skip to content
Hoody Documentation
Hoody.com
Download Page (.md)

Program Sandbox

Section titled “Program Sandbox”

This page covers the optional sandbox block on a daemon program: what each field does, how the daemon normalizes and validates the block, what changes for a program that carries one, what the container needs for it, and how to read the sandbox state back.

A sandbox confines one program with kernel-enforced mechanisms: bubblewrap in a user namespace for the filesystem, Landlock for TCP bind and connect, an optional block on UDP sockets, a transient systemd scope for memory, task and CPU limits, in-process limits on open files and file size, and one nftables table for inbound connections. You set it from programs.json or from the add and edit endpoints, like every other program field. Programs without an effective sandbox keep the ordinary rendering path, including the serialization of their environment values. No program’s environment values are escaped: a value that contains a double quote is written in single quotes. Which quoted values a request may send is described under required fields and defaults. A sandboxed program’s values may contain no quote of either kind, %, \ or control character; such a value in its own environment is refused, and one taken from the container’s environment is dropped. There is no periodic enforcement loop.

What the sandbox does not do

Section titled “What the sandbox does not do”

A sandbox applies the restrictions you configure, and nothing else. Read this before relying on one against hostile code.

  • It restricts writes, not reads. A read-only root still lets the program read every file its account can read, including secrets in trees that cannot be hidden. The container’s /proc is the program’s own, so other processes in the container stay visible.
  • restricted limits TCP ports, not hosts. The listed ports are allowed to any destination. UDP, ICMP and raw sockets are outside the policy, and so is a socket that starts listening without binding. To stop UDP as well, set udp: "deny", which works under every mode. Unix sockets reachable by path stay reachable, under every mode.
  • restricted has no MPTCP, no io_uring and no TCP Fast Open send. So that the port lists hold, a restricted program cannot create an MPTCP socket (the request answers EPROTONOSUPPORT) or an SMC socket, cannot set up an io_uring ring (io_uring_setup answers ENOSYS), and cannot send with MSG_FASTOPEN (the send answers EOPNOTSUPP). Software that falls back to plain TCP, to epoll, or to connect-then-send keeps working. A program that requires io_uring does not run in this mode. No other mode is affected.
  • Every sandbox hides /run/user. A sandboxed program sees an empty /run/user in place of the real one, so systemctl --user, the account’s session bus and anything else in its runtime directory are out of reach from inside the sandbox.
  • Ingress rules cover new IPv4 TCP connections on eth0, on the ports the policy declares. Established connections are never cut, and packets whose SOURCE is loopback or the gateway are exempt by default. That exemption is by address, not by caller: requests Hoody Proxy forwards from the internet arrive with the client’s own address and are filtered and metered like any other source.
  • The rate limit is best effort. Budgets reset whenever the table is rebuilt, and once its 65535-source set is full, new sources are not metered at all. The allowlist keeps applying.
  • Running programs are not re-checked. A launch refuses when the firewall table does not match the committed state, but a program already running keeps its listener unfiltered until a firewall transaction or a daemon start reinstalls the rules.
  • Hidden paths can be moved. An actor who can rename a validated directory, or a directory above it, can change where a hidden mount lands. Writable mounts are pinned by descriptor, but their destination is still resolved by path, and nothing is re-checked while the program runs.
  • Confining a program does not clean up what ran before it. A detached child of an earlier unsandboxed run is outside everything the daemon observes.
  • There are no disk-space quotas. max_file_size caps each file and tmp_size caps the private temporary directories, but nothing limits the total space a program writes to its writable paths.
  • It needs a non-root account, and it is per program: creating a container confines nothing by itself.

API Endpoints Summary

Section titled “API Endpoints Summary”

All endpoints are relative to your Daemon Manager service URL:

https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com

The examples on this page show the CLI, the SDK and HTTP. The SDK examples use a containerClient created as on the Daemons page.

The CLI adds one guard of its own. An edit that sets any --sandbox-* flag first reads the stored block (configured). When the new block would omit or null a stored setting, it sends nothing and lists each dropped setting with the flag that keeps it. A stored false counts as a setting; stored nulls and empty lists do not. If that first read fails, the edit is refused too. --sandbox-replace skips the check entirely, and --sandbox null clears the block without it. The guard only looks for dropped settings: it does not judge whether the values you send are weaker. HTTP and SDK edits have no such guard.

When to use a sandbox

Section titled “When to use a sandbox”

Use a sandbox for a program that serves traffic or runs code you do not fully control: a web app that should write only to its data directory, a worker that should reach only selected TCP destination ports, or an internet-facing service that should accept connections from a known network at a bounded rate. Each concern maps to one sub-object, and you can set any one of them on its own.

The sandbox block

Section titled “The sandbox block”

The block has three sub-objects, filesystem, network and process. Every one is optional, and an optional field set to null means absent. Unknown keys are rejected at the sandbox root and inside each sub-object, so a misspelt field is a 400 rather than a quietly weaker policy, and a field a newer client sends is a 400 against an older daemon. private_pid, no_new_privileges and capability-grant fields are unsupported. A resolved policy larger than 64 KiB is refused with sandbox policy is too large. That size is measured on the spec as it will be transported, carrying the program’s own name, the widest port of its instance range and the widest id the daemon can assign, so a policy accepted here is never refused afterwards by the stage that receives it.

{
  "name": "my-api",
  "command": "node server.js",
  "user": "api-svc",
  "directory": "/hoody/storage/apps/my-api",
  "port_range": { "start": 8000, "end": 8003 },
  "sandbox": {
    "filesystem": { "read_only_root": true, "writable": ["/hoody/storage/apps/my-api"], "hidden": ["/root"] },
    "network": {
      "mode": "restricted",
      "bind_ports": ["8000-8003"],
      "connect_ports": [443, 53],
      "ingress_allow_from": ["203.0.113.0/24"],
      "ingress_allow_platform": true,
      "ingress_rate_limit": "50/s"
    },
    "process": { "max_memory": "512M", "max_pids": 256, "max_open_files": 1024, "private_tmp": true }
  }
}

Filesystem fields

Section titled “Filesystem fields”

read_only_root stops writes to the root tree; it does not restrict reading, and hidden is what removes a directory from view. Protected trees such as /etc, /run and /hoody cannot be hidden, so secrets a program’s account can read there stay readable. /proc is the container’s, bound read-write, so other processes in the container remain listed; /dev, a private /tmp and the hidden tmpfs mounts are separate mounts and are not read-only. /run/user is hidden in every sandbox, whatever the policy, including a runtime directory that appears after the program started; a sandboxed program whose directory is under /run/user is rejected with 400.

FieldAcceptsDefaultEffect
read_only_rootbooleanfalseMounts the root tree read-only. Three mounts are outside that: /proc is bound read-write, so per-process files such as /proc/self/oom_score_adj stay writable; /dev is a fresh device mount; and /sys is read-only for every sandboxed program regardless of this flag. Private temporary directories and hidden mounts have their own behaviour, and ordinary account permissions still apply.
writablearray of absolute directory stringsnoneBound read-write inside the read-only root. A non-empty list requires read_only_root: writable on its own is a 400, not a restriction. An empty or null list is inert.
hiddenarray of absolute directory stringsnoneReplaced by an empty, unreadable tmpfs. Follows the same overlap rule as writable, and additionally may not overlap /usr, /bin, /sbin, /lib, /lib64, /lib32, /libx32, /hoody, /hoody/storage, the program’s own directory, or any writable entry. An empty or null list is inert.

Both lists require existing absolute directories with no .. component, using only ASCII letters and digits, /, _, ., -, space and tab; each is canonicalized, and / itself is forbidden. A path may neither equal, contain, nor be contained by a protected root: the storage and supervisord configuration trees, the log base directory, /etc, /run, /var/run, /hoody/plugins, /proc, /sys, /dev, /boot, and the daemon executable’s directory. /hoody and /hoody/storage are therefore refused because they contain the daemon’s own tree, while a directory such as /hoody/storage/apps/my-api is accepted as writable when its canonical path avoids this daemon’s configured protected roots. Each writable and hidden path is checked again at every launch, and a path that changes identity during the launch, or that is no longer a directory, is refused. The identity the API saw when the policy was stored is not pinned. The directory mounted writable is the directory that was checked. The DESTINATION of every mount is still resolved by pathname: an actor who can rename a validated directory, or any directory above it, can move where a hidden mount lands and leave the contents it was meant to hide visible under another name. Nothing rechecks these paths while the program runs, so do not rely on hidden where untrusted code can rename those directories.

Network fields

Section titled “Network fields”
FieldAcceptsDefaultEffect
modefull, restricted or nonefullfull adds no Landlock bind or connect restriction; ingress allowlist and rate settings still apply. restricted confines TCP bind and connect with Landlock to bind_ports and connect_ports. none runs the program in an empty network namespace, which closes IP sockets and abstract unix sockets but not a unix socket reachable by a filesystem path.
bind_portsports and "a-b" rangesunion of port_range and ready_port when omitted or nullTCP ports the program may bind under restricted. Also the listening set for ingress rules and port ownership. An explicitly empty list receives no defaults.
connect_portsports and "a-b" rangesnoneTCP destination ports the program may connect to, on any host. Valid only with mode: restricted; an empty list under restricted denies every TCP connect, and a non-empty list with any other mode is a 400.
ingress_allow_fromup to 64 CIDRs, each a strict IPv4 dotted quad a.b.c.d/n with octets 0 to 255 and a prefix from 0 to 32 in plain decimal (/24; /024 and /+24 are refused)noneSource networks allowed to open new connections to the listening set on eth0. Requires a non-empty listening set. An empty list, a null, or the field omitted applies no source restriction rather than denying everyone; a rate limit still applies.
ingress_allow_platformbooleantrueReturns traffic from 127.0.0.0/8, and from the platform address, before the rate meter and the allowlist. The platform address is the eth0 default-route gateway, and when there is no default route it is the first host address of the first connected eth0 route. It is looked up on the first firewall transaction and remembered once found; a lookup that fails is retried by the next transaction, and the table rendered in between covers loopback only.
ingress_rate_limit"N/s" or "N/m", N from 1 to 100000nonePer-source budget for new connections, evaluated before the allowlist. The rule meters matching SYN packets and allows a burst of N packets on top of its replenishment rate, so it is not a fixed-window count of completed connections. Also requires a non-empty listening set.
udpallow or denyallowdeny stops the program from creating UDP sockets, under every mode: the request fails with EACCES. ICMP ping sockets and SCTP sockets are refused too, and the program cannot set up an io_uring ring. Name resolution through the system’s standard (glibc) resolver switches to TCP, so under mode: restricted add TCP port 53 to connect_ports.

restricted allows the listed TCP destination ports on any host: it is a port policy, not a host policy. It does not cover UDP (set udp: "deny" for that), ICMP, raw sockets or Unix sockets, and it restricts bind and connect, so a socket that starts listening without binding is outside it and has no ingress rule. full and restricted share the container’s abstract Unix-socket namespace; none gives the program its own. Under mode: none the filesystem policy is what decides who the program can still talk to, because a unix socket it can open by path stays reachable. A port item is an integer from 1 to 65535 or an inclusive range "a-b" with 1 <= a <= b <= 65535; zero, reversed ranges and out-of-range endpoints are rejected, and overlapping or adjacent items are merged. Landlock restricts TCP destination ports, not hosts or protocols: under restricted, UDP, ICMP and raw sockets are outside the policy. It covers bind and connect, not a socket that begins listening without binding, so the declared listening set is what the ingress rules cover. A bind or connect that Landlock denies fails inside the program with EACCES (Permission denied), and the restriction is inherited by every process the program starts. Ingress filtering behaves differently: it silently drops denied packets. A source the allowlist refuses normally sees a connection timeout, while a rate-limited connection can still succeed when a retransmitted SYN arrives after the budget refills. For an effective sandbox, bind_ports must cover port_range and ready_port when those are set; when neither is set, bind_ports is required for mode: restricted. none rejects port_range, ready_port, and any non-null bind_ports value including []; it rejects a non-empty connect_ports or ingress_allow_from and any rate limit, while empty connect and allowlist arrays are inert and ingress_allow_platform is accepted and inert. Firewall records committed by an older daemon with a leading-zero prefix such as /024 are read as the decimal /24 that was meant. The program’s stored sandbox block is not rewritten, so correct such a spelling with an edit before the policy is applied again; a committed allowlist entry that cannot be read at all makes loading the firewall state fail.

With udp: "deny", software that brings its own resolver, or a C library other than glibc, has to be configured for DNS over TCP itself, or its lookups fail. Under every mode except none, the launch is refused when the container’s /etc/resolv.conf is missing or cannot be read, is not a regular file, is larger than 64 KiB, or resolves into a hidden, writable or private temporary path or under /run/user or /dev, which every sandbox mounts over. Under mode: none the program has no network to ask, so /etc/resolv.conf is not checked. A 32-bit x86 program that creates sockets through the legacy socketcall interface cannot create any socket that way under deny, Unix sockets included. A socket created outside the sandbox and passed in over a Unix socket is not covered.

The Hoody Kit ports 4, 5, 40, 44, 45, 46, 48, 50, 55, 59, 60, 61, 75, 76, 77, 78, 3971, 3998, 3999, 4000 and 23333, plus the daemon’s own effective port, are reserved against sandboxed listening sets.

Process fields

Section titled “Process fields”
FieldAcceptsDefaultEffect
max_memory<n>[KMG], or a bare decimal byte count, from 16M to 64GnoneWritten to the scope as MemoryMax= together with MemorySwapMax=0 and OOMPolicy=continue.
max_pidsinteger from 4 to 65536noneTasksMax= on the scope. The count includes the two wrapper helpers, so budget at least 2 above the program’s own process and thread count.
max_open_filesinteger from 8 to 1048576noneRLIMIT_NOFILE, soft and hard, set inside the sandbox after the helpers are set up and immediately before the program starts, so the helpers’ descriptors are not charged to it.
max_cpu"<n>%", n from 1 to 102400noneCPU time cap, measured against one CPU: "50%" is half of one CPU and "200%" is two full CPUs. A port-range program gets the cap once per instance, so its instances can use that many times the cap together.
max_file_size<n>[KMG], or a bare decimal byte count, from 1K to 1024GnoneThe largest single file the program may write. A write that would grow a file past it fails with EFBIG (File too large), and the program handles that error; it is normally not killed. It caps each file, not total disk use, and does not cover the program’s log files.
private_tmpbooleanfalseGives the program its own empty /tmp and /var/tmp. A 400 when any canonical filesystem.writable path equals, contains, or lies inside /tmp or /var/tmp: the private mounts go on first, so a writable bind that meets one of them would put the container’s shared directory back. Also a 400 when the program’s directory lies strictly below either path, or when the daemon itself is installed at or below either: the fresh mounts hide them, and the launch would fail without reaching the program.
tmp_size<n>[KMG], or a bare decimal byte count, from 1M to 64GnoneSize of each private /tmp and /var/tmp. Requires private_tmp: true, and is a 400 without it. Each of the two directories gets the full size, so together they can hold twice the value. Files there are held in memory and count against max_memory. A write past the size fails with ENOSPC.

K, M and G are binary units. A quoted decimal byte count within the same limits, such as "16777216", is accepted too. The swap cap is what makes the memory limit enforceable: on a container with swap, MemoryMax alone lets a program exceed its cap by swapping. With OOMPolicy=continue the kernel kills only the offending process; when that process is the program itself, supervisord sees exit code 137 and the wrapper then tears down the rest of the scope. The program sees /sys read-only, so it cannot raise its own limits.

Creating the scope is not taken as proof that the limits exist: systemd creates a scope even when a controller is unavailable and drops the limit. Before the program runs, the wrapper reads back each limit the policy requested from the scope’s cgroup, memory.max and memory.swap.max for max_memory, pids.max for max_pids, and the CPU quota for max_cpu. A file that is missing or unreadable, a value of max, or a value other than the request refuses the launch with exit 78 and sandbox: the scope was created but its limits were not installed: .... memory.max may equal the request rounded down to a page boundary, and the CPU quota may differ from the request by up to 1%. Limits the policy did not request are not checked, and systemd_run: true on health only says that the executable exists.

Normalization and edits

Section titled “Normalization and edits”

Defaults and merged port ranges determine the resolved policy. The block is effective only if a restriction remains: read_only_root, a non-empty hidden, a mode other than full, a non-empty ingress_allow_from, ingress_rate_limit, udp: "deny", max_memory, max_pids, max_cpu, max_open_files, max_file_size or private_tmp. A block that restricts nothing is stored as absent and claims no ports; ingress_allow_platform alone, at either value, and bind_ports alone are both ineffective. The API removes an entirely ineffective block but does not expand an effective block into its resolved defaults: effective blocks retain their configured values, and optional null fields are treated as absent and may be omitted when read back. A no-op block can survive only in a hand-edited programs.json.

On programs/edit/{id} the block is three-state:

  • Absent leaves the current block unchanged.
  • null clears it.
  • An object replaces the stored block as a unit. There is no field-level merge into a sandbox block, so an edit must resend every field it wants kept.

On programs/add, absent and null mean the same thing. Quick-start does not support sandboxed execution: any non-null sandbox value, including {}, is a 400, and sandbox: null is accepted as absent. A top-level key that looks like a misspelt sandbox, such as sandbx or Sandbox, is a 400 there too, which catches common typos. A quick-start program always runs without confinement, so use a persistent program when you need it.

What changes for a sandboxed program

Section titled “What changes for a sandboxed program”
  • The pid supervisord tracks is the wrapper, not the program. The rendered configuration keeps the environment= line, drops user= because the uid drop happens inside the wrapper chain, and adds killasgroup=true.
  • Exit codes shift. Only the program is signalled. bubblewrap reports a signal death as 128+n and the wrapper exits with that code, so a SIGKILL reaches supervisord as 137 rather than as a signal. An OOM kill under max_memory is the same 137.
  • A policy change replaces the loaded definition. The 12-hex policy revision is part of the supervisord command line, so a change to the resolved policy changes the rendered configuration and supervisord reloads it. The reload stops whatever was running under the old definition; whether a new instance starts follows boot and lazy_load, so a boot program comes back and a lazy or non-boot one stays down until something starts it. Reordering or merging entries in bind_ports and connect_ports preserves the revision when the resolved ranges are unchanged. writable, hidden and ingress_allow_from keep their submitted order, so reordering one of them changes the revision and replaces the loaded definition, with the same start rules as any other policy change. An unchanged configuration skips the reload only when no earlier reload of that program is still unconfirmed. A disabled program’s configuration is removed instead, and a lazy or non-boot program is rendered with autostart=false, so an edit does not by itself start an instance.
  • Some environment names are refused. For an effective block, the glibc-unsecure names LD_*, GCONV_PATH, GETCONF_DIR, HOSTALIASES, LOCALDOMAIN, LOCPATH, MALLOC_*, NIS_PATH, NLSPATH, RESOLV_HOST_CONF, RES_OPTIONS, TMPDIR, TZDIR and GLIBC_TUNABLES are a 400 in environment. They are also filtered out of the container environment the daemon injects, with a warning.
  • Supervisord’s own variables are removed. The SUPERVISOR_* names, including the address of its control socket, are stripped from the program’s environment. An empty network namespace does not close a unix socket opened by path, so a confined program must not be handed the address of the process that supervises it.
  • A sandboxed program’s environment values must be literal. Supervisord expands %(...)s and %(...)c across the whole environment= line and then splits it with a shell-style lexer, and released versions build that lexer in a mode where a backslash is an ordinary character rather than an escape. A value carrying %, a quote of either kind, a backslash or a control character can therefore reappear as further assignments in the environment of the root wrapper process, and a value wrapped in single quotes would arrive with those quotes stripped rather than as written, which runs before any of this daemon’s checks. No escaping is correct for every supported parser version, so an effective block refuses such values outright, with a 400. A stored display is validated with the environment, so an unsafe one refuses the policy rather than quietly disappearing; the render path additionally drops an unsafe value from the sources the API does not own, the container environment the daemon injects and daemon variables a command references. Environment names are refused the same way when they belong to the dynamic loader or to the sandbox wrapper’s own transport, which is every name beginning HOODY_SANDBOX_, and both kinds are dropped from the container environment the daemon injects. Environment NAMES are held to the same split: a sandboxed program’s names must be C identifiers, while an ordinary program keeps whatever names it always had, minus any that would break the configuration file itself. Programs without an effective block follow the ordinary rule instead: a value may carry one kind of quote but not both, and no quote at either end, and a backslash is an ordinary character; see required fields and defaults. Where the configuration is written, command and environment are stripped of line feeds, carriage returns and NULs for every program, so a stored program cannot open a second supervisord section.
  • A program name may not be . or ... The name becomes a supervisord section header and a path component of the default log directory, so those two would put the daemon’s log writes outside the log base. The API refuses them, and so does the render path for a hand-edited or restored programs.json.
  • user: root is refused. An effective block on a program running as uid 0 is an HTTP 400 during API validation, and the wrapper refuses it again at launch. Root inside the sandbox is still uid 0 to the system bus and to supervisord’s socket, so the confinement would not hold.
  • terminal_id is refused. An effective block combined with terminal_id is a 400.
  • Launches fail closed; running programs are not re-checked. If anything flushes or changes the table under a program that is already running, that program keeps running without the filtering those rules gave it, and an identical edit or start does no firewall work, so it repairs nothing: a successful firewall transaction or a daemon start is what reinstalls the rules. Any stored sandbox block is validated when the program is rendered, not only one that restricts something, so a block that breaks a rule refuses the render instead of running the program unconfined. A program whose sandbox cannot be established never starts. An ingress policy must have matching installed rules, and the whole live ip hoody_daemon table must match what the last firewall change applied, so an external firewall reset is caught even though the stored policy is unchanged. A table that differs anywhere refuses the launch, and that includes a chain whose rules were flushed while its name survived. So does a live-table check that cannot complete within 5 seconds, or a table listing larger than 8 MiB; the listing grows with the rules and allowlists, not with the traffic a program receives. A firewall change already in progress delays a launch, and refuses it only when the change outlasts that 5-second check. A launch whose sandboxed stage does not start in time fails with handshake failed: payload never announced itself or outer never released the payload. That check covers launches; it does not repair a table that disappears under a program that is already running, which keeps its listener unfiltered until the next firewall transaction or daemon start reinstalls the rules. A removal or disable that completed before the launch refuses it; one that lands during the launch lets this instance start under the draining rules and drain with them, and a drain never removes the rules of an instance that is still starting.
  • Every launch rereads the stored program. The wrapper reads config.json and the whole programs.json at each launch. A file it cannot read or parse, including a malformed entry for another program, a row that is missing or disabled, a block that is no longer effective, and a policy whose revision differs from the one in the supervisord command line (revision mismatch: ... (re-apply the program)) all refuse the launch with exit 78. A hand edit of programs.json is read at the next launch but does not change the revision in the existing supervisord command. If the edit changes the resolved policy revision, that command refuses to launch. Use the edit endpoint, which stores the policy and republishes the configuration.
  • The command is resolved inside the sandbox. A first word containing / is used as written. Any other name is looked up on the program’s own PATH, where an empty component is ignored instead of meaning the working directory, so write . explicitly to search it. The sandbox tries each candidate as the program’s own user, so it skips a file that user may not execute and goes on to the next directory, as a shell’s lookup does. It also skips a path it cannot read for another reason, such as a symbolic-link loop or an over-long name. A file that is found but is not in a runnable format stops the search. A command that is missing, not executable by the program’s user anywhere on PATH, or cannot be executed refuses with sandbox: inner: ... and exit 78, and there is no shell fallback for a script without a valid interpreter line.
  • A refusal is loud. A configuration refusal prints sandbox: <reason> on stderr and exits with code 78, so the program’s own error log names the reason it never started. A launch cancellation or a handshake failure also prints a diagnostic, but can exit with a child’s failure code or with 143.

Inside the sandbox, --cap-drop ALL, --unshare-pid, --unshare-user, --unshare-ipc and no_new_privs are unconditional, so there is one wrapper shape for every policy. The IPC namespace is the program’s own, so System V shared memory and message queues do not reach other processes in the container. /proc is the container’s, bound rather than mounted fresh, because a fresh proc mount is refused by the container’s locked mounts; /sys is bound read-only. Runtime scopes live under system.slice and are named hd-<id>-<rev6>-<port|0>-<outerpid>.scope; the diagnostic endpoint renders the scope arguments with placeholders rather than identifying a running instance. TCP restriction uses Landlock, not systemd’s IPAddressDeny.

Ingress firewall

Section titled “Ingress firewall”

Ingress rules live in one nftables table, ip hoody_daemon, with one chain per program and policy revision. The table is built from the active and draining ingress records and is rebuilt as a single atomic transaction; it remains until both sets are empty and their removal is successfully applied, so it can outlive the last ingress policy until the drain finishes. An identical admission performs no nft update and does not reset meter budgets. A lazy-start can still regenerate the table when it finalizes a completed drain. Startup reconciles the recorded state, and a container with no prior state and no desired ingress policy skips firewall work. The table coexists with the iptables-nft tables that carry the localhost DNAT rule. Everything here is IPv4 TCP on eth0.

Rules match by destination port only, never by destination address, so they hold whether or not the DNAT rule is installed. Two consequences follow:

  • Port ownership is container-wide. A port in a program’s listening set is reserved across every local address. A create or edit whose declared ports overlap another program’s declared ports or a still-draining policy is rejected with 400, for sandboxed and unsandboxed programs alike, so an unsandboxed program cannot take a sandboxed one’s port either. Two unsandboxed programs keep exactly the pre-sandbox rule between themselves: overlapping port_range values are refused and ready_port is not checked. A port that a firewall record still reserves is refused to every applicant, sandboxed or not. Disabling or removing a program persists the change first and then attempts to retire its firewall record, and its ports stay reserved while any active or draining record for it exists. A retirement that fails does not fail the call: it is logged, the record and its port reservation stay, and the next daemon start reconciles it. Retiring a record moves it to draining; it does not release the ports. Only an existing program owns the records under its id, so an add that reuses a removed program’s explicit id is checked against those records like any other applicant. Once its firewall records are gone, a disabled program stops conflicting in the sandbox-specific ownership checks. The legacy port_range overlap check still includes disabled programs. Enabling a program re-checks port ownership, whether or not it carries a sandbox block, so a port taken while it was disabled refuses the enable.
  • An undeclared service on the same port is filtered too. Ports that an unsandboxed command opens without declaring them cannot be reserved, but traffic to such a service is still filtered when its destination port matches installed ingress rules.

Inside a program’s chain the order is: established and related connections return first, invalid packets drop, then the platform exception, then the rate meter, then the allowlist, then drop. A chain with only a rate limit has no terminal drop. Three things follow:

  • Only new connections are filtered, so the packet rules themselves never cut live ones. Editing the policy is a separate matter: it reloads the program definition and can stop the listener, which closes what was open.
  • With ingress_allow_platform at its default of true, 127.0.0.0/8 and the platform address return before the meter and the allowlist, so callers whose packets carry one of those two sources are neither filtered nor throttled. The exemption matches the packet’s source address, not the caller’s identity. Hoody Proxy forwards ordinary public traffic transparently, with the client’s own address as the source, so that traffic meets the allowlist and the meter like any other source. A hairpinned request, one a container sends to a service URL on its own server, is not exempt either: the proxy binds it to the address the client dialed, which for a public URL is the host’s public address rather than the gateway. The loopback half relies on the host and the container’s network interface filtering to drop packets that arrive on eth0 with a spoofed loopback source, because the daemon enables route_localnet on eth0 for its localhost redirect, which turns off the kernel’s own check for them.
  • The rate meter is evaluated before the allowlist. It matches SYN under the fin|syn|rst|ack mask, so ECN-capable SYNs are counted.

The meter counts connection attempts, not requests or bytes, and one source shares a single budget across every port and instance of that revision. It is a named dynamic set, hd_<id>_<rev12>_rate, with a 60 s timeout and 65535 entries. It has two documented behaviours. Budgets reset whenever the table is regenerated, which an admission, retirement, finalize or rollback of an ingress policy causes unless it changes nothing, and which every daemon start and reset causes. And when the set is full, new sources are untracked, so the rate limit fails open while the allowlist keeps applying.

Policy changes and draining

Section titled “Policy changes and draining”

When a policy changes, the old revision’s chain stays installed until every instance supervisord is running under it is confirmed stopped. A process that escaped supervisord’s tracking, a descendant an ordinary program forked and detached, is outside what any of this observes: supervisord signals the process it started, and the daemon reads the processes supervisord reports. Draining completes on its own after the old revision’s processes have stopped, any launch in progress has finished, and supervisord has applied the program’s latest configuration; poll GET /api/v1/daemon/programs/{id}/sandbox until firewall no longer reads draining. Rules and port reservations can outlive the condition that held them for a short while. If /sys/fs/cgroup/system.slice is missing and this daemon process has never read it successfully, every revision is treated as still alive. It also waits on a program whose earlier, unconfined definition the daemon could not establish as stopped, which is recorded in the firewall state and is independent of the scopes: that process was never wrapped, so no scope describes it. Disabling a program moves its current revision into draining without any policy change, so a drain does not always follow an edit. Chains are named hd_<id>_<rev12>, so the two revisions are separate objects rather than an edit of one. Concurrent revisions are traversed in program-id and revision order, so on ports they share a packet must pass both allowlists, and the old listening set stays reserved.

A successful add or edit returns once supervisord has applied the change; an apply that fails returns an error with the program already persisted. Its response body is unchanged and carries no firewall field. The drain continues in the background and is visible only through GET /api/v1/daemon/programs/{id}/sandbox, which reports firewall: "draining" until the old revision is gone, as long as nothing else is wrong. A damaged table, or an earlier unconfined definition that is not yet resolved, reports firewall: "degraded" instead, with the reason in problems, while the old revision remains. A removed program’s endpoint answers 404 from then on, so a record retained for a deleted program is visible only in the daemon’s retirement and finalization logs.

Container prerequisites and restart recovery

Section titled “Container prerequisites and restart recovery”

Every effective sandbox needs an executable /usr/bin/bwrap and /usr/bin/systemd-run, usable user namespaces, working systemd scopes, and an existing /run/user directory. mode: restricted additionally needs Landlock ABI 4 or higher. Ingress policies additionally need /usr/sbin/nft. Memory, task and CPU limits are enforced by systemd; the file-descriptor and file-size limits are in-process rlimits. udp: "deny" needs a readable /etc/resolv.conf under every mode except none. The daemon executable and the config.json path that go into the wrapper command must be absolute and may contain only ASCII letters and digits, /, _, . and -; a path that would need quoting is refused, which matters for a custom installation location. A program with a writable entry also needs a bubblewrap that supports --bind-fd, and a program with tmp_size one that supports --size; without it, such a program does not start. Provisioning installs bubblewrap and nftables with the daemon.

Beside programs.json, which describes the desired programs, the daemon keeps sandbox-firewall-state.json in its configuration directory as the firewall’s only durable input. It records a generation number, the fingerprint of the table as applied, the digest of the snapshot it was published with, the active and draining policy records, and any unresolved unconfined installations. sandbox-firewall.nft is a snapshot of the rules derived from it, carrying a # generation=N comment. New commits increment the recorded generation, and the marker described below carries the highest generation ever committed in that directory. A failed publication puts back the previous generation together with the content that describes it, and a recovery that lost both files continues above the marker’s number, so no committed generation number is ever handed out twice. A proposal whose publication failed and was put back was never committed, and its number can be used again. A restore refuses to run when the state and snapshot generations disagree.

The first firewall commit also writes sandbox-firewall.initialized beside those files. A state file that has gone missing while the snapshot survives is lost history rather than an empty firewall: the daemon refuses to regenerate, its startup reconciliation fails, and the programs that carry an ingress policy are skipped. When the state file and the snapshot are both missing while the marker survives, the daemon first asks whether anything could still be relying on the rules they described: a sandbox scope holding processes, a launch in progress, or an installed ingress table. If any of the three is present, or cannot be read, it refuses to regenerate for the same reason, and you recover by restoring the committed state and its matching snapshot. If none of them is present the container is verifiably idle, so the daemon logs the loss as a warning and publishes an EMPTY state, and a container that legitimately cleared its files recovers instead of failing every sandboxed create. That happens at daemon start, where the reconcile that follows re-admits every enabled program’s policy, and again inside the next mutation that needs the state, where only that mutation’s own program is re-admitted and the others come back at their next admission or at the next daemon start. Losing both files while the daemon is running therefore does not wedge it until a restart. Read-only paths never rebuild: health and the sandbox endpoint report the loss and write nothing. With both files absent and the marker present, the restore hook exits 11 without checking whether the container is idle, and the provisioning guard likewise treats that as unknown history. Only the daemon makes the idle check, and only the daemon can write the recovered empty state before a later supervisor start. Remove the marker by hand only when no sandboxed instance can still be running.

Provisioning installs /etc/systemd/system/supervisor.service.d/hoody-daemon-firewall.conf with ExecStartPre=/hoody/plugins/daemon/sandbox-firewall-restore.sh. The restore runs before every supervisor start, including one that happens while the daemon is down, and a failing restore blocks supervisord rather than letting listeners come up unfiltered. Because the rules match by port on eth0, they are effective before the daemon has installed its DNAT rule. When nft or the restore script is missing, provisioning asks whether rules are owed: a state file with records, a surviving snapshot, or an enabled program with an ingress policy keeps the guard, and so does a file it cannot parse. The drop-in is removed only on a verified “nothing owed”, so a container without nftables and without policies keeps booting. Keeping the drop-in is not by itself a guarantee that startup is blocked: the hook decides that, and a state file that holds no active or draining record with no snapshot beside it is accepted as “nothing to restore” before the hook looks at programs.json, even when an enabled program now declares an ingress policy.

Restore needs Python 3 and flock. It never runs at the same time as a daemon firewall change, so it cannot install an older generation over a newer one. Without flock, or when it cannot get exclusive access, it refuses with exit 12. A surviving snapshot is checked against the state file before it is applied. The state records name the chains the kernel must end up with, one per policy, and the snapshot has to define every one of them inside table ip hoody_daemon, with a body, and jumped to from a chain that actually carries an input hook, since a jump in a chain the kernel never calls filters nothing; a snapshot that declares chains and leaves them empty, puts them in another table, or never jumps to them is refused with exit 13. The requirement comes from the records, never from the snapshot’s own # chains= comment, so a truncated or hand-edited file cannot describe its own contents into acceptance. Before any of that, the digest decides. Each commit records the SHA-256 of the snapshot file in the state file it writes first, and the hook refuses a snapshot whose bytes do not hash to that value, and refuses a state that carries no digest at all. That applies to every snapshot, including one belonging to a state with no policies: an empty record list is not a reason to install a file nothing vouches for. Structure alone could not do this: a snapshot can keep every chain the records name, defined and reachable, while its terminal deny is gone, its destination port is different, or a second destroy table follows it. A generation mismatch blocks it too, and so does a snapshot with no state file beside it, since there are then no records to check it against. What survives those checks is validated with nft -c and applied. Those checks establish that the snapshot is the file the state vouches for and that it defines the chains the records name; the hook does not read the table back from the kernel afterwards, so a self-consistent state and snapshot pair that describes less than the desired policies is installed without complaint. The next launch does not catch that either: it checks that the live table matches the fingerprint recorded in the state file, not that the recorded policies are the ones you intended, so check a state and snapshot pair yourself before restoring it by hand. Recorded policies with no snapshot block startup. With neither state nor snapshot, a missing programs.json is a first boot and the restore succeeds, while a programs file that the hook finds and cannot read or evaluate, and an enabled program that declares an ingress policy with no snapshot to supply it, both block startup.

Both the guard and the hook start from the default config.json under /hoody/storage/hoody-daemon/config and follow its storage.config_path to the directory that actually holds the state file, the snapshot and the marker. A config.json that exists and cannot be parsed is “cannot tell”, so the guard keeps its drop-in. The guard distinguishes a missing file from a stat error throughout its inspection. The hook does so for the state file, the snapshot and the marker, but uses shell existence tests for config.json and programs.json. A programs.json that exists and cannot be read stops startup with exit 8, while a path whose directory cannot be searched reads as absent for either file. The hook also accepts a configuration path as its first argument or in HOODY_DAEMON_CONFIG; the guard reads the default location only, so a relocated config.json is outside what it inspects.

Every non-zero exit of the restore hook blocks supervisord:

ExitMeaning
0Nothing to restore, or the snapshot applied cleanly
2config.json cannot be evaluated, or the state file, the snapshot or the evidence files exist but cannot be read or evaluated
3Rules must be applied and nft is not installed
4The state file lists active or draining policies and the snapshot is gone
5The state and snapshot generations disagree, or the snapshot’s first line is not a # generation= header
6nft -c rejects the snapshot
7nft -c accepts the snapshot and applying it fails
8No state and no snapshot, and programs.json exists but cannot be read or evaluated
9No state and no snapshot, and an enabled program declares an ingress policy
10Python 3 is missing
11History lost: the marker exists and both files are gone, or a snapshot exists without its state file
12The state directory or the firewall lock cannot be used, including a missing flock
13The state records no snapshot digest, the digest cannot be computed or does not match, the required-chain inspection fails, or the snapshot does not enforce every chain the records require

When a program declares a sandbox the daemon cannot install, it tries to stop any unconfined process running under it. That applies at startup, when the policy is refused, its wrapped configuration cannot be published, or the firewall reconcile failed and its ingress rules cannot be shown installed. It also applies to an add, edit, enable or start whose apply failed. The daemon decides from what supervisord is actually running under the program’s current and earlier names, not from the configuration file, since a failed reload can leave a wrapped file in front of an unwrapped process. When the wrapped configuration is published but supervisord has not acknowledged it, the daemon stops every group the program may be running under and keeps the configuration, so the next apply installs the confinement. Otherwise it withdraws the configuration. In both cases it checks again afterwards, and the quarantine ends only on a measurement that finds nothing running or finds the wrapper; if the process cannot be shown to have stopped, the daemon stops it directly and logs an outcome it still cannot confirm as possibly unconfined. It also records that obligation in sandbox-firewall-state.json, and while it stands this program’s firewall records are kept instead of drained. The retention depends on that write: a record the daemon cannot write is logged as one it could not write, and the next drain can then release the rules of a listener that may still be up. A drain is finalized on the absence of processes in the policy’s scopes, and a process that was never wrapped has no scope, so without that record a later disable would release the rules of a listener that is still up. The same record is written when a supervisord configuration cannot be removed, because an unconfirmed removal says nothing about what is still loaded. It is recorded only while the program still owns an active or draining record, since there are otherwise no rules to retain, and it lists the supervisord group names that have to be measured. Two cases record nothing: a program whose first apply never reached supervisord, because nothing was ever acknowledged for it and no definition of it can be loaded, and a program that owns no record at all. Two events clear it without a measurement: a removal of the program’s configuration that supervisord confirmed by reloading, and an enabled program that no longer declares confinement and whose own ordinary configuration supervisord has acknowledged. Otherwise it takes a measurement. An apply whose publication supervisord acknowledged and a daemon start each trigger one; neither is evidence on its own, because each rests on a record that can be lost. A measurement clears the obligation when the recorded group list is complete, every group in it was measured, no unwrapped configuration of the program remains for supervisord to revive, and supervisord reports either nothing running or only this program’s own wrapper. A quarantine that measures the program itself and finds nothing running, or finds its own wrapper, clears it directly, so a daemon start can resolve a hold while the wrapped program keeps running. Until then the program’s listening set stays reserved and GET /api/v1/daemon/programs/{id}/sandbox carries the reason in problems. A running wrapper is left alone. A program with nothing running is left alone too, unless the definition supervisord still holds for it is unwrapped: that definition is withdrawn, because supervisord would start it unconfined. If supervisord cannot be asked, the daemon leaves a program alone only when every configuration that may still be loaded for it was wrapped. A program renamed by the failing edit is stopped under its old name too. A name another program has taken since is left alone only once supervisord has applied that program’s own configuration. Every daemon start republishes every enabled program with an effective block, so a program whose configuration was withdrawn is re-evaluated at the next start. This matters most for the firewall case: the daemon’s unit starts after supervisord, so a program already in supervisord’s configuration is already running by the time the daemon looks, and declining to apply a new configuration would leave it listening under rules nobody could establish. That is deliberate: the alternative is a process running with none of the confinement its stored policy declares. The program row is untouched, so repairing the cause and enabling or editing the program brings it back. Programs without a sandbox are never quarantined; see errors and status codes for how their failed apply is reported.

Adding an ingress policy to a program that is already running unconfined stores the policy and its firewall rules before the sandboxed configuration reaches supervisord. Until supervisord has taken that configuration, the old process may still be listening. If the daemon restarts or a reset runs in that window, the daemon keeps the rules in place for as long as the old configuration is still on disk, its replacement has not been confirmed, or the configuration cannot be read. Once the sandboxed configuration is confirmed or the old one is withdrawn, the rules drain like any others and their ports are free for other programs again. A program that could not start at boot because such rules still held its port is retried while the boot replay still finds programs that can now start. If the rules clear only after the replay has finished, the program stays stopped: start it, or edit or re-enable it, once they have drained.

At startup the daemon also republishes the supervisord configuration of every enabled program that carries an effective block, including one with boot: false and no lazy loading. Adding confinement to a program that is already running is otherwise durable only once supervisord has acknowledged its wrapped configuration, and a daemon that stopped between persisting the policy and publishing that configuration would leave the old unwrapped configuration and its process in service for the life of the container. When the bytes already match, the configuration is not rewritten, and the reload is skipped unless an earlier publication of that program is still unacknowledged; when they do not match, the reload replaces the unconfined process.

A configuration change that supervisord has not confirmed is applied again by a supervisord reload at the next daemon start. If the daemon cannot record a change as pending, the apply fails before the configuration is touched.

Supervisord’s own inherited environment is a provisioning invariant of its systemd unit. The daemon filters unsecure names from program environments and from the container environment it injects, but it does not sanitize what supervisord itself inherits.

Examples

Section titled “Examples”

Each example below is an add or edit request body, not a programs.json file. Before running one:

  • the named non-root account, working directory, application files and writable directories must already exist, with suitable ownership and permissions;
  • a program with a port_range receives its port as --port=<port> (or <port_param>=<port> when port_param is set), appended to the command unless it already names a port, so the command must accept that equals form;
  • the ports must be free and outside the reserved Kit ports;
  • replace PROJECT_ID, CONTAINER_ID, SERVER, PROGRAM_ID and the example network 203.0.113.0/24 with your own, and note that the HTTP and URL tabs still go through the proxy, so its permissions apply.

The add calls omit boot, whose default is false, so they register the program without starting it; start it as shown on the Daemons page (a new program is enabled by default, so enabling is only needed for one you disabled).

Read-only root with a writable data directory

Section titled “Read-only root with a writable data directory”

The program sees a read-only root, with its data directory bound writable and /root hidden.

Terminal window
# Register web-app with a read-only root and one writable data directory
hoody daemon programs create -c "$CONTAINER_ID" \
  --name web-app \
  --command "/usr/bin/node /hoody/storage/apps/web-app/server.js" \
  --user web \
  --directory /hoody/storage/apps/web-app \
  --port-range-start 8080 --port-range-end 8080 \
  --sandbox-filesystem-read-only-root \
  --sandbox-filesystem-writable /hoody/storage/apps/web-app/data \
  --sandbox-filesystem-hidden /root

This restricts writes and hides /root; like every sandbox, it also hides /run/user. It adds no network policy or resource limits and hides nothing else the account can read, and /proc and /dev keep the exceptions described above.

A restricted-network worker

Section titled “A restricted-network worker”

The worker has no port_range, so bind_ports is required under mode: restricted. It may make TCP connections to destination ports 5432 and 53 on any host; UDP, ICMP and raw sockets are outside this Landlock policy. It is capped at 256 MiB and 64 tasks.

Terminal window
# Register a worker limited to TCP destination ports 5432 and 53
hoody daemon programs create -c "$CONTAINER_ID" \
  --name queue-worker \
  --command "/usr/bin/python3 /hoody/storage/apps/worker/main.py" \
  --user worker \
  --directory /hoody/storage/apps/worker \
  --sandbox-network-mode restricted \
  --sandbox-network-bind-ports 9100 \
  --sandbox-network-connect-ports 5432,53 \
  --sandbox-process-max-memory 256M \
  --sandbox-process-max-pids 64

This allows TCP binds on 9100 and TCP connections to ports 5432 and 53 on any host, and caps memory and tasks. UDP, other non-TCP protocols and Unix sockets reachable by path are outside the network policy, although the mode refuses MPTCP, io_uring and TCP Fast Open sends, which could reach a TCP port around it; the account keeps its ordinary filesystem access, and no ingress rules are installed.

Allowlist and rate limit on a public service

Section titled “Allowlist and rate limit on a public service”

The edit targets an existing non-root program without terminal_id whose declared ports are compatible with the listening set. It replaces the sandbox block as a unit: new connections to ports 8000 to 8003 are accepted from one network, metered at 50 SYN packets per second per source with a burst of 50. With ingress_allow_platform: true, loopback sources and the gateway address bypass both checks. Public requests forwarded by Hoody Proxy carry the client’s address, so they do not.

Terminal window
# Refused, listing each setting, if the stored block has settings this edit drops.
# Add them to the command, or add --sandbox-replace to drop them.
hoody daemon programs update "$PROGRAM_ID" -c "$CONTAINER_ID" \
  --sandbox-network-bind-ports 8000-8003 \
  --sandbox-network-ingress-allow-from 203.0.113.0/24 \
  --sandbox-network-ingress-allow-platform \
  --sandbox-network-ingress-rate-limit 50/s

Because this edit replaces the whole block, it also removes any filesystem and process restrictions the program had and puts its network mode back to full. It filters new IPv4 TCP connections to ports 8000 to 8003 arriving on eth0, with the platform bypass described above. It does not restrict outbound connections or cover other ports, and the meter can reset or fill. The rules do not cut established connections, though applying this edit reloads the program and can stop its listener.

A program with no IP networking

Section titled “A program with no IP networking”

mode: none puts the program in an empty network namespace. Declare no ports with it: port_range, ready_port and any bind_ports list, an empty one included, are refused.

{
  "name": "offline-worker",
  "command": "/usr/bin/python3 /hoody/storage/apps/worker/main.py",
  "user": "worker",
  "directory": "/hoody/storage/apps/worker",
  "sandbox": {
    "filesystem": { "read_only_root": true },
    "network": { "mode": "none" },
    "process": { "private_tmp": true }
  }
}

Unix sockets it can open by path and files its account can read stay reachable, except under the directories the sandbox hides: private_tmp gives it empty /tmp and /var/tmp, and every sandbox hides /run/user.

A batch job with CPU, file and UDP limits

Section titled “A batch job with CPU, file and UDP limits”

The job may use half of one CPU and 1 GiB of memory, which includes whatever it keeps in its private /tmp and /var/tmp of 256 MiB each. It cannot write a file larger than 2 GiB, and it cannot create UDP sockets, so its name lookups go over TCP.

{
  "name": "batch-job",
  "command": "/usr/bin/python3 /hoody/storage/apps/batch/run.py",
  "user": "worker",
  "directory": "/hoody/storage/apps/batch",
  "sandbox": {
    "network": { "udp": "deny" },
    "process": { "max_cpu": "50%", "max_memory": "1G", "max_file_size": "2G", "private_tmp": true, "tmp_size": "256M" }
  }
}

The network mode stays full, so TCP connections are not restricted. Under mode: restricted, list TCP port 53 in connect_ports as well, or name resolution fails.

Remove a sandbox

Section titled “Remove a sandbox”
Terminal window
# Clear the block, then read back what the daemon still holds
hoody daemon programs update "$PROGRAM_ID" -c "$CONTAINER_ID" --sandbox null
hoody daemon programs sandbox get "$PROGRAM_ID" -c "$CONTAINER_ID"

Clearing the block replaces the loaded definition, so the instance running under the old policy stops and boot and lazy_load decide whether one starts again. The old revision’s ingress rules stay installed until every instance under them is confirmed stopped, so configured: null is not proof that the rules are gone: watch firewall until it reads none.

Reinstall rules that something else removed

Section titled “Reinstall rules that something else removed”

If a tool inside the container flushed the ip hoody_daemon table, programs that are already running keep listening unfiltered. A disable followed by an enable retires and re-admits the policy, which rebuilds the table; it interrupts the program.

Terminal window
# Interrupts the program. Check each response before the next command.
hoody daemon programs disable "$PROGRAM_ID" -c "$CONTAINER_ID"
hoody daemon programs enable "$PROGRAM_ID" -c "$CONTAINER_ID"
hoody daemon programs start "$PROGRAM_ID" -c "$CONTAINER_ID" --port 8080 --if-not-running
hoody daemon programs sandbox get "$PROGRAM_ID" -c "$CONTAINER_ID"

Expect firewall: "ok" once any drain finishes, and read problems if it says degraded. Rebuilding the table also rewrites the snapshot from the state records, so while the state file is intact a missing snapshot is replaced, and so is a damaged one that can still be read as text. A corrupt state file, or a missing state file beside a surviving snapshot, is not repaired: the daemon refuses to regenerate it. If both files are missing and the marker remains, the daemon rebuilds an empty state only when no sandbox scope, launch in progress or ingress table remains. For the cases it refuses, and when the boot restore hook blocks a restart, restore the committed pair from backup and run the restore hook inside the container, as described under container prerequisites and restart recovery.

To remove a sandbox, send "sandbox": null on the edit endpoint, sandbox: null from the SDK, or --sandbox null from the CLI.

Inspect a sandbox

Section titled “Inspect a sandbox”

GET /api/v1/daemon/programs/{id}/sandbox reports what the daemon would enforce beside what the kernel is holding. It is diagnostic only; nothing in it is settable. An existing program always answers 200, and an unknown id is a 404. A program with no effective sandbox, or one whose stored policy no longer validates, answers with rev: null and effective: null.

FieldMeaning
configuredThe stored sandbox block, exactly as persisted, or null when the program has none. A no-op block appears here only when it was loaded from a hand-edited file.
revPolicy revision: the first 12 hex digits of the SHA-256 of the canonical spec. It is the suffix of the program’s chain name, and its scope names carry the first six digits. Null when the program has no effective sandbox.
effective.bwrap_argvThe bubblewrap command line built from the stored policy: namespace flags, the root bind, the writable and hidden mounts, and the working directory. It is a diagnostic template, not a runnable command or a reading of a running process: <per-launch-fd> stands for each writable mount’s descriptor and <program argv> for the command. Landlock and RLIMIT_NOFILE are applied inside the wrapper, so they do not appear.
effective.scope_argvThe systemd-run --scope command line carrying the process limits, with <port> and <pid> placeholders in the scope unit name.
live.table_presentWhether the ip hoody_daemon table was read successfully. false covers both an absent table and a failed query.
live.chainThe expected chain name for the current revision, hd_<id>_<rev12>, when table inspection succeeds; null when the policy owes no ingress rules or the inspection failed. It does not by itself prove the chain exists.
live.rulesAn array of nftables JSON rule objects for the current policy chain, in evaluation order. Empty when the chain does not exist or the inspection failed.
problemsStrings for each diagnostic collected: the stored policy no longer validates, the firewall state file cannot be read, an enabled program’s ingress policy has neither an active record matching its current revision nor any draining record for this program (a disabled program owes no rules, so it reports no problem for that), an active record belongs to a revision other than the current one, the live table does not match the committed fingerprint, or an earlier definition of this program may still be running without the confinement its policy declares, which names the groups that have to be measured before its records can drain and appears only while the program still owns a record. Empty when no diagnostic was collected.
firewallOne of ok, degraded, draining or none.

The four firewall values:

  • ok: this program’s current revision has a matching active ingress record, no drain is in progress, and the live table matches the fingerprint recorded at the last successful apply.
  • degraded: problems is non-empty, which includes a live table that is missing, unreadable, or different from the committed fingerprint. The fingerprint excludes handles, metadata and named-set elements, so rate-meter churn never causes this; static allowlist expressions are included. degraded takes precedence over draining.
  • draining: a draining record exists for this program, its current revision after a disable or an older revision after an edit, because instances running under it have not been confirmed stopped. This is reported even after the block was cleared.
  • none: verified that no active or draining record exists for this program under any revision and that nothing about it owes rules. A disabled program with an ingress policy and no remaining record reads none; its configured, rev and effective are still reported. An active record left behind by another revision, which is what a retirement that did not complete looks like, is reported as a problem rather than passed over. Another program may still own a live table.

Read firewall and problems together: problems names the reason whenever firewall is degraded.

Terminal window
# Read the stored policy, its argument vectors and the live firewall state
hoody daemon programs sandbox get "$PROGRAM_ID" -c "$CONTAINER_ID"

Response for an unsandboxed program in a container with no prior firewall history, no records and no table:

{
  "success": true,
  "configured": null,
  "rev": null,
  "effective": null,
  "live": {
    "table_present": false,
    "chain": null,
    "rules": []
  },
  "problems": [],
  "firewall": "none"
}

Health

Section titled “Health”

GET /api/v1/daemon/health carries a sandbox object that probes the mechanisms this host has. A sandboxed program fails closed when a mechanism its selected policy requires is missing. Capability probes are cached after their first evaluation, firewall integrity is queried separately, and sandbox problems do not change the endpoint’s HTTP 200 or its top-level status: "ok".

FieldTypeMeaning
bwrapbooleanWhether /usr/bin/bwrap is present and executable.
nftbooleanWhether /usr/sbin/nft is present and executable. Without it no ingress policy can be applied.
systemd_runbooleanWhether /usr/bin/systemd-run, which owns the memory and task limits, is present and executable.
landlock_abiintegerLandlock ABI level reported by the kernel. mode: restricted needs 4 or higher. 0 when Landlock is unavailable.
firewallstringContainer-wide state of the ingress table: ok when the live table matches the recorded fingerprint, degraded when it does not, when the table cannot be read while a fingerprint is recorded, or when the firewall state file cannot be read, none when the committed state holds no active and no draining record. It never says draining; per-program drains are visible only on the sandbox endpoint.

This value is computed from committed records alone. It never validates the desired policies in programs.json, and it reads ok while records are still draining after the last ingress policy was removed. It also returns none when its first existence checks find no state file, no snapshot and no marker, so an error those checks conceal is not diagnosed here.

Errors and status codes

Section titled “Errors and status codes”
  • 400 with the JSON envelope is every controller policy rejection: user: root, terminal_id, mode: restricted without a resolvable listening set, a writable or hidden path overlapping a protected root, an unsecure environment name, a port already owned in either direction, and an explicit program id of 2147483647, since the accepted range stops one below it. These carry { "success": false, "error": "..." }. Validating a sandbox object that restricts something reads the firewall state, and an unreadable state there returns 500 with INTERNAL: . A block that normalizes away restricts nothing and is treated as an ordinary program, so on a program with no effective sandbox it does not enter that read and an unreadable state cannot make it fail. An edit validates the merged program, so omitting sandbox does not bypass the checks for a block that is already stored. An ordinary unsandboxed request can also read the firewall state for its port-ownership check, but an unreadable state on that path is logged and tolerated. Enabling or replacing an effective block can fail on the firewall state or on admission. A sandbox: null value skips the validation read, but an edit that clears or replaces a program’s existing effective sandbox still reads the firewall state first: if that state cannot be read, the edit returns 500 with INTERNAL: and the stored sandbox is unchanged. An earlier validation failure can return before it.
  • 400 with invalid program body: ... is a body the add or edit handler cannot parse: malformed JSON, a field of the wrong type, or an unknown key inside sandbox. It carries the same JSON envelope. A repeated top-level key answers duplicate field "<key>": ..., and a near-miss of sandbox such as sandbx or Sandbox is refused when the exact sandbox key is absent; other unknown top-level keys are ignored.
  • Plain-text 413 is a body over the 1 MiB limit, refused before the handler runs.
  • 400 with the program persisted is a create, edit or enable of a program that has or had an effective block whose supervisord apply failed, for example a reload timeout or a validation refusal. The response says so, the firewall rules are kept, and the next apply retries the reload. A sandboxed start whose apply fails refuses to start the program unconfined. A program without an effective block also answers 400 with the program saved (saved but NOT applied) when supervisord refuses its configuration, cannot confirm it can load it, or the configuration cannot be written; other apply failures are logged.
  • 500 is a failed write of programs.json, with an error string starting PERSIST: , for every program, sandboxed or not; the change is not kept. It is also, for a request that carries or replaces an effective block, any firewall-state read failure, nft failure, or applied table that could not be read back, starting INTERNAL: . Not every endpoint can produce both halves, and the ones that cannot do not document them: add, edit and enable can produce either; start admits a policy but persists nothing, so only INTERNAL: ; disable, remove and reset persist but make no admission of their own, so only PERSIST: , and a retirement that fails is logged rather than returned; stop neither persists nor enters a firewall transaction, so it has no 500 at all. If an applied table cannot be read back, the previous rules are put back and nothing changes. For PERSIST, the firewall admission made for that request is reverted for that program alone; if the revert itself fails, the rules stay in place and the failure is logged. If a firewall change is applied and then cannot be saved, the daemon puts the previous rules and both state files back. If that restoration also fails, or if an apply times out and its retry fails too, the installed rules no longer match the saved state. The daemon then refuses to start programs that need ingress rules. A program that owns no rules, including one with a filesystem-only or process-only policy, is unaffected. The refusal lifts by itself as soon as any firewall change commits cleanly; repair the committed state, its matching snapshot and the kernel rules, then make a change that commits. The refusal lives in the running daemon and gates admission and boot application only. It does not stop processes that are already running, and neither health nor the sandbox endpoint reports it, so an ok there is a statement about records and table integrity rather than proof that the refusal has lifted.
  • Retirement failures never block cleanup. On remove, disable, or a disabling edit of a program with an effective block, a failure to retire its firewall record is logged and does not stop the daemon from attempting the supervisord cleanup. The record stays, with its rules and port reservation, until a later successful daemon start reconciles it. The supervisord cleanup itself is reported. Remove, disable and a disabling edit all answer with the JSON error envelope, not with success, when the configuration could not be removed or the reload failed after the change was persisted; the file may already be gone, so read the daemon log for which of the two it was. At startup the daemon sweeps configurations belonging to program ids that no longer exist. A disabled program keeps its id, so its configuration is not swept: fix the cause and call disable again. That covers the retirement step of remove, disable and a disabling edit; it is not an account of every firewall-state access. A disabling edit can also fail earlier, in the validation of the merged policy or in the firewall checkpoint.
  • Reset replaces the program definitions with the configured defaults. A delayed apply from an earlier reset generation is withdrawn before publication, so it cannot republish a stale configuration afterwards. Firewall reconciliation errors are logged, and a reset can return success while the programs that carry an ingress policy are skipped at boot.

Known limitations

Section titled “Known limitations”
  • The launch compares each writable and hidden path by device and inode, and passes each writable directory to bubblewrap as an open descriptor, so the directory mounted writable cannot be swapped. bubblewrap still resolves the destination of every mount by pathname, for writable binds and hidden mounts alike. A tenant who can replace a validated directory entry, or any mutable directory above it, can therefore move where a hidden mount lands, so the content it was meant to hide stays visible under the name it was moved to, or move where the pinned writable directory appears. A hidden mount only covers what is at its destination, so neither case reaches anything the account could not otherwise reach, but the first defeats a restriction the sandbox was asked to add. Nothing rechecks identity while the program runs.
  • The provisioning guard reads the default config.json only, and the restore hook needs Python 3 and flock. See container prerequisites and restart recovery.
  • /proc is the container’s, so processes outside the sandbox are listed. See what changes for a sandboxed program.
  • Landlock is TCP-only, and it covers bind and connect, not a socket that starts listening without binding. Ingress rules are per destination port on eth0, IPv4 only. See ingress firewall.
  • udp: "deny" refuses every socket() call made through the legacy 32-bit socketcall interface (a socketpair() is still allowed), and moves DNS to TCP only for the standard glibc resolver. See network fields.
  • There is no disk-space quota for writable paths and no limit on disk bandwidth. max_file_size caps one file at a time, and max_cpu applies per instance of a port-range program.
  • mode: restricted has no MPTCP, no SMC, no io_uring and no TCP Fast Open send. A program that requires io_uring does not run under restricted. A 32-bit x86 program that uses the legacy socketcall interface cannot create any socket through that interface there (Unix sockets included) or send through it; socketpair through it, and the direct 32-bit socket calls, still work. A socket or a ring created outside the sandbox and passed in over a Unix socket is not covered. A program that was already running under restricted gets this at its next start.
  • A sandboxed program with a display can lose sound, because the display’s audio socket normally lives under /run/user.
  • Hiding /run/user covers the standard location of the user manager and the session bus. One that is reachable through another mount path, a runtime directory configured elsewhere, or a session bus on an abstract socket (reachable under full and restricted) is not covered.
  • Ownership rules are enforced by the API and by the boot path, which both see every program. The renderer and the launch wrapper validate the policy’s own shape only, so neither re-checks ownership against neighbours it cannot see.
  • A signal death reaches supervisord as exit code 128+n, and a policy change replaces the loaded definition, stopping the instance that ran under the old one. See what changes for a sandboxed program.
  • Rate-limit budgets reset on every table regeneration, and the meter fails open when its set is full. See ingress firewall.
  • A firewall drain can outlive the API call that started it. See policy changes and draining.
  • A read-only root is not a confidentiality boundary. Every file the program’s account can read stays readable unless a hidden entry covers it, and protected trees such as /etc, /run and /hoody cannot be hidden. A program with inject_container_env: true also receives the container’s environment, from which only specific unsafe names are filtered, never your application’s own secrets; see container environment. Run untrusted programs without ambient credentials, and give their account only the file permissions they need.
  • Quarantine ownership is best effort. When two programs have both used one group name, the program being quarantined keeps the name and can stop the other program’s group. The ownership check and the stop are separate steps, so a name can change hands between them. The fallback reads configuration files and markers, which do not certify what supervisord has actually loaded. See container prerequisites and restart recovery.
  • For an effective sandbox, user: root, terminal_id, capability grants, and private_tmp together with a writable path that meets /tmp or /var/tmp, a working directory below either, or a daemon installed below either, are all rejected. Quick-start rejects any non-null sandbox value. See the sandbox block for the exact overlap rule.

Use Cases

Section titled “Use Cases”

Application servers

Section titled “Application servers”

Give a Node.js, Python or Go server a read-only root with one writable data directory, hide /root, and declare its port in bind_ports so the port is reserved. Reserving a port does not itself filter traffic; add ingress_allow_from or ingress_rate_limit for that.

Background workers

Section titled “Background workers”

Run a queue processor under mode: restricted with connect_ports limited to selected TCP destination ports, and cap memory and tasks to reduce the worker’s impact on other programs.

Public services

Section titled “Public services”

Front an internet-facing service with ingress_allow_from and ingress_rate_limit. With ingress_allow_platform: true, loopback sources and the gateway address bypass both. A public request through the proxy keeps the client’s address, so the allowlist has to include the networks you expect clients to come from. For when to filter here rather than at the proxy or the container firewall, see allow only certain client IPs.

Best Practices

Section titled “Best Practices”

Start with the filesystem

Section titled “Start with the filesystem”

read_only_root with one writable directory is the cheapest policy to reason about. Create the writable directory before the call; the daemon refuses a path that does not exist.

Declare every port

Section titled “Declare every port”

Under mode: restricted, list every port the program binds for TCP. A port outside the listening set cannot be bound for TCP: that set is bind_ports, or port_range and ready_port when bind_ports is omitted. Undeclared ports cannot be reserved from the command itself, but traffic to an undeclared service is still filtered when its destination port matches installed ingress rules.

Resend the whole block on edit

Section titled “Resend the whole block on edit”

An edit replaces the sandbox as a unit, so read the program first, change the fields you need, and send the complete block back. See normalization and edits.

Watch the drain after a change

Section titled “Watch the drain after a change”

Poll GET /api/v1/daemon/programs/{id}/sandbox until the drain ends. Expect ok when the current policy has an active ingress record, or none when this program has no remaining ingress record. Investigate degraded, its problems entries, or an HTTP error.

Useful Questions

Section titled “Useful Questions”

Q: Does adding a sandbox change anything for my other programs? Other programs keep their existing rendering. Sandbox port reservations can reject conflicting changes, and ingress rules also affect other listeners on the same destination port.

Q: Why did my program restart after an edit? The resolved policy changed, so its revision and rendered configuration changed. See what changes for a sandboxed program.

Q: Why does my program show exit code 137? A signal death is reported as 128+n, so a SIGKILL and an OOM kill are both 137. See what changes for a sandboxed program.

Q: I sent a sandbox block but the program does not show one. The block restricted nothing, so it was stored as absent. See normalization and edits.

Q: Can I sandbox a program that runs as root? No. See what changes for a sandboxed program.

Q: Does the rate limit apply to traffic through Hoody Proxy? Yes, for ordinary public requests: the proxy forwards them with the client’s own source address, which the platform exemption does not cover. It applies to hairpinned requests too, since the proxy binds those to the address the client dialed rather than to the gateway. The exemption covers only traffic whose source is loopback or the gateway. See ingress firewall.

Troubleshooting

Section titled “Troubleshooting”

400 writable without read_only_root

Section titled “400 writable without read_only_root”

Cause: filesystem.writable was set on its own. Solution: Set read_only_root: true, or drop writable.

400 on a path under /hoody

Section titled “400 on a path under /hoody”

Cause: /hoody and /hoody/storage contain the daemon’s own tree, so they are protected roots, and overlap is checked in both directions. Solution: For writable, use a subdirectory such as /hoody/storage/apps/<name>. A hidden path may not overlap the /hoody tree at all.

400 mode: restricted without a listening set

Section titled “400 mode: restricted without a listening set”

Cause: The program has neither port_range nor ready_port, and bind_ports is empty or absent. Solution: Add bind_ports. Under restricted the program cannot bind any TCP port that is not listed.

400 on a port another program declares

Section titled “400 on a port another program declares”

Cause: The port is in an enabled program’s port_range, ready_port or bind_ports where a sandbox is involved on either side, or a firewall record, active or draining, still exists for a program that declared it, including a disabled or removed one. The legacy port_range overlap check counts disabled programs too. Solution: For a declaration conflict, change or remove the declaration, or choose another port. Waiting helps only with a record conflict: GET /api/v1/daemon/programs/{id}/sandbox on the other program shows the drain, and a removed program answers 404.

400 on an environment variable name

Section titled “400 on an environment variable name”

Cause: The name is one of the glibc-unsecure names, such as LD_PRELOAD or TMPDIR, and the program has an effective sandbox. Solution: Remove the variable.

Name resolution fails with udp: "deny"

Section titled “Name resolution fails with udp: "deny"”

Cause: Under mode: restricted, TCP port 53 is not in connect_ports, or the program resolves names with its own resolver or a C library other than glibc, which keeps using UDP. Solution: Add 53 to connect_ports, or configure the program’s resolver for DNS over TCP. If neither is possible, keep udp at allow.

Program never starts and health shows a missing mechanism

Section titled “Program never starts and health shows a missing mechanism”

Cause: A sandboxed program fails closed when a mechanism its policy requires is missing: bwrap or systemd_run is false, nft is false with an ingress policy, or landlock_abi is below 4 with mode: restricted. Solution: Read the sandbox object on GET /api/v1/daemon/health and install the missing mechanism. The wrapper’s own sandbox: line in the program’s error log names the refusal.

firewall reports degraded

Section titled “firewall reports degraded”

Cause: problems names it: the live table is missing, unreadable, or differs from the committed fingerprint; the stored policy no longer validates; the state file cannot be read; an enabled program’s ingress policy holds no record; or an earlier definition of the program may still be running unconfined, which keeps its records and its port reservation. Solution: Inspect problems and the daemon’s nft and state errors. Once those are resolved, an ingress-policy change that commits a new table can restore ok; an identical edit or an unrelated sandbox change may do nothing. For the unconfined-definition entry, repair what stopped supervisord from applying or removing the configuration and then apply or remove it again, or restart the daemon, which measures the recorded group names. That measurement clears the record when the recorded group list is complete and fully measured, no unwrapped configuration remains for supervisord to revive, and supervisord reports nothing running or only this program’s own wrapper. A confirmed removal of the configuration, or removing the confinement from an enabled program whose ordinary configuration supervisord then acknowledges, clears it without a measurement. If the daemon is also refusing to start programs that need ingress rules, see errors and status codes for what lifts that refusal.

firewall stays draining

Section titled “firewall stays draining”

Cause: Processes from the previous revision remain, or the daemon cannot confirm their absence. Solution: Stop the old processes and resolve any scope-observation errors. Finalization requires no processes in the old revision’s scopes, no launch in progress, no supervisord change of the program still being applied, and no unconfined hold; see policy changes and draining and the hold conditions under container prerequisites and restart recovery.

What’s Next

Section titled “What’s Next”
Daemons Create, start, and monitor daemon programs
Program Management Add, edit, and inspect programs and their sandbox state
Firewall Container-wide ingress and egress rules on the host kernel