Skip to content
Hoody Documentation
Hoody.com
Download Page (.md)

Daemons

Section titled “Daemons”

Hoody Daemons is a process manager you drive over HTTP. It supervises any executable: Node.js, Python, Go, Rust, compiled binaries, and shell scripts. Run programs in the foreground: a process that detaches from the one the daemon started is outside its status, and for a program without a sandbox it is also outside stop cleanup and survives a stop. There is no CLI to install and no language-specific tooling, and a REST API creates programs, starts and stops them, and reports their state.

Capabilities

Section titled “Capabilities”
  • Program management - Create, configure, and delete daemon programs
  • Process control - Start, stop, enable, and disable programs
  • Status monitoring - Read process state, uptime, and PID
  • Auto-restart - Choose whether a program restarts always, never, or only after an unexpected exit
  • One-off commands - A command that finishes runs once, and its exit code is reported as last_run
  • Priority control - Order programs within a supervisord update (not a dependency guarantee)
  • Logging - Point stdout and stderr at log files
  • User isolation - Run each process as a specific system user
  • Environment - Set custom environment variables per program, and choose whether it also receives the container’s environment
  • Multiple instances - Run one program once per port with port_range, instead of copying the configuration
  • On-demand start - lazy_load leaves a program stopped until something starts it, and quick-start programs run without being added to the persistent program list
  • Webhooks - Have a webhook fire on program lifecycle events

API endpoints summary

Section titled “API endpoints summary”

All endpoints are relative to your Daemon Manager service URL:

https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com

Program management:

Process control:

Monitoring:

Logs:

Quick start (ephemeral programs):

Before you start

Section titled “Before you start”

These examples run against an existing container: substitute its project id, container id and server name. The daemon supervises programs, it does not install them, so the runtime, the application, the system account and any directories must already exist in that container, with suitable permissions.

The CLI examples need the Hoody CLI configured for that container, the SDK examples need hoody-sdk and the client setup shown below, and the HTTP examples that pipe through jq need it locally. Every call goes through Hoody Proxy, so its permissions apply.

Process lifecycle

Section titled “Process lifecycle”

Enable → Start → Stop → Disable → Remove

New programs are enabled by default, so the enable call matters only for a program created or later set disabled. Disable stops the program and removes its supervisord definition while keeping the stored program. A stop that answers success is not proof the process ended, because the daemon does not check supervisord’s answer: read the status afterwards. Remove deletes the program. Reset replaces the entire program list with programs.default.json, removes the old supervisord definitions, attempts a reload and applies the defaults. It can report success even when that reload fails, so check the daemon log and the running processes afterwards, and save the current list first if you need to recreate anything. A missing, unreadable or malformed defaults file refuses the reset without replacing the stored programs or removing their supervisord definitions; a start that was being prepared at that moment can still be refused, so send it again. A mutation whose supervisord step failed can answer with an error after the change was saved: read the error, which says so, and the current program before retrying. The reverse also happens: for an ordinary program without a sandbox, create, enable and edit can answer success even though the supervisord definition was not applied, so what is stored and what is running can differ. Check the status and the daemon log after a change. A start that is still being prepared when an edit, removal or reset changes the same program is refused with an error saying nothing was started; send the start again once the change has finished. A change that arrives after the start was issued does not refuse it: the start goes ahead, and the change then applies to the process it started.

Terminal window
# Create a daemon program (required fields are passed as flags)
hoody daemon programs create --name web-server \
  --command '/usr/bin/node /app/server.js' --user www-data -c my-container-id


# Enable and start (use the numeric Program ID returned by create/list)
hoody daemon programs enable <program-id> -c my-container-id
hoody daemon programs start <program-id> -c my-container-id


# Check status
hoody daemon programs status <program-id> -c my-container-id


# Stop and remove
hoody daemon programs stop <program-id> -c my-container-id
hoody daemon programs disable <program-id> -c my-container-id
hoody daemon programs delete <program-id> -c my-container-id --yes

Add a program disabled

Section titled “Add a program disabled”

enabled defaults to true, which makes the program eligible to run. It does not by itself register the program with supervisord: on create the daemon publishes a supervisord configuration only for a program that carries a port_range, boot: true, lazy_load: true, or an effective sandbox block, and a plain enabled program is published the first time something starts it. An enabled boot: true program is therefore applied by the create call itself and can start at once; any other program waits for an explicit start. Pass enabled: false to add the program unregistered, then enable and start it when ready:

Terminal window
hoody daemon programs create --name web-server \
  --command '/usr/bin/node /app/server.js' --user www-data \
  --no-enabled --autorestart true --directory /app -c my-container-id

Then enable, start, and monitor it as shown in the lifecycle example above.

Enable vs start

Section titled “Enable vs start”

Enable is a configuration change:

  • Makes the program available to supervisord
  • Does not start the process by itself, though enabling a program the daemon applies (a port range, boot enabled, lazy loading, or an effective sandbox) writes its configuration and reloads Supervisor, and a boot-enabled program can start at once (check status to confirm it is running)
  • Needed before starting only when the program is disabled

Start is a runtime action:

  • Launches the process
  • Only works if the program is enabled: a start on a disabled program fails with Program with ID <id> is disabled
  • Returns {"success": true} for a standard program without waiting for it to be ready
  • Starts a new run of a program that has finished, even one whose status is exited

Without wait, the daemon asks supervisord to start the program and returns without its own status or readiness polling, so success: true alone does not confirm that anything is running. Send wait: true (with an optional timeout in seconds, default 30) to poll until the program is running or has finished. A program that finishes during the wait answers with status: "exited" and its last_run, whatever the exit code; a start that supervisord gives up on (fatal) is an error that also carries last_run. A finished run never counts as ready: a start that needs the program to serve, such as a waiting start with a ready_port or an on-demand start of a lazy_load program, fails with an error saying the program exited before accepting connections. When the program declares a non-zero ready_port, a waiting start also requires 127.0.0.1:<ready_port> to accept a TCP connection, which shows that something listens on IPv4 loopback, not that the application is healthy. if_not_running: true makes the start idempotent and reports already_running. Two cases skip the readiness probe: an already-running port-range instance, and a start that reuses a concurrent caller’s result, which may have been produced without waiting. timeout budgets the polling that follows configuration and the supervisord start, so it is not a deadline for the whole request, and a timeout does not cancel the program. lazy_load: true leaves a program stopped until something starts it on demand, and the API refuses a program that sets boot: true alongside it. Lazy loading is the stronger of the two wherever both are present: a lazy program never autostarts.

Only a disabled program needs enabling first:
POST /api/v1/daemon/programs/{id}/enable   Configuration: "program can run"
POST /api/v1/daemon/programs/{id}/start    Runtime: send {} or {"wait": true}

Auto-restart policies

Section titled “Auto-restart policies”

Configure what happens when a program exits:

{
  "name": "critical-service",
  "command": "/usr/bin/service",
  "user": "service",
  "autorestart": "true"
}

Options:

  • "true" - Always restart (recommended for services)
  • "false" - Run once: the program is not restarted after it exits, whatever its exit code
  • "unexpected" (default for persistent programs) - Restart after an exit code other than 0, including a deliberate nonzero exit, not only a crash

A program that exits on its own has finished, however quickly it exits, and its exit code is reported in last_run. With autorestart: "false" it runs once and its status becomes exited, also for a non-zero exit code or a signal. With "unexpected", an exit with code 0 is final in the same way.

The startup window still protects against crash loops. A program counts as started once it has stayed up for delay_seconds (when it is 0, the default, the window is 1 second). Under "unexpected" and "true", a non-zero exit or a signal inside that window is a failed start: supervisord retries it and reports fatal after repeated failures, so a command that crashes on startup does not restart forever. Under "true", a program that exits with code 0 is started again, at most once per window. A command that cannot be executed at all, such as a missing file, ends fatal with the reason in last_run.error; under "false" it is not retried.

A few behaviours follow from this:

  • A finished program can settle until its startup window ends. The window counts from the start, not from the exit. Its status already reads exited, but an edit or removal sent before delay_seconds have passed since the start (1 second at the default) can take until then to return. A stop through the stop endpoint ends the settling at once.
  • A program runs once per start, not once ever. A supervisord restart, a container restart, or a reload of the program’s changed definition starts a boot: true program again, as before, so a one-off boot command runs at every boot.
  • Hoody Kit programs keep the earlier rules. For a program the daemon reports as hoody_kit, an exit inside its startup window is a failed start and is retried whatever autorestart says.

Quick-start programs follow the same rules, but their autorestart defaults to "false": see quick-start programs.

Program configuration

Section titled “Program configuration”

Full program example:

{
  "name": "api-server",
  "description": "Main REST API server",
  "command": "/usr/bin/node /app/api/server.js",
  "user": "api",
  "enabled": true,
  "boot": true,
  "autorestart": "true",
  "directory": "/app/api",
  "priority": 10,
  "environment": {
    "NODE_ENV": "production",
    "PORT": "3000",
    "DB_HOST": "localhost"
  }
}

Required fields and defaults

Section titled “Required fields and defaults”

A create needs a unique name, a command and an existing system user. Names start with an ASCII letter or digit and contain only ASCII letters, digits, ., - and _, up to 128 characters. command must not contain a line feed, carriage return or NUL, and environment values must not contain a line feed or carriage return. An environment value must not contain whitespace followed by ; or # either, because supervisord reads the rest of the line as a comment and the program could not load: such a value is a 400 on create, on an edit that sends it and on quick-start, except for a program with terminal_id and no port_range, which accepts it. A NUL, carriage return or line feed in the value does not hide the pair, since those characters are removed before the line is written. a;b and #fff are fine. Environment names are C identifiers (^[a-zA-Z_][a-zA-Z0-9_]*$) of up to 128 characters, and every value is a string. An explicit directory must be an existing absolute path of ASCII letters and digits, /, _, ., -, spaces and tabs, with no .. path component.

An environment value may contain double quotes or single quotes, but not both kinds in one value, and it must not start or end with a quote character: supervisord strips quotes at the ends of a value, so such a value could not arrive as written. say "hi" now and it's here are fine; "quoted" and it's "both" are not. Such a value is a 400 on create, on an edit that sends it and on quick-start, and the error names the variable, never the value. A program with terminal_id, no port_range and a user other than root accepts it. A program saved earlier is not refused when an edit changes something else: a stored value with both kinds of quote is left out of the program’s environment and logged, and the rest of the program loads; a stored value with a quote at either end is written, and arrives without that quote. A value that uses %(ENV_NAME)s is expanded before supervisord reads the line, so the line can break when the variable itself contains a quote; keep such values simple. A sandboxed program’s values are held to a stricter rule, described under what changes for a sandboxed program.

FieldDefault
enabledtrue
bootfalse
lazy_loadfalse
autorestart"unexpected"
priority999
delay_seconds0
directorythe user’s home directory, or /tmp when it cannot be found
logs_enabledtrue
stdout_logfile, stderr_logfile/hoody/storage/hoody-daemon/logs/<name>/stdout.log and stderr.log; for a port-range program stdout_<port>.log and stderr_<port>.log
log_max_bytes, log_backups5 MiB per file, 2 rotated backups
inject_container_envtrue for Hoody Kit programs, false for every other program

Log paths are optional; without them the daemon uses the default directory and creates it. A custom path must stay under /hoody/storage/hoody-daemon/logs; any other path, such as /var/log/app.log, is a 400. When the directory a custom path names does not exist yet, the daemon creates it, inside the log directory only, and only when every directory above it is owned by root and not writable by other accounts and the path goes through no symbolic link. Otherwise, or when creating it fails, the program is saved but its configuration is not written: the create, edit, enable or start that applies it answers with an error naming the directory and the reason, and a quick-start is refused with a 400. A directory that already exists is used as before.

The stored environment is not always the whole process environment: inject_container_env decides whether the container’s variables are added, as described under container environment. hoody_kit is derived by the daemon and ignored in requests.

Command substitution is a separate thing from this map: for a standard program without terminal integration, $NAME and ${NAME} in the command are expanded from the daemon’s own environment when the definition is written, and an unknown name is left as written. A command that contains a supervisord %(ENV_NAME)s expression gets no $NAME or ${NAME} expansion at all. Port-range commands are not substituted. For a shell pipeline or a redirection, use a wrapper script.

terminal_id (1 to 65535, unique among programs) runs the program in a Hoody Terminal session. A non-empty terminal_shell (bash, zsh, fish, sh or tmux, installed in the container) requires it. Omit it for direct execution; an empty or whitespace-only string is accepted and means the same thing. terminal_interactive overrides the automatic choice between an interactive shell and a replaced one. Terminal integration needs socat in the container. A port-range program cannot use it: a request that combines terminal_id with port_range is refused with 400. display sets the X11 display for GUI programs, see Displays. A program with an effective sandbox cannot use terminal_id.

Container environment

Section titled “Container environment”

inject_container_env decides whether a program receives the container’s environment variables, the set the container’s init process holds, HOODY_* variables included. It defaults to true for Hoody Kit programs and to false for every other program, quick-start programs included.

  • true adds the container’s environment: ordinary programs receive all of it and Kit programs receive its HOODY_* variables (the daemon options --inject-all-env and --inject-hoody-env-only change that for every program). A value you set in environment overrides the injected one, so clearing your map does not give the program an empty environment.
  • false adds none of it, except the variables the command references by name as $NAME, ${NAME} or %(ENV_NAME)s, which are still provided. The program also inherits the small environment supervisord itself runs with, such as PATH and LANG, and receives everything you set in environment.

Only those three spellings count as a reference. A shell form with a modifier, such as ${NAME:-default}, ${NAME:+alt} or ${#NAME}, does not, and neither does a variable read only inside a script file the command runs, so that variable is not passed. To give the program one specific value, set it in environment; to give it the whole container environment, set inject_container_env: true.

On an edit, leaving the field out keeps the stored value. To return to the default, send the default value explicitly.

// Give an existing program the container's environment again
await containerClient.daemon.programs.update(programId, { inject_container_env: true });

As with any edit, a program that the daemon does not apply on edit keeps its old definition until its next start: see editing a program.

Watching and driving a terminal program

Section titled “Watching and driving a terminal program”

A program with terminal_id: N runs on terminal N of the container, and terminal N belongs to that program while it is configured. Open it like any terminal: in a browser at the terminal-N host, with the CLI, or from the SDK. Every attached client sees the same screen, writable clients type into it (Ctrl-C goes to the program), and the program gets the smallest size among the writable clients (read-only clients count only when no writable client is attached) and receives SIGWINCH when it changes. Detaching leaves the program running; when the last client leaves, it keeps its size.

Terminal window
# By id or by exact name. Ctrl-] detaches; --readonly watches without sending keys.
hoody daemon programs attach web-console -c my-container-id
hoody daemon programs attach 12 --readonly -c my-container-id

The terminal’s REST automation works on the program without a WebSocket: snapshot, find, press, mouse, paste, write, wait, raw and screenshot connect to the program on demand. Nothing is spawned on that terminal. Shell, working directory, user, environment, display, cmd and pid parameters are ignored there, and running a command (/execute), creating a session and the display redirect answer 409 DAEMON_TERMINAL.

SituationWebSocketREST automation
Program runningAttaches and shows its screenWorks
Program stopped or not startedCloses with 4404 daemon program not running409 DAEMON_PROGRAM_NOT_RUNNING
Program ends while attachedCloses with 4404 daemon program endedA pending wait answers exited
Program restartedA new attach reaches the new instance on a clean screen, at the clients’ sizeWorks
A shell session opened on the id before the program took itCloses with 4409409 DAEMON_TERMINAL; delete that session

The terminal session list (hoody terminal sessions list, SDK terminal.sessions.list()) shows every program terminal, opened or not, with daemon_program: true, no shell, and a daemon object holding the program’s program_id, program_name, user and enabled, whether it is running (its launcher is listening), whether the terminal is connected to it, and whether it ended while attached; pid is the terminal’s session leader (the program runs under it) while connected, 0 otherwise, and cols and rows are the terminal’s current size (null until something sized it). A local shell already open on the program’s id stays listed as that shell, with daemon_program: false and the program in its daemon object; delete it to reach the program. Deleting the terminal session (DELETE /api/v1/terminal/N) disconnects its clients and leaves the program running. Read-only is a per-connection choice, not a permission: any client that can reach the terminal can attach writable. The browser terminal keeps retrying while the program is stopped and attaches again once it runs; the CLI and the SDK stop at a 4404 and report it. A program that starts before anything attaches sees a terminal size of 0x0 until the first attach or REST call. Output it writes while hoody-terminal has no connection to it waits in the terminal, up to about 19 KB, and a program that writes more blocks until something attaches; once anything has attached or used the terminal’s REST automation, hoody-terminal stays connected until the program ends or the terminal session is deleted.

Editing a program

Section titled “Editing a program”

An edit changes only the top-level fields you send. An omitted field keeps its stored value, and so does a null one, except sandbox, where null clears the block. The create defaults do not apply. Four fields need care:

  • environment replaces the whole map. Editing {"A": "1", "B": "2"} with {"environment": {"A": "9"}} removes B, so resend every variable you want kept; {} clears the map.
  • port_range needs both start and end, and replaces the stored range.
  • webhooks merges field by field. It needs both enabled and urls, which replace the stored ones; omitted events, headers, timeout and retry keep their values, supplied headers replace the whole header map, and webhooks: null does not clear stored webhooks.
  • sandbox replaces the stored block as a unit, and null clears it. See Program Sandbox.

A partial port_range or webhooks object is a 400 naming the missing field, such as missing field `end` . A repeated top-level key is a 400. A near-miss of sandbox such as sandbx is a 400 when the real sandbox key is absent, and unknown fields inside a sandbox object are refused. Other unknown top-level fields are ignored, so a misspelt field name does nothing: use the documented names exactly.

When an edit takes effect depends on the program. An edit that leaves the program disabled removes its supervisord definition, which normally stops it. For an enabled program with a port_range, boot, lazy_load or a current or former sandbox, the edit rewrites the supervisord definition before returning and reloads it when the definition changed, which can stop or restart the running process. Any other enabled program keeps running its old definition until its next start.

Multiple instances with port_range

Section titled “Multiple instances with port_range”

port_range runs one process per port from a single program: {"start": 8000, "end": 8003} runs four. Each instance receives its port as --port=<port>, or as <port_param>=<port>, appended to the command, so the command must accept that equals form. Leave the port out of the command itself: a command that already has -p VALUE, -p=VALUE, --port VALUE or --port=VALUE suppresses the injection, and every instance then runs on whatever port the command names. An attached form such as -p8000 is not recognised, so the port argument is still appended. The range is inclusive, lies within 1 to 65535, covers at most 4096 ports, and must not overlap another program’s range. Start an instance with {"port": 8001}; stop one with the same body or every instance with {"all": true}. Start has no all-instances option.

Webhooks

Section titled “Webhooks”

A program can post its state changes to HTTPS endpoints:

{
  "webhooks": {
    "enabled": true,
    "urls": ["https://hooks.example.com/events"],
    "events": ["FATAL", "BACKOFF"]
  }
}

An enabled configuration takes 1 to 5 URLs. By default they must be HTTPS without credentials, and a URL whose host is localhost or a private or reserved IP address is refused; a DNS name is accepted when saved, and a private address it resolves to is refused at delivery, which the daemon log shows. A daemon running with NODE_ENV=development accepts HTTP and relaxes the destination checks. A disabled configuration is not checked. events names states without the PROCESS_STATE_ prefix (STARTING, RUNNING, BACKOFF, STOPPING, STOPPED, EXITED, FATAL, UNKNOWN); omitting it, [], "all" or "*" selects all. Up to 10 headers, timeout from 1 to 60 seconds (default 10) and retry from 0 to 5 (default 3) are optional. Each delivery is an independent in-memory request: nothing is queued across a daemon restart, ordering is not guaranteed, and a retry can deliver the same event twice (retry: 3 means up to four attempts, and timeout applies to each). Any response other than 2xx counts as a failure, redirects are not followed, and proxy environment variables are ignored; a destination the policy refuses is not retried. Make the receiver tolerate duplicates and answer 2xx once it has accepted the event.

Sandboxing a program

Section titled “Sandboxing a program”

Only the restrictions you write apply, and a block that restricts nothing ({} included) is stored as absent. A read-only root is opt-in; restricted limits TCP bind and connect ports, to any host, and does not cover UDP, and to keep that limit it also refuses MPTCP, io_uring and TCP Fast Open sends, so a program that requires io_uring does not run in that mode; every sandbox hides /run/user, so systemctl --user and the session bus are out of reach; ingress rules cover new IPv4 TCP connections on the declared ports on eth0, by default exempt packets whose source is loopback or the gateway (set ingress_allow_platform: false to filter those too; public requests through the proxy keep the client’s address and are never exempt), and their rate limit resets on every table rebuild and stops metering once its source set is full. Read what the sandbox does not do before relying on it.

A program may carry an optional sandbox block that confines it with kernel-enforced mechanisms: a read-only root with named writable directories, Landlock limits on the TCP ports it may bind and connect to, systemd limits on memory and tasks, an in-process limit on open files, and a per-port ingress allowlist and rate limit. A program without the block keeps the ordinary rendering path. The block is set on programs/add and programs/edit/{id} like any other field; on edit, an object replaces the stored block as a unit and null clears it.

{
  "name": "api-server",
  "command": "/usr/bin/node /hoody/storage/apps/api/server.js",
  "user": "api",
  "port_range": { "start": 3000, "end": 3000 },
  "sandbox": {
    "filesystem": { "read_only_root": true, "writable": ["/hoody/storage/apps/api/data"] },
    "network": { "mode": "restricted", "connect_ports": [5432, 53] },
    "process": { "max_memory": "512M" }
  }
}

A change to a sandboxed program’s resolved policy replaces its loaded definition, which stops the instance running under the old one. A sandboxed program reports a signal death as exit code 128+n, cannot run as any account that resolves to uid 0, and cannot carry terminal_id. Adding a sandbox can also reject environment values the program already had: dynamic-loader names, HOODY_SANDBOX_*, and values containing %, quotes, backslashes or control characters. The example above needs the api account and its directories to exist, and the container needs bubblewrap with --bind-fd, working systemd scopes, an existing /run/user directory, and Landlock ABI 4 or newer for restricted; GET /api/v1/daemon/health reports what is available. CLI sandbox edits refuse to drop stored settings unless you pass --sandbox-replace; HTTP and SDK edits replace the block as sent. Every field, the ingress firewall model, and the GET /programs/{id}/sandbox status endpoint are documented on Program Sandbox.

Process states

Section titled “Process states”

Monitor process health through the status endpoint:

  • running - Process running normally with PID
  • stopped - Process not running (expected; also returned when supervisord cannot be asked at all)
  • starting - Currently launching (temporary)
  • stopping - Gracefully shutting down (temporary)
  • backoff - Failed to start, retrying
  • exited - Process ran and then exited on its own; last_run holds its exit code
  • fatal - Failed to start after retries, or the command could not be executed; manual intervention needed
  • unknown - supervisord answered with something that could not be read as a status: a lookup failure, or a state outside the known set
  • failed - Quick-start programs only: supervisord never loaded the program’s definition, so nothing ran; last_run.error gives the reason

When the logical status differs from supervisord’s own state, status and quick-start responses add supervisor_state with supervisord’s state. This happens when a run has ended but its launcher is still finishing: status is exited while supervisor_state is running or starting.

Status response for a standard program:

{
  "success": true,
  "status": {
    "id": 1,
    "status": "running",
    "pid": 12345,
    "uptime": "2:15:30"
  }
}

A port-range program answers differently. With ?port=8001 the response carries instance; without it, instances with running_count and total_count. That list leaves out stopped instances and includes every other state, while running_count counts only the instances that are starting or running: check each instance’s own status for the rest. include_stats=true adds CPU, memory and process-tree figures, at the top level for a standard program and on each instance for a port-range program. cpu_percent is an average over the process’s whole life, not usage since the last poll, and can exceed 100% for a multi-threaded program. A status read whose supervisord call fails can report stopped or an empty list, which does not prove the process ended. Quick-start status is shaped differently: temporary_id, name, status and created_at sit beside success, with last_run once the program has finished, and a status command that cannot answer reports unknown.

Results of a finished run

Section titled “Results of a finished run”

Once a run of a program has ended, status responses and start responses carry a last_run object. That covers single status, each port-range instance in the program’s own status, standard programs in whole-list status, programs listed with include_status=true, and quick-start programs. Whole-list status reports a port-range program only as running or stopped, without its instances or last_run. For a program that exited with code 3:

{
  "success": true,
  "status": {
    "id": 7,
    "status": "exited",
    "last_run": {
      "exit_code": 3,
      "signal": null,
      "error": null,
      "ended_at": 1791370000123,
      "final": true
    }
  }
}
FieldMeaning
exit_codeThe program’s exit code, or null when it was ended by a signal or never ran
signalThe name of the signal that ended the program, such as "SIGKILL", or null
errorWhy the program never ran, such as a command that cannot be executed or a sandbox that cannot be set up, or why the daemon ended it, such as ttl expired or daemon restarted before the program finished; otherwise null
ended_atWhen the run ended, in milliseconds since the Unix epoch, or null when unknown
finaltrue when no automatic restart or retry follows, whether the run succeeded or not; false while supervisord will start the program again

A sandboxed program and a terminal program report a signal death as an exit code instead: 128+n, such as 137 for SIGKILL or 143 for SIGTERM, with signal set to null. When the daemon has only supervisord’s record of a run, for example for a run that ended while the daemon was restarting, and always for a Hoody Kit program, signal is null. exit_code holds the exit status supervisord recorded, except after a signal death, where supervisord records no usable code and exit_code is null as well. A run that could not be spawned has error set to supervisord’s spawn error and exit_code set to null. A run that exited too quickly to count as started is recorded by supervisord the same way: error reads Exited too quickly (process log may have details) and exit_code is null.

Listing programs

Section titled “Listing programs”

GET /api/v1/daemon/programs accepts filters that combine: exact name, enabled, boot, lazy_load and hoody_kit as true or false, port, and port_from with port_to (both required) for overlapping ranges. The port filters match port_range only, so they never find a program whose port lives in its command or in ready_port. include_status=true adds each program’s runtime status, and include_stats=true adds resource figures and also turns status on unless include_status=false is sent. A malformed boolean or port is a 400.

Reading logs

Section titled “Reading logs”

GET /api/v1/daemon/programs/{id}/logs returns the last lines of a log file as a snapshot, not a live stream. type is stdout (default) or stderr, lines defaults to 100 and is capped at 10000, and a port-range program needs port. Only the final 10 MiB of the file are read. The response carries logs, type, lines and log_file. Disabled logging, a missing file, and a file that cannot be read are errors, and bytes that are not valid UTF-8 come back as replacement characters rather than failing the request. When the file is larger than the 10 MiB window, a partial line at the start of the window is dropped; a single line longer than the whole window comes back as its truncated tail.

The endpoint reads the current file only. It does not join rotated files or earlier sessions: the default paths are symlinks to timestamped files, and session files older than the retention period (30 days by default) are deleted the next time a changed definition is applied. Copy anything you need to keep through the Files service first.

Terminal window
# Read the last 200 lines of the program's stderr log
hoody daemon programs logs get "$PROGRAM_ID" --type stderr --lines 200 -c "$CONTAINER_ID"

Quick-start programs

Section titled “Quick-start programs”

A quick-start program is a temporary supervised program that is not added to programs.json. It runs its command once by default: autorestart defaults to "false" for quick-start, so a command that exits is not started again, whatever its exit code, and a command that cannot be executed is not retried. Set autorestart explicitly to change that, for example "unexpected" to have a temporary server retried when it fails. Once per launch is not exactly once: sending the same launch request twice runs the command twice. It needs command and user, starts immediately, and answers with a temporary_id of the form quick_<milliseconds>_<sequence>, which is not a numeric program id. Keep that id: status, logs and stop take it. Every response after the program is created carries it, error responses included. Its command and environment are written to disk while it runs, as a supervisord definition and a recovery record. The definition is removed once the program finishes, is stopped or expires, so it cannot start again; the record keeps the result until the retention period below ends.

  • Lifetime. An optional ttl, in seconds, bounds the whole execution: it runs from launch until the program finishes, across startup, retries and automatic restarts. When it runs out the program is stopped and its last_run.error reads ttl expired. The launch response’s expires_at is that deadline. The check runs every 10 seconds, so it is not exact. Without ttl there is no timed expiry.
  • Results. A finished program keeps its status, last_run and logs reachable for 10 minutes after it finishes; after that its status and logs answer 404, and the log files stay on disk. Finished means exited with no automatic restart to follow, fatal, failed (supervisord never loaded the definition), or stopped through the stop endpoint. An exited program that autorestart will start again has not finished.
  • Daemon restarts. Finished programs keep their remaining retention across a daemon restart. A program that finished while the daemon was down keeps the exit code supervisord recorded. One that was still running is stopped, and its last_run.error reads daemon restarted before the program finished.
  • Names are not unique. Two runs with the same name share the default log paths; give separate runs distinct names or paths. Log files stay on disk after the program’s entry is gone.
  • Waiting. wait: true returns when the command finishes, or when a long-running program is running. A finished command answers with status: "exited" and its last_run, whatever its exit code; a program that ends fatal or cannot be executed answers with an error that carries temporary_id and last_run. Under the default autorestart: "false", a command that exits non-zero is not retried: the answer is HTTP 200 with status: "exited" and the exit code in last_run. Under "unexpected" or "true", a non-zero exit inside the startup window is retried and can end fatal. timeout defaults to 30 seconds and is capped at 300; it is the polling budget after the configuration and supervisord start attempts, not a deadline for the whole request. A wait that times out does not cancel the launch. When the daemon cannot tell whether supervisord started the program, it answers 502 with the temporary_id, and nothing is retried: read the program’s status before launching again.
  • Environment. inject_container_env defaults to false, so the program receives only the container variables its command references by name; set it to true to receive the container’s environment. See container environment.
  • No sandbox. A non-null sandbox is a 400, and so is a close misspelling such as sandbx, which catches common typos; other unknown keys are ignored. A quick-start program always runs without confinement, so use a persistent program when you need it. Quick-start also does not check terminal_id for uniqueness, or validate the shell you name.
Terminal window
# Launch a temporary program and keep its id (-o json prints the response)
QUICK_ID=$(hoody -o json daemon ephemeral programs start \
  --command "/usr/bin/python3 /opt/scripts/report-server.py" --user worker --ttl 600 \
  -c "$CONTAINER_ID" | jq -r '.temporary_id')


# Read its logs, then stop it
hoody daemon ephemeral programs logs get "$QUICK_ID" -c "$CONTAINER_ID"
hoody daemon ephemeral programs stop "$QUICK_ID" -c "$CONTAINER_ID"

To run a command once and receive its result in the response, set wait: true. The command runs once because of the default autorestart: "false". A command that finishes during its startup window answers with its result; one still running when the window ends answers running, and its status carries last_run once it finishes:

Terminal window
curl -X POST "https://PROJECT_ID-CONTAINER_ID-daemon-1.SERVER.containers.hoody.com/api/v1/daemon/quick-start" \
  -H "Content-Type: application/json" \
  -d '{"command": "/usr/bin/python3 /opt/scripts/migrate.py", "user": "worker", "wait": true, "ttl": 300}'
{
  "success": true,
  "temporary_id": "quick_1791370000000_1",
  "status": "exited",
  "last_run": {
    "exit_code": 0,
    "signal": null,
    "error": null,
    "ended_at": 1791370004210,
    "final": true
  }
}

Startup priority

Section titled “Startup priority”

Control the order in which programs are applied. These are configuration fragments for three existing programs, not a bulk request: apply each one with its own edit call.

[
  {
    "name": "database",
    "priority": 1,
    "boot": true
  },
  {
    "name": "cache",
    "priority": 5,
    "boot": true
  },
  {
    "name": "web-server",
    "priority": 10,
    "boot": true
  }
]

Lower priority number = starts earlier within one supervisord reload. It is not a dependency guarantee: at boot the daemon applies stored programs one at a time, so a program can start before a lower-numbered one is ready. A program that depends on another should wait and retry until that dependency answers.

Hoody Kit services

Section titled “Hoody Kit services”

Use cases

Section titled “Use cases”

Web servers and APIs

Section titled “Web servers and APIs”

Run Node.js, Python, or Go servers as daemons with auto-restart. Configure logging for debugging, set a startup priority if there are dependencies, and monitor them through the status endpoint.

Background workers

Section titled “Background workers”

Queue processors (Bull, Sidekiq, Celery), scheduled task runners, data sync services, cleanup jobs.

Custom background services

Section titled “Custom background services”

Run custom application services, workers, and scripts with a defined startup order and auto-restart policy. Do not use Hoody Daemon to manage package-installed system services such as PostgreSQL, MySQL, Redis, MongoDB, nginx, or apache. Use the system service manager (systemd, OpenRC) for those.

Monitoring agents

Section titled “Monitoring agents”

Prometheus exporters, log shippers, health check agents, metrics collectors.

Development servers

Section titled “Development servers”

Hot-reload dev servers, file watchers, test runners, development proxies.

Microservices

Section titled “Microservices”

Independent service processes, inter-service communication, graceful shutdown coordination, centralized process management.

Best practices

Section titled “Best practices”

Initial configuration

Section titled “Initial configuration”

Always add programs with enabled: false. Verify the configuration before enabling, test manually before setting boot: true, and turn on auto-restart only after stability testing.

Configuration updates

Section titled “Configuration updates”

Edit the program with only the fields you are changing. Resend the complete environment map, port_range or sandbox block when you change one, since an edit replaces them; webhooks merges field by field, with enabled and urls required. An enabled program with a port range, boot, lazy_load, or a current or former sandbox is applied by the edit itself, and an edit that disables a program removes its definition; restart any other enabled program so it picks up the change.

Logging strategy

Section titled “Logging strategy”

Logs are on by default under /hoody/storage/hoody-daemon/logs/<name>/ with rotation at 5 MiB and two backups. Adjust log_max_bytes and log_backups for chatty programs, keep custom paths under that root, and watch the logs for errors and warnings.

User permissions

Section titled “User permissions”

Run each program as an appropriate system user, never as root unless it is required. Create a dedicated user per service type and give it only the privileges it needs.

Dependency management

Section titled “Dependency management”

Use the priority field for startup ordering, low numbers for core services and higher numbers for dependent ones, and make dependent programs retry until their dependencies answer, because priority does not wait for readiness.

Monitoring

Section titled “Monitoring”

Poll the /status endpoint regularly, alert on backoff or fatal states, track uptime for reliability metrics, and run health checks through other services.

Useful questions

Section titled “Useful questions”

Q: What’s the difference between enable and start? Enable is configuration (“program can run”), start is runtime action (“launch the process”). Programs are enabled when created unless you pass enabled: false, and a disabled program must be enabled before it can start.

Q: How do I make a program start on boot? Set boot: true on an enabled program. It starts when the daemon applies boot programs at startup, and applying a boot program during a create, enable or edit can start it at once.

Q: Can I update a running program? Yes. Call the edit operation with only the fields you are changing; unspecified fields keep their current values, except that environment, port_range and sandbox replace the stored value when sent, and webhooks replaces enabled and urls. Treat an edit as potentially disruptive: when the program is one the daemon applies (a port range, boot enabled, lazy loading, or an effective sandbox, including an edit that clears a sandbox the program had), the edit writes the new configuration before it returns and reloads supervisord when the definition changed, which can stop or restart the running process. An edit that leaves the program disabled removes its definition, which normally stops it. Any other enabled program keeps its old configuration until you start it again. Disabling and re-enabling is not required.

Q: What happens if a process crashes? Depends on autorestart: "true" restarts it, "false" leaves it stopped, and "unexpected" (the default) restarts it after any exit code other than 0. Under "unexpected" and "true", a crash within delay_seconds of starting is a failed start, which supervisord retries before reporting fatal. Under "false" the program runs once, and last_run reports how it ended.

Q: How do I run a one-off command and get its exit code? Give the program autorestart: "false" and start it with wait: true. When the command finishes within its startup window (delay_seconds, 1 second at the default), the response carries status: "exited" and last_run.exit_code. A longer command answers running: read its status until last_run appears. A quick-start program runs once by default, so it needs only wait: true. See results of a finished run.

Q: How do I run multiple instances of the same program? Give one program a port_range, and the daemon runs one instance per port, passing each its port. See multiple instances with port_range.

Q: Can I see process output? Yes. Logs are on by default. Read a snapshot with the logs endpoint, as shown in reading logs, or tail the files under /hoody/storage/hoody-daemon/logs/<name>/ through the Files service.

Q: How do I check if a program is running? Call GET /api/v1/daemon/status/{id} and look for status: "running" and a pid field. For a port-range program, check each instance’s own status.

Troubleshooting

Section titled “Troubleshooting”

Program won’t start (backoff / fatal)

Section titled “Program won’t start (backoff / fatal)”

Cause: Command fails, port already in use, missing dependencies, wrong user permissions. Solution: Read the program’s last_run: error names a command that could not be executed, and exit_code shows how the last attempt ended. Check the stderr log file, verify the command works manually, make sure the port is free, confirm the user exists and has permissions, and run the command directly as that user to test.

An edit did not change the running program

Section titled “An edit did not change the running program”

Cause: The program is enabled and has no port_range, boot, lazy_load, or current or former sandbox, so the edit was saved but its running process keeps the old definition until its next start. Or the edit replaced a map it was meant to extend: environment is replaced as a whole. Solution: Stop and start the program. Read the program back to confirm the stored fields, and resend the full environment map when you change one variable. Disabling and re-enabling is not required.

Auto-restart not working

Section titled “Auto-restart not working”

Cause: autorestart: "false" or program disabled. Solution: Set autorestart: "true", make sure the program is enabled, check it is not in fatal state, and review the supervisord logs.

Program stops immediately after start

Section titled “Program stops immediately after start”

Cause: The command finished: it exits normally, has no keep-alive loop, or detaches into the background. A command that exits is reported as exited, with its exit code in last_run. Under "unexpected" and "true", a non-zero exit within delay_seconds of starting (1 second when it is 0) is a failed start and is retried until fatal. Solution: Read last_run and the stderr log. A service must stay in the foreground, so add a keep-alive loop or stop it from detaching. A one-off task is expected to end exited; give it autorestart: "false".

A program no longer sees a container variable

Section titled “A program no longer sees a container variable”

Cause: inject_container_env is false, the default for every program that is not a Hoody Kit program, including programs created before the field existed. The program then receives only the container variables its command references by name as $NAME, ${NAME} or %(ENV_NAME)s. A form such as ${NAME:-default}, or a variable read only inside a script file, is not such a reference. Solution: Set inject_container_env: true on the program, or set the variables it needs in environment. See container environment.

Priority not respected on boot

Section titled “Priority not respected on boot”

Cause: Priority orders programs only within one supervisord update, and at boot the daemon applies programs one at a time, so priority cannot make one program wait for another. A program may also lack boot: true. Solution: Make dependent programs wait and retry until their dependencies answer, and confirm each boot-started program is enabled with boot: true. delay_seconds is a start-success threshold, not an inter-program delay.

Status shows wrong state

Section titled “Status shows wrong state”

Cause: There is no status cache to go stale: every status read asks supervisord. A wrong state means the supervisord call did not answer or finish, in which case a single status read reports stopped and a whole-range read lists no instances, without the process having ended; or it answered with something that is not a status, in which case a single program or instance reports unknown and a whole-range read skips what it cannot parse and can list no instances; or supervisord’s own view differs from the process table, which happens when a process detaches or is killed outside supervisord. Solution: Read the status endpoint again, then check supervisord directly and confirm with ps whether the program is actually running. For a port-range program, read each instance’s own status rather than running_count. Restarting the daemon manager does not refresh anything.

What’s next

Section titled “What’s next”
Program Management Create and configure daemon programs
Process Control Start, stop, enable, and disable processes
Status Monitoring Track process states and uptime
Program Sandbox Confine a program's filesystem, network, and process limits