# Daemons reference **Page:** kit/daemons/reference [Download Raw Markdown](./kit/daemons/reference.md) --- # Daemons reference Lookup tables for [Hoody Daemons](/kit/daemons/). Every route, field and response is in the [API reference](/api/daemon/management/). ## Program fields | Field | Default | |---|---| | `enabled`, `logs_enabled` | `true` | | `boot`, `lazy_load` | `false` | | `boot_if_exists` | unset | | `autorestart` | `"unexpected"`; quick-start `"false"` | | `priority` | `999` | | `delay_seconds` | `0`, a startup window of 1 second | | `directory` | the user's home directory, or `/tmp` when it cannot be found | | `stdout_logfile`, `stderr_logfile` | `/hoody/storage/hoody-daemon/logs//stdout.log` and `stderr.log`; port-range `stdout_.log` and `stderr_.log` | | `log_max_bytes`, `log_backups` | 5 MiB per file, 2 rotated backups | | `inject_computer_env` | `true` for Hoody Kit programs, `false` otherwise | - **`name`:** starts with an ASCII letter or digit, then ASCII letters, digits, `.`, `-`, `_`, up to 128 characters. - **`directory`:** an existing absolute path of ASCII letters and digits, `/`, `_`, `.`, `-`, spaces and tabs, with no `..` component. - **`command`:** no line feed, carriage return or NUL. For a pipeline or redirection, use a wrapper script. - **`user`:** a dedicated account with only the privileges it needs; root only when required. - **`port_param`:** names the flag used to append the port, such as `-p`; valid only with `port_range`. See the [management API](/api/daemon/management/). - **`terminal_shell`:** the shell that loads the environment, or `tmux`: an installed `bash`, `zsh`, `fish`, `sh` or `tmux`. Non-blank requires `terminal_id`; an empty or whitespace-only string means direct execution. See the [management API](/api/daemon/management/). - **`terminal_interactive`:** overrides the automatic choice between an interactive shell and a replaced one. See the [management API](/api/daemon/management/). - **`display`:** a Hoody Display number from 1 to 40000 for GUI programs. See [Displays](/kit/displays/). - **`delay_seconds`:** a start-success threshold, not an inter-program delay. - **Unknown fields:** a repeated top-level key is a 400, and so is a near-miss such as `sandbx` when `sandbox` is absent. Other unknown top-level fields are ignored, so a misspelt name does nothing. ### Environment values | Rule | Detail | |---|---| | Names | C identifiers (`^[a-zA-Z_][a-zA-Z0-9_]*$`), up to 128 characters. | | Values | Strings, with no line feed or carriage return. | | Whitespace then `;` or `#` | 400: the rest would read as a comment and the program could not load. A NUL, carriage return or line feed does not hide the pair. `a;b` and `#fff` are fine. A program with `terminal_id` and no `port_range` accepts it. | | Quotes | One kind only, and none at either end. `say "hi" now` and `it's here` are fine; `"quoted"` and `it's "both"` are not. The 400 names the variable, never the value. A program with `terminal_id`, no `port_range` and a `user` other than `root` accepts it. | | Saved earlier | Not refused when an edit changes something else. Both quote kinds: left out and logged, the rest loads. A quote at either end: arrives without it. | | `%(ENV_NAME)s` | Expanded before the definition is read, which can break it if the variable holds a quote. | | Sandboxed | [Stricter rule](/kit/program-sandbox/#what-changes-for-a-sandboxed-program). | Both refusals apply on create, on an edit that sends the value, and on quick-start. ### Command substitution For a standard program without terminal integration, `$NAME` and `${NAME}` expand from the daemon's own environment when the definition is written, and an unknown name stays as written. A command with `%(ENV_NAME)s` gets no such expansion. Port-range commands are not substituted. ### Log paths - Without a path, the daemon uses the default directory and creates it. Rotation defaults are under [program fields](#program-fields). - A custom path must stay under `/hoody/storage/hoody-daemon/logs`. Any other, such as `/var/log/app.log`, is a 400. - A missing directory is created only there, when every directory above is owned by root and not writable by other accounts, and no symbolic link is involved. - Otherwise, or when creating it fails, the program is saved but not written. The create, edit, enable or start that applies it names the directory and reason; a quick-start is a 400. ## Computer environment This is the set the computer's init process holds. | `inject_computer_env` | The program receives | |---|---| | `true` | Ordinary programs: all of it. Kit programs: its `HOODY_*` variables. Daemon options `--inject-all-env` and `--inject-hoody-env-only` change that for every program. `environment` overrides, so clearing it does not empty the environment. | | `false` | Only variables the `command` references as `$NAME`, `${NAME}` or `%(ENV_NAME)s`, plus the process manager's own small environment (such as `PATH`, `LANG`) and `environment`. | - `${NAME:-default}`, `${NAME:+alt}` and `${#NAME}` are not references. - Through the process manager's environment line, an injected or referenced value with both quote kinds is left out, and a quote at either end is stripped. A non-root terminal program passes them through its launcher instead. A sandboxed program leaves out any non-[literal](/kit/program-sandbox/#what-changes-for-a-sandboxed-program) value. - Set values in `environment` to override injected variables. Unsupported quoting is refused with 400, but NUL characters are accepted and removed, and expansion expressions can change a value. Use literal strings without NUL or expansion syntax when you need exact bytes. - Programs created under earlier daemon versions, which gave every program this environment, now take the default. A running one picks that up the next time its definition is applied, which restarts it once. ## Editing An omitted field keeps its value. `null` also keeps it, except that it clears [`sandbox`](/kit/program-sandbox/#normalization-and-edits), `port_range` or `terminal_id`. Clearing `port_range` also clears an omitted `port_param`; clearing `terminal_id` also clears omitted `terminal_shell` and `terminal_interactive`. Create defaults do not apply. | Field | On edit | |---|---| | `environment` | Replaces the map. `{}` clears it. | | `port_range` | Needs `start` and `end`; replaces the range. | | `webhooks` | Needs `enabled` and `urls`, which replace the stored ones. Omitted `events`, `headers`, `timeout`, `retry` stay; supplied `headers` replace the map. `null` does not clear. | | `sandbox` | An object replaces the stored block as a unit, with no field merge, so resend every setting you want kept. `null` clears it; omitting it keeps it. The CLI guards against dropped settings, and the Hub sandbox editor refuses to save settings it cannot preserve. | | `boot_if_exists` | `null` keeps it; `""` clears it. | A partial `port_range` or `webhooks` is a 400 naming the field, such as ``missing field `end` ``. ### When a change applies | Situation | Result | |---|---| | Create | Applied only with a `port_range`, `boot: true`, `lazy_load: true` or an effective `sandbox`; otherwise at first start. | | Enable | The same programs are applied, and a boot-enabled one can start at once. | | Edit leaving it disabled | Removed from the process manager, which normally stops it. | | Edit, enabled, with `lazy_load`, current or former `port_range`, current or former `boot`, or current or former sandbox | Applied before returning; reloaded when the definition changed. | | Edit of any other enabled program | Keeps its old definition until its next start. | ### Results that mislead | You see | It means | |---|---| | Error after a change | Apply failed after saving, and the error says so. Read it and the program before retrying. | | Success on create, enable or edit | For an ordinary program without a sandbox, the definition may not be applied. Check status and the daemon log. | | Start refused during a change | An edit, removal or reset hit the program while the start was prepared, and nothing started. Send it again once the change has finished. A change after the start was issued does not refuse it, and applies to the process it started. | ### Reset Reset replaces the program list with the computer's default list, removes old programs from the process manager, attempts a reload and applies the defaults. Every program you added goes, so save the list first with `hoody daemon programs list -o json`. It can report success when the reload fails, so check the daemon log and processes. A missing, unreadable or malformed default list refuses it and changes nothing, though a start being prepared can still be refused. ## Starting - **`wait: true`:** a finished program answers `status: "exited"` with `last_run`, whatever the exit code. `fatal` is an error that also carries `last_run`. - **`timeout`:** budgets polling after configuration and the start request, not the whole request, and does not cancel the program. Quick-start too. - **Finished runs are never ready:** a finished run cannot satisfy a waiting start with `ready_port` or an on-demand `lazy_load` start; the start fails saying the program exited before accepting connections. One exception: if a waiting start joined a `ready_port` run that was already running, and that run exits before serving (a program that exits when idle), the start launches one successor when the old process shows as exited or still running and at least 2 seconds of its timeout remain. A stopped run is not restarted, and a run the start launched itself is never retried. - **`ready_port`:** an accepted connection shows something listens, not that the application is healthy. A crash before listening is a failed start. ## Restart rules - A program that exits on its own has finished, however quickly. `exited` under `"false"` for any code or signal; under `"unexpected"` for code 0. - A program counts as started once it has stayed up for `delay_seconds`. - Under `"true"`, an exit with code 0 starts again, at most once per window. - A command that cannot be executed ends `fatal`, with the reason in `last_run.error`; under `"false"` it is not retried. - A finished program can settle until the window ends, counted from the start: an edit or removal can take until then to return. A stop ends it at once. - A process manager or computer restart, or a reload of a changed definition, starts a `boot: true` program again, so a one-off `boot` command runs at every boot. - For a program reported as `hoody_kit`, an exit inside the window is a failed start, retried whatever `autorestart` says. ## Port ranges - The port is appended as `--port=` or `=`, so the command must accept that equals form. - `-p VALUE`, `-p=VALUE`, `--port VALUE` or `--port=VALUE` in the command suppresses this, and every instance runs on that port. An attached `-p8000` is not recognised. - The range is inclusive, within 1 to 65535, at most 4096 ports, and cannot overlap another program's. ## Marker files `boot_if_exists` adds a start and never holds one back: a `boot: true` program starts at boot either way. Use `boot: false`, usually with `lazy_load: true`. At each Daemon service start, an enabled program with the field starts when the path is a regular file (not a symbolic link or directory) and its status reads `STOPPED Not started`. | Event | Result | |---|---| | Computer restart with marker | Starts | | Daemon service restart only | A running program is left alone. A stopped one stays stopped until a computer restart with the marker. | | [Reset](/api/daemon/management/) | A program with a marker starts, even if you stopped it. | | No marker | Not started; a lazy program waits for its first request. | | Setting the field | Starts nothing. The file is not watched. | The value is an absolute path of at most 4096 bytes, with no `..` segment or control character. A `port_range` wider than one port is a 400, on create or edit. The program is applied at start-up even without `boot` or `lazy_load`. ## Webhooks Webhook delivery is off by default. Saving a configuration does not establish that notifications will be delivered; poll program status for monitoring you rely on. ```json { "webhooks": { "enabled": true, "urls": ["https://hooks.example.com/events"], "events": ["FATAL", "BACKOFF"] } } ``` | Field | Rule | |---|---| | `urls` | 1 to 5 when enabled: HTTPS, no credentials, public destinations. A name resolving to a non-public address is saved, then refused at delivery (see the daemon log). A disabled configuration is not checked. | | `events` | `STARTING`, `RUNNING`, `BACKOFF`, `STOPPING`, `STOPPED`, `EXITED`, `FATAL`, `UNKNOWN`, without the `PROCESS_STATE_` prefix. Omitted, `[]`, `"all"` or `"*"` selects all. | | `headers` | Up to 10. | | `timeout` | 1 to 60 seconds, default 10, per attempt. | | `retry` | 0 to 5, default 3. `retry: 3` means up to four attempts. A refused destination is not retried. | Delivery is in-memory: nothing is queued across a daemon restart, order is not guaranteed, and a retry can deliver twice. Only 2xx succeeds, redirects are not followed, and proxy variables are ignored. Make the receiver tolerate duplicates and answer 2xx once it has accepted the event. State events are matched to ordinary programs and port-range instances. Port-range webhook payloads include `metadata.instance` and `metadata.port`. Delivery requires `notifications.enabled` and `notifications.event_listener_enabled` set to `true` in `/hoody/storage/hoody-daemon/config/config.json`, and the daemon running as the process manager's event listener for process state events. Saving a program's webhook configuration alone delivers nothing. ## Process states | State | Meaning | |---|---| | `running` | Running normally, with a PID | | `stopped` | Not running, or the process manager cannot be asked at all | | `starting`, `stopping` | Launching, or shutting down gracefully | | `backoff` | Failed to start, retrying | | `exited` | Ran and exited on its own | | `fatal` | Failed after retries, or could not be executed; needs manual intervention | | `unknown` | The answer was unreadable: a lookup failure, or a state outside the known set. Quick-start: a status command that cannot answer. | | `failed` | Quick-start only: the definition never loaded, so nothing ran. `last_run.error` gives the reason. | `supervisor_state` appears when the process manager's state differs, as when a run ended and the daemon is finishing up. ### Status responses The [monitoring API](/api/daemon/monitoring/) has every shape. In whole-list status a port-range program shows one summary, with no instances or `last_run`. - `cpu_percent` is a whole-life average and can exceed 100% for a multi-threaded program. - With the process manager unreachable, one program or instance can report `stopped`, no proof it ended. A whole-range read is an error; an empty `instances` means a successful read. ### Results of a finished run `last_run` appears once a run has ended, in status and start responses. | Field or case | Report | |---|---| | `error` | Why it never ran (bad command, sandbox setup), or why the daemon ended it: `ttl expired`, `daemon restarted before the program finished` | | `source` | `launcher`, or `supervisor` for the process manager's record or the daemon's own, such as `ttl expired` | | Sandboxed or terminal program, signal death | `exit_code` `128+n` (137 for `SIGKILL`, 143 for `SIGTERM`), `signal` `null` | | Only the process manager's record (run ended during a daemon restart; always for Kit programs) | `signal` `null`; `exit_code` as recorded, `null` after a signal death | | Could not be spawned | `error` is the spawn error, `exit_code` `null` | | Exited too quickly to count as started | With only the process manager's record, `error` reads `Exited too quickly (process log may have details)` and `exit_code` is `null`. With the launcher's report, `exit_code` is the real exit code | ### Listing programs Port filters match `port_range` only, never a port in the command or `ready_port`. Combined filters, `include_status`, `include_stats` and query errors are in the [management API](/api/daemon/management/#programs). ## Logs - **Errors:** disabled logging, a missing file, or an unreadable file. - **Window:** only the final 10 MiB. A partial line at its start is dropped; a line longer than the window comes back as its truncated tail. - **Encoding:** invalid UTF-8 comes back as replacement characters. - **Files:** the current file only. For persistent programs with both log paths omitted, the defaults are symlinks to timestamped files; old session files past retention (30 days by default) are deleted when a changed definition is applied. Copy what you need through the Files service, which can also tail them. ## Quick-start The [API reference](/api/daemon/management/) covers the rest. | Topic | Rule | |---|---| | `ttl` | Optional, in seconds, counted from launch across retries and restarts. `expires_at` is the deadline. Expiry checks run every 10 seconds, so stopping can happen later. On expiry, `last_run.error` reads `ttl expired`. Without `ttl`, there is no timed expiry. | | Daemon restart | Finished programs keep their remaining retention. | | Repeat request | Runs it again. | | On disk | Command and environment are written while it runs. The definition goes when it finishes, stops or expires; the recovery record lasts through retention. | | Finished | Includes `failed`, but not an exit `autorestart` will restart. | | Names | Not unique. Same-name runs share default log paths, so use distinct ones. | | Waiting, default `autorestart` | Non-zero exit: HTTP 200, `status: "exited"`. Under `"unexpected"` or `"true"`, a non-zero exit in the window is retried and can end `fatal`. | ## Terminal programs A program with `terminal_id: N` owns terminal N while it is configured. Attach to it or request a terminal snapshot after starting it. Without a connection, output can fill the terminal buffer and block the program. Once connected, the terminal continues consuming output after clients disconnect. If the program restarts, connect again. The stdout log may omit output when no terminal connection is reading it or when the log cannot keep up. Open it like any terminal: ```bash # By id or by exact name. Ctrl-] detaches; --read-only watches without sending keys. hoody daemon programs terminal connect web-console -c "$COMPUTER_ID" hoody daemon programs terminal connect 12 --read-only -c "$COMPUTER_ID" ``` ```typescript import { HoodyClient } from 'hoody-sdk'; const client = new HoodyClient({ baseURL: 'https://api.hoody.com', token: process.env.HOODY_TOKEN }); const computer = { id: COMPUTER_ID, project_id: PROJECT_ID, server: SERVER }; const computerClient = await client.withComputer(computer); const { terminalId, ws } = await computerClient.daemon.programs.connectTerminal({ name: 'web-console' }, { cols: 120, rows: 40, onOutput: (bytes) => process.stdout.write(bytes), onDisconnect: (code, reason) => console.log(`closed ${code} ${reason}`), }); ws.sendInput('q'); ws.disconnect(); // the program keeps running ``` Choose an unused `terminal_id` from 1 to 9999 outside the agent-reserved range (4501 to 5000 by default), then open its terminal host, replacing `N` with that id: ```text https://PROJECT_ID-COMPUTER_ID-terminal-N.SERVER.computers.hoody.com/ ``` | Situation | WebSocket | REST automation | |---|---|---| | Running | Attaches | Works | | Stopped or not started | Closes `4404` `daemon program not running` | `409 DAEMON_PROGRAM_NOT_RUNNING` | | Ends while attached | Closes `4404` `daemon program ended` | A pending `wait` answers `exited` | | Restarted | A new attach reaches it on a clean screen, at the clients' size | Works | | A shell opened on the id first | Closes `4409` | `409 DAEMON_TERMINAL`; delete that session | - **Image:** keep its base packages installed. - **Size:** 0x0 until the first attach or REST call; kept when the last client leaves. The smallest reported width and height among writable clients set it, with `SIGWINCH`. If no writable client has reported a size, the smallest among read-only clients sets it instead. - **Clients:** Ctrl-C reaches the program. Read-only is a per-connection choice, not a permission: any client that can reach the terminal can attach writable. - **REST:** snapshot, find, press, mouse, paste, write, wait, raw and screenshot connect on demand. Shell, working directory, user, environment, display, `cmd` and `pid` parameters are ignored. Nothing is spawned: `/execute`, creating a session and the display redirect answer `409 DAEMON_TERMINAL`. - **Session list:** the [`daemon` object](/api/terminal/sessions/) has `running` (ready to connect) and `ended` (while attached). `pid` is the session leader while connected, else 0; `cols`, `rows` are `null` until sized. A local shell on the id stays listed as that shell, with `daemon_program: false` and the program in `daemon`. `DELETE /api/v1/terminal/N` disconnects clients and leaves the program running. - **Reconnects:** the browser retries while stopped. The SDK stops at `4404`. The CLI retries `4404 daemon program not running` for up to 5 seconds when status is `starting` or `running`; otherwise it stops. ## Kit services List all programs to see the Kit services in your computer, such as `hoody-terminal`. The Daemon service is not among them. To change a Kit service's flags, edit its program's `command`; see [change a Kit service's flags](/kit/daemons/#kit-service-flags). ### Daemon startup script The service manager starts the Daemon service through `/hoody/storage/hoody-daemon/scripts/startup.sh`. Root owns it, anyone can read it, changing it takes `sudo`, and updates keep your changes. Use it for a Daemon service variable or a step before start: ```bash #!/bin/bash export MY_VAR=value cd /hoody/storage/hoody-daemon || exit 1 exec /hoody/plugins/daemon/hoody-daemon "$@" ``` - Program variables belong in their `environment`. - Keep the last line: `"$@"` passes the service manager's options, and `exec` hands over the script's place. - A change applies at the next start, such as `sudo systemctl restart hoody-daemon`, and the API does not answer during it. - Options: `systemctl cat hoody-daemon` (`ExecStart`), `/etc/init.d/hoody-daemon` or root's crontab. ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | `backoff` or `fatal` | Failing command, port in use, missing dependencies, wrong permissions | Read `last_run` and stderr. Run the command as that user. | | Edit had no effect | Program not applied on edit, or `environment` replaced | Stop and start it. Read it back; resend the full map. | | No auto-restart | `autorestart: "false"`, disabled, or `fatal` | Set `"true"`, enable it, read the stderr and daemon logs. | | Stops right after start | Exited, no keep-alive, or detached | Read `last_run`. Keep services in the foreground; a one-off should end `exited`. | | Computer variable missing | `inject_computer_env` is `false` | Set it `true`, or use `environment`. | | Priority ignored at boot | At boot, programs are applied one at a time, so priority cannot make one wait; or no `boot: true` | Retry until dependencies answer; confirm enabled with `boot: true`. | | Wrong state | No status cache: no answer, an unreadable answer (a whole-range read skips such lines), or a process detached or killed outside the manager | Read again, confirm with `ps`. Restarting the daemon refreshes nothing. |