# Agent: Bot voice **Page:** api/agent/bots/voice [Download Raw Markdown](./api/agent/bots/voice.md) --- # Agent: Bot voice A Bot can be spoken to. `POST /api/v1/agent/bots/{id}/messages/audio` transcribes a recording and queues the transcript as the user's message. Three WebSocket doors make a voice controller for one Bot at a time: `/voice` (a custom voice protocol with subprotocol `hoody.voice.v1`), `/realtime` (an OpenAI Realtime client), and `/twilio` (one phone call as a Twilio Media Stream). `GET /api/v1/agent/bots/{id}/voice/check` runs the same speech readiness check that `/voice` runs with `require`, and answers whether the Bot's speech settings are ready without opening a connection. In the Hub at `hub.hoody.com`, open Bots and select the Bot. Open Settings, choose Voice and press `Check it works` to check readiness. To speak to the Bot, press `Talk` in its composer, allow microphone access, then use `Hold to talk`. Clients learn whether an agent supports `/voice/check`, `speak=false` and `require` from the `bot_voice_check` capability of `GET /version`. The speech and transcription endpoints themselves live on the [Audio](/api/agent/audio/) page; text messages are on the [Bot](/api/agent/bots/) page. The terminal command `hoody agent bots voice connect --id ` uses the `/voice` connection. See [kit/agent/voice](/kit/agent/voice/) for its flags. The base URL for every endpoint on this page is the same computer URL as every other agent page: `https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com` ## Check voice readiness ### `GET /api/v1/agent/bots/{id}/voice/check` Checks whether a Bot can talk by voice before `/voice` is opened, the same way speech is resolved when it is used: which speech-to-text model and, unless `speak=false`, which text-to-speech model the Bot would use (its own setting, else the agent's audio settings), that the model can do that, that the voice and language are ones it offers, and that the agent holds a key for its provider or can use the Hoody AI voice relay. The route answers `200` either way. `ready` says whether the configuration is ready; it is not a promise that a call to the speech provider will succeed. The `stt` and `tts` stages each carry `ok` and `model`; when `ok` is `false`, also `code`, `reason`, `message` (what happened and what to do next), `retryable` and, when known, `retry_after_secs` and `diag_id`. `tts` is `null` when `speak=false`. `max_utterance_secs` is the longest utterance a voice connection takes (60, or the speech-to-text model's limit when that is lower). When `ready` is `false`, `diag_id` names this check in the agent's log. The check bills nothing and takes at most 6 seconds. `/voice` with `require` makes the same check before its upgrade. The query parameters `in_rate`, `out_rate`, `in_format` and `mode` are checked for a valid value only (else `400 bad_request`); they do not change the result. ### Parameters | Name | In | Type | Required | Description | |------|-----|------|----------|-------------| | `id` | path | string | Yes | The bot id. | | `in_rate` | query | integer | No | Input sample rate: `8000`, `16000` (default), `24000` or `48000`. | | `out_rate` | query | integer | No | Output sample rate: `8000`, `16000`, `24000` (default) or `48000`. | | `in_format` | query | string | No | Input encoding: `pcm16` (16-bit little-endian, default), `pcmu` or `pcma`. | | `mode` | query | string | No | `vad` (default): the agent detects when the user starts and stops speaking. `ptt` (push to talk): the client sends `ptt.down` and `ptt.up`. | | `voice` | query | string | No | The voice to speak in; omitted, the Bot's own. | | `language` | query | string | No | The language the user speaks; omitted, the Bot's own. | | `speak` | query | boolean | No | `false`: check speech-to-text only, as for a voice connection opened with `speak=false`. Default `true`. | | `X-Hoody-Realm` | header | string | No | Per-request realm selector: `"global"` (not tied to a realm) or a 24-hex realm id (also accepted as `?realm=`). | | `realm` | query | string | No | The in:query alias of the `X-Hoody-Realm` header (read when the header is absent or empty). | ### Response Media type: application/json ```json { "diag_id": "9b2e47c01d5f8a36", "max_utterance_secs": 60, "ready": false, "speak": true, "stt": { "code": "speech_not_configured", "language": "auto", "message": "No speech model is set. Set the Bot's speech.stt_model or speech.tts_model, the agent setting audio.stt_model or audio.tts_model, or set audio.relay_default back to true (the Hoody AI voices).", "model": "", "ok": false, "reason": "not_configured", "retryable": false }, "tts": { "model": "minimax/speech-2.8-turbo", "ok": true, "voice": "English_expressive_narrator" } } ``` | Field | Type | Required | Description | |-------|------|----------|-------------| | `ready` | boolean | Yes | Whether every checked stage is ready. | | `speak` | boolean | Yes | The speak value the check ran with. | | `stt` | object | Yes | The speech-to-text stage. | | `stt.ok` | boolean | Yes | Whether this stage is ready. | | `stt.model` | string | Yes | The speech model spec that was checked; empty when none is set, or when the check could not run. | | `stt.language` | string | No | The language speech-to-text is asked for: a language code, or `auto` (empty when no language is set) for the model to detect it. | | `stt.code` | string | No | When `ok` is `false`: the error code (`speech_not_configured`, `speech_unavailable`, `model_unavailable`, `bad_request`, `service_unavailable`, ...). | | `stt.reason` | string | No | When `ok` is `false`: the machine reason (`not_configured`, `no_credentials`, `relay_not_configured`, `unknown_model`, `not_an_stt_model`, `voice_not_found`, `language_not_supported`, `check_timeout`, ...). | | `stt.message` | string | No | When `ok` is `false`: what happened and what to do next. | | `stt.retryable` | boolean | No | When `ok` is `false`: whether checking again later can succeed with nothing changed. | | `stt.diag_id` | string | No | When known: names the failure in the agent's log. | | `stt.retry_after_secs` | integer | No | When known: seconds to wait before trying again. | | `tts` | object | Yes | The text-to-speech stage; `null` with `speak=false`. | | `tts.ok` | boolean | Yes | Whether this stage is ready. | | `tts.model` | string | Yes | The speech model spec that was checked; empty when none is set, or when the check could not run. | | `tts.voice` | string | No | The voice a call would use. | | `tts.max_input_chars` | integer | No | When `ok` is `true` and known: the most characters of text one speech call takes with this model, for any of its languages and voices. Longer text is split before it is sent. | | `tts.code` | string | No | When `ok` is `false`: the error code (`speech_not_configured`, `speech_unavailable`, `model_unavailable`, `bad_request`, `service_unavailable`, ...). | | `tts.reason` | string | No | When `ok` is `false`: the machine reason (`not_configured`, `no_credentials`, `relay_not_configured`, `unknown_model`, `not_a_tts_model`, `voice_not_found`, `language_not_supported`, `check_timeout`, ...). | | `tts.message` | string | No | When `ok` is `false`: what happened and what to do next. | | `tts.retryable` | boolean | No | When `ok` is `false`: whether checking again later can succeed with nothing changed. | | `tts.diag_id` | string | No | When known: names the failure in the agent's log. | | `tts.retry_after_secs` | integer | No | When known: seconds to wait before trying again. | | `max_utterance_secs` | integer | Yes | The longest utterance a voice connection takes, in seconds: 60, or the speech-to-text model's limit when that is lower. | | `diag_id` | string | No | When `ready` is `false`: names this check in the agent's log. | Media type: application/json ```json { "code": "bad_request", "message": "invalid request" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `bad_request` | Bad request | The request was malformed or carried invalid parameters. | Correct the request body or query parameters. | | `realm_scope_unsupported` | Realm scope unsupported | A per-request scope header this route does not take: a realm or computer header on a route without that dimension, or, on an agent pinned to one realm, a different realm. | Omit the header, or name the realm the agent is pinned to. | | `invalid_realm` | Invalid realm selector | The realm selector is malformed (not `""`/`"global"`/a 24-hex id). | Pass a valid realm selector. | Media type: application/json ```json { "code": "forbidden", "message": "Hoody Kit programs are reached through their URLs only" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `forbidden` | Forbidden (Source IP Guard) | Refused by the Source IP Guard: Hoody Kit programs are reached through their URLs only. | Call the agent through its URL, for example with `hoody agent …`. | Media type: application/json ```json { "code": "not_found", "message": "resource not found" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `not_found` | Not found | The requested resource does not exist. | Verify the path and identifier. | Media type: application/json ```json { "code": "realm_blocked", "message": "realm is blocked" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `realm_blocked` | Realm blocked | The requested realm is blocked or corrupt; the request fails closed before dispatch. | Select a valid, reachable realm. | | `bot_owned_elsewhere` | Bot served by another agent | Another agent process on this machine serves the Bot (`details.reason` `held`), or its owner lock cannot be taken on this filesystem (`details.reason` `lock_unavailable`). Nothing of the Bot was read or changed. The answer carries `Retry-After: 5`. `details.owner` names the serving process (`pid`, `started_at`) when known, else `null`. | Retry after `Retry-After` seconds, or reach the Bot through the agent that serves it. `GET /api/v1/agent/bots` lists such a Bot as a row with `owned_elsewhere: true`. | Media type: application/json ```json { "code": "rate_limited", "message": "request rate limit exceeded" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `rate_limited` | Too many requests | The per-client request rate limit was exceeded; the gateway throttled the request before dispatch. | Honor the `Retry-After` header and retry; reduce the request rate. | Media type: application/json ```json { "code": "internal_error", "message": "internal server error" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `internal_error` | Internal error | An unexpected error occurred while handling the request. | Retry; if persistent, inspect the daemon logs. | Media type: application/json ```json { "code": "service_unavailable", "message": "service unavailable" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `service_unavailable` | Service unavailable | The daemon could not service the request (too busy, or a per-client stream concurrency cap was hit). | Honor `Retry-After` and retry. | | `restriction_unknown` | Login restriction unknown | The agent cannot read yet which realms its login may serve (`GET /api/v1/agent/hoody/auth/status` reports restriction `"unknown"`), so it refuses realm-scoped requests until it can. Nothing was read or changed. | Retry after the restriction is readable. | ### SDK usage ```ts import { HoodyClient } from 'hoody-sdk'; const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' }); await client.agent.bots.getVoiceCheck('67e89abc123def456789abcd'); ``` ```bash curl -G \ "https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com/api/v1/agent/bots/67e89abc123def456789abcd/voice/check" ``` ## Voice WebSocket ### `GET /api/v1/agent/bots/{id}/voice` Upgrades to a WebSocket with subprotocol `hoody.voice.v1` that carries the user's speech to the Bot and the Bot's spoken replies back. Binary frames from the client are input audio: one byte `0x00`, a big-endian u32 sequence number, then 20 to 200 ms of audio in `in_format` at `in_rate`. A `pcm16` frame of an odd byte length is not decoded: it counts as lost, and an utterance it falls in ends with `transcript.final` rejected `frame_loss`. Audio sent faster than 1.5 times real time over 2 seconds is also lost. Binary frames from the agent are output audio: one byte `0x01`, a big-endian u32 `out`, a big-endian u32 sample offset, then 16-bit little-endian PCM at `out_rate`. Text frames are JSON objects with a `type`. The client sends `input.commit`, `ptt.down`, `ptt.up`, `output.cancel`, `playback.mark`, `own_gate.decide`, `own_gate.answer`, `text` and `ping`; the agent sends `session`, `vad.start`, `vad.end`, `transcript.partial`, `transcript.final`, `reply.provisional`, `reply.invalidate`, `reply.final`, `reply.diverged`, `output.start`, `output.end`, `output.stop`, `cancel.result`, `gate.announced`, `own_gate.pending`, `own_gate.resolved`, `busy`, `error`, `takeover` and `pong`. `mode` is fixed for the connection: `vad` (the default) or `ptt`. In `ptt` mode only audio between `ptt.down` and `ptt.up` counts; other audio is dropped. One utterance is at most 60 seconds, or the speech-to-text model's limit when that is lower. A spoken message counts once the Bot's log shows it. Only replies to this connection's own messages are spoken (and replies the Bot gives on its own, with `speak_wakes`); every reply's text still arrives in `reply.final`. `reply.final` carries `own: true` when the turn answers a message sent on this connection, `false` when it does not, and no `own` when the connection cannot tell. An absent `own` means unknown, and you can then match `turn_id` in `GET /api/v1/agent/bots/{id}/log`. `own` is about this connection only: a message sent on an earlier connection is answered with `own false`. With `speak=false` no speech is made: replies arrive as text only, every `reply.final` has `spoken false`, and nothing is billed for speech synthesis (speech-to-text is still billed); the `session` frame then shows `speak false`. A speech failure is described by an error object `{code, reason, message, diag_id, retryable, retry_after_secs}`. Its `message` says what happened and what to do next; `diag_id` names the failure in the agent's log. The error object appears: - on `transcript.final` when the transcription failed or was rejected as `stt_suspect`, - on `reply.final` when `reason` is `tts_failed` or `tts_incomplete`, - on an `error` frame with `code service_unavailable` when the speech service failed on an announcement (`tts_failed` or `tts_incomplete`, once per code and reason per connection). Voice never answers a delegate's approval or question: answer it on screen or by typing. The Bot's own approvals and questions are answered with `own_gate.decide` and `own_gate.answer`; their outcome (`applied`, `resolved`, `stale`, `unavailable`, `not_offered`, `no_pending_gate`, `stale_gate`, `gate_cancelled` or `decision_unconfirmed`) arrives in an `own_gate.resolved` frame, never as an HTTP status. One connection controls a Bot's voice at a time: a second one is refused `409 voice_busy` unless it asks `takeover=true`, which closes the first. With `require=stt`, `require=tts` or `require=stt,tts` the agent first checks those speech settings as `GET /api/v1/agent/bots/{id}/voice/check` does: when one is not ready it answers `400 bad_request` if every failure is a voice or language the speech model does not offer, otherwise `503 speech_unavailable`, with the check's result in `details`, and takes nothing over. Without `require` nothing is checked before the upgrade. Take the realm selector from `X-Hoody-Realm` (or `?realm=`); omit it for the agent's current realm. A realm this login does not serve, or a Bot id the realm does not hold, answers `404 not_found`. `"all"` is accepted only as `?realm=all` on `GET /api/v1/agent/bots` and `POST /api/v1/agent/bots/resume`; anywhere else it is `400 bad_request`. An agent pinned to one realm refuses any other realm, and `"all"`, with `400 realm_scope_unsupported`. The cwd, config-dir and computer headers are refused with `400 realm_scope_unsupported`. ### Parameters | Name | In | Type | Required | Description | |------|-----|------|----------|-------------| | `id` | path | string | Yes | The bot id. | | `in_rate` | query | integer | No | Input sample rate: `8000`, `16000` (default), `24000` or `48000`. | | `out_rate` | query | integer | No | Output sample rate: `8000`, `16000`, `24000` (default) or `48000`. | | `in_format` | query | string | No | Input encoding: `pcm16` (16-bit little-endian, default), `pcmu` or `pcma`. | | `mode` | query | string | No | `vad` (default): the agent detects when the user starts and stops speaking. `ptt` (push to talk): the client sends `ptt.down` and `ptt.up`. | | `voice` | query | string | No | The voice to speak in; omitted, the Bot's own. | | `language` | query | string | No | The language the user speaks; omitted, the Bot's own. | | `speak_wakes` | query | boolean | No | `true`: also speak replies the Bot gives on its own; omitted, the Bot's setting. | | `takeover` | query | boolean | No | `true`: take over from the connection that controls the Bot's voice now. | | `speak` | query | boolean | No | `false`: replies arrive as text only (`reply.final`); no speech is made and nothing is billed for speech synthesis. Speech-to-text still runs. Default `true`. | | `refuse` | query | string | No | `frame`: when the agent refuses the connection, accept the upgrade, send one refused frame with the code, message, details and retry_after_secs of the refusal, then close with 4000 plus its HTTP status (4400, 4404, 4409, 4500 or 4503), so a client that cannot read an HTTP refusal of a WebSocket, such as a browser, learns why. Omitted: refusals are HTTP answers. | | `play_lead` | query | boolean | No | `true`: the agent keeps at most 8 seconds of reply audio ahead of what `playback.mark` says was played, and a reply whose audio waits 5 seconds with no mark moving on stops speaking (`output.stop` and `reply.final` reason `no_marks`, `spoken unknown`, or `partial` when whole segments were heard; the turn goes on). Send `playback.mark` at least every 500 ms while playing. Omitted or `false`: reply audio is sent as fast as the connection takes it. | | `require` | query | string | No | `stt`, `tts` or `stt,tts`: check those speech settings before the upgrade and refuse it when they are not ready (`400 bad_request` for a voice or language the model does not offer, else `503 speech_unavailable`). `tts` needs `speak=true`. Omitted: no check. | | `X-Hoody-Realm` | header | string | No | Per-request realm selector: `"global"` (not tied to a realm) or a 24-hex realm id (also accepted as `?realm=`). | | `realm` | query | string | No | The in:query alias of the `X-Hoody-Realm` header (read when the header is absent or empty). | ### Connection lifecycle The connection closes with: - `1000` when the client closes it, after 10 minutes without a frame from the client (the agent's own output does not count; send `ping` to stay connected), when another connection takes over, or when the Bot is deleted or suspended; - `1001` when the Bot is stopped, as when the agent shuts down; - `1002` when the client did not offer `hoody.voice.v1`; - `1008` for a malformed frame, more than 50 control frames a second, an input frame outside 20 to 200 ms, or a playback mark past the audio sent; - `1009` for a frame over 64 KiB; - `1011` when a write fails, the agent runs out of voice memory, or on an internal error. A client that does not read gets no close frame: the connection just ends. ### Response Media type: none The voice connection is open. The 101 response carries no body. Media type: application/json ```json { "code": "bad_request", "message": "invalid request" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `bad_request` | Bad request | The request was malformed or carried invalid parameters. | Correct the request body or query parameters. | | `realm_scope_unsupported` | Realm scope unsupported | A per-request scope header this route does not take: a realm or computer header on a route without that dimension, or, on an agent pinned to one realm, a different realm. | Omit the header, or name the realm the agent is pinned to. | | `invalid_realm` | Invalid realm selector | The realm selector is malformed (not `""`/`"global"`/a 24-hex id). | Pass a valid realm selector. | | `bad_request` | Voice or language not offered | Only with `require` (capability `bot_voice_check`): every required speech stage that is not ready fails on a voice or language its model does not offer. `details` is the check's result (as `GET /api/v1/agent/bots/{id}/voice/check` answers it), with a stage that was not required `null`. Nothing was taken over. A 400 `bad_request` for a malformed query parameter carries no `details`. | Pick a voice or language the model offers (or leave it out), then connect again. | Media type: application/json ```json { "code": "forbidden", "message": "Hoody Kit programs are reached through their URLs only" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `forbidden` | Forbidden (Source IP Guard) | Refused by the Source IP Guard: Hoody Kit programs are reached through their URLs only. | Call the agent through its URL, for example with `hoody agent …`. | Media type: application/json ```json { "code": "not_found", "message": "resource not found" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `not_found` | Not found | The requested resource does not exist. | Verify the path and identifier. | Media type: application/json ```json { "code": "voice_busy", "message": "another voice connection controls this Bot" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `voice_busy` | Voice busy | Another connection controls this Bot's voice. | Close the other connection, or connect with `takeover=true` to take over from it. | | `realm_blocked` | Realm blocked | The requested realm is blocked or corrupt; the request fails closed before dispatch. | Select a valid, reachable realm. | | `bot_owned_elsewhere` | Bot served by another agent | Another agent process on this machine serves the Bot (`details.reason` `held`), or its owner lock cannot be taken on this filesystem (`details.reason` `lock_unavailable`). Nothing of the Bot was read or changed. The answer carries `Retry-After: 5`. `details.owner` names the serving process (`pid`, `started_at`) when known, else `null`. | Retry after `Retry-After` seconds, or reach the Bot through the agent that serves it. `GET /api/v1/agent/bots` lists such a Bot as a row with `owned_elsewhere: true`. | Media type: application/json ```json { "code": "rate_limited", "message": "request rate limit exceeded" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `rate_limited` | Too many requests | The per-client request rate limit was exceeded; the gateway throttled the request before dispatch. | Honor the `Retry-After` header and retry; reduce the request rate. | Media type: application/json ```json { "code": "internal_error", "message": "internal server error" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `internal_error` | Internal error | An unexpected error occurred while handling the request. | Retry; if persistent, inspect the daemon logs. | Media type: application/json ```json { "code": "service_unavailable", "message": "service unavailable" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `service_unavailable` | Service unavailable | The daemon could not service the request (too busy, or a per-client stream concurrency cap was hit). | Honor `Retry-After` and retry. | | `restriction_unknown` | Login restriction unknown | The agent cannot read yet which realms its login may serve (`GET /api/v1/agent/hoody/auth/status` reports restriction `"unknown"`), so it refuses realm-scoped requests until it can. Nothing was read or changed. | Retry after the restriction is readable. | | `speech_unavailable` | Speech not ready | Only with `require`: a required speech stage is not ready for a reason other than a voice or language the speech model does not offer. `details` is the check's result (as `GET /api/v1/agent/bots/{id}/voice/check` answers it), with a stage that was not required `null`. Nothing was taken over. | Do what the failing stage's message says, then connect again; honor `Retry-After` when it is sent. | ### SDK usage `client.agent.bots.connectVoice` returns an unopened typed socket that is not an EventEmitter. Register the handlers, then call `await socket.connect()` and keep the returned socket. ```ts import { HoodyClient } from 'hoody-sdk'; const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' }); const socket = await client.agent.bots.connectVoice('67e89abc123def456789abcd', { mode: 'vad', speak: true }); socket.onTranscriptFinal((m) => console.log('heard', m.utt, m.text)); socket.onReplyFinal((m) => console.log('reply', m.turn_id, m.text)); socket.onError((err) => console.error(err)); await socket.connect(); // send microphone frames with socket.sendInputAudio(seq, pcm); later: socket.close(); ``` ```bash curl -i -N \ -H "Connection: Upgrade" \ -H "Upgrade: websocket" \ -H "Sec-WebSocket-Version: 13" \ -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \ -H "Sec-WebSocket-Protocol: hoody.voice.v1" \ "https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com/api/v1/agent/bots/67e89abc123def456789abcd/voice?mode=vad&speak=true" ``` ## Realtime WebSocket ### `GET /api/v1/agent/bots/{id}/realtime` Upgrades to a WebSocket that speaks the OpenAI Realtime protocol (GA event names), so a Realtime client such as LiveKit Agents or Pipecat talks to the Bot. The client sends `session.update`, `input_audio_buffer.append`, `.commit` and `.clear`, `conversation.item.create`, `.delete`, `.retrieve` and `.truncate`, `response.create` and `response.cancel`; the agent answers with the matching server events. Input audio is `audio/pcm` (24 kHz, 16-bit little-endian, the default) or `audio/pcmu` or `audio/pcma` (8 kHz); output audio follows the session's output format. With `server_vad` the agent hears when the user starts and stops speaking and answers on its own; without turn detection the client commits and asks with `response.create`. `semantic_vad` runs the same detection; with no `silence_duration_ms` it ends a turn after 1200 ms of silence for `eagerness` `low`, 800 ms for `medium` or `auto`, and 500 ms for `high`. A `conversation.item.create` is never undone: a user item reaches the Bot once, by its id. The Bot records every user item id it receives and never forgets one, so a client replaying its history on reconnect does not send an old item again; at the Bot's limit (`realtime_items` on `GET /api/v1/agent/bots/{id}`) a new item is refused and the connection closed. An event that would change an item already sent (deleting, truncating or editing a user item that waits for a response) closes the connection with `1008`; so do a binary or non-JSON frame, a frame over 1 MiB, more than 50 control events a second, and input audio faster than 1.5 times real time. One connection controls a Bot's voice at a time, over this door, the voice WebSocket or a phone call: another is refused `409 voice_busy` unless it asks `takeover=true`, which closes the one before. A request that is not a WebSocket upgrade is refused `400 bad_request` and takes nothing over. ### Parameters | Name | In | Type | Required | Description | |------|-----|------|----------|-------------| | `id` | path | string | Yes | The bot id. | | `takeover` | query | boolean | No | `true`: take over from the connection that controls the Bot's voice now. | | `X-Hoody-Realm` | header | string | No | Per-request realm selector: `"global"` (not tied to a realm) or a 24-hex realm id (also accepted as `?realm=`). | | `realm` | query | string | No | The in:query alias of the `X-Hoody-Realm` header (read when the header is absent or empty). | ### Response Media type: none The Realtime connection is open. The 101 response carries no body. Media type: application/json ```json { "code": "bad_request", "message": "invalid request" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `bad_request` | Bad request | The request was malformed or carried invalid parameters. | Correct the request body or query parameters. | | `realm_scope_unsupported` | Realm scope unsupported | A per-request scope header this route does not take: a realm or computer header on a route without that dimension, or, on an agent pinned to one realm, a different realm. | Omit the header, or name the realm the agent is pinned to. | | `invalid_realm` | Invalid realm selector | The realm selector is malformed (not `""`/`"global"`/a 24-hex id). | Pass a valid realm selector. | Media type: application/json ```json { "code": "forbidden", "message": "Hoody Kit programs are reached through their URLs only" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `forbidden` | Forbidden (Source IP Guard) | Refused by the Source IP Guard: Hoody Kit programs are reached through their URLs only. | Call the agent through its URL, for example with `hoody agent …`. | Media type: application/json ```json { "code": "not_found", "message": "resource not found" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `not_found` | Not found | The requested resource does not exist. | Verify the path and identifier. | Media type: application/json ```json { "code": "voice_busy", "message": "another voice connection controls this Bot" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `voice_busy` | Voice busy | Another connection controls this Bot's voice. | Close the other connection, or connect with `takeover=true` to take over from it. | | `realm_blocked` | Realm blocked | The requested realm is blocked or corrupt; the request fails closed before dispatch. | Select a valid, reachable realm. | | `bot_owned_elsewhere` | Bot served by another agent | Another agent process on this machine serves the Bot (`details.reason` `held`), or its owner lock cannot be taken on this filesystem (`details.reason` `lock_unavailable`). Nothing of the Bot was read or changed. The answer carries `Retry-After: 5`. `details.owner` names the serving process (`pid`, `started_at`) when known, else `null`. | Retry after `Retry-After` seconds, or reach the Bot through the agent that serves it. `GET /api/v1/agent/bots` lists such a Bot as a row with `owned_elsewhere: true`. | Media type: application/json ```json { "code": "rate_limited", "message": "request rate limit exceeded" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `rate_limited` | Too many requests | The per-client request rate limit was exceeded; the gateway throttled the request before dispatch. | Honor the `Retry-After` header and retry; reduce the request rate. | Media type: application/json ```json { "code": "internal_error", "message": "internal server error" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `internal_error` | Internal error | An unexpected error occurred while handling the request. | Retry; if persistent, inspect the daemon logs. | Media type: application/json ```json { "code": "service_unavailable", "message": "service unavailable" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `service_unavailable` | Service unavailable | The daemon could not service the request (too busy, or a per-client stream concurrency cap was hit). | Honor `Retry-After` and retry. | | `restriction_unknown` | Login restriction unknown | The agent cannot read yet which realms its login may serve (`GET /api/v1/agent/hoody/auth/status` reports restriction `"unknown"`), so it refuses realm-scoped requests until it can. Nothing was read or changed. | Retry after the restriction is readable. | ### SDK usage `client.agent.bots.connectRealtime` returns an unopened typed socket that is not an EventEmitter. Register the handlers, then call `await socket.connect()` and keep the returned socket. ```ts import { HoodyClient } from 'hoody-sdk'; const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' }); const socket = await client.agent.bots.connectRealtime('67e89abc123def456789abcd'); socket.onSessionCreated(() => console.log('session ready')); socket.onResponseDone((m) => console.log('response done', m)); socket.onError((err) => console.error(err)); await socket.connect(); // later: socket.close(); ``` ```bash curl -i -N \ -H "Connection: Upgrade" \ -H "Upgrade: websocket" \ -H "Sec-WebSocket-Version: 13" \ -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \ "https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com/api/v1/agent/bots/67e89abc123def456789abcd/realtime" ``` ## Twilio Media Stream ### `GET /api/v1/agent/bots/{id}/twilio` The WebSocket URL for a TwiML `` element containing a `` child: one phone call becomes the Bot's voice controller. Twilio's stream events are taken as documented (`connected`, `start`, `media`, `mark`, `stop`, `dtmf`); audio is μ-law at 8 kHz, mono. What the caller says is cut into utterances by voice activity detection; each one is transcribed in order and posted to the Bot as the user's message. The Bot's reply comes back as 20 ms media frames, one `mark` per spoken segment. When the caller speaks over the reply, its playback stops (a `clear` is sent). A `stop` event, or the call ending, drops the utterance still being spoken: it is never transcribed or posted; utterances already complete are still posted. DTMF digits are ignored. Gates are never announced or answered by voice. The call takes over from another open voice controller only when the stream's `start` event carries the custom parameter `takeover=true` (or the URL has `?takeover=true`); otherwise a second controller is refused and the stream closes with code `1008`. Access to this URL is controlled by the agent's proxy permission rules; configure one before pointing a Twilio number at it. Live Twilio accounts are not supported in this release: the door is tested against a Twilio Media Streams simulator. ### Parameters | Name | In | Type | Required | Description | |------|-----|------|----------|-------------| | `id` | path | string | Yes | The bot id. | | `takeover` | query | boolean | No | `true`: take over from the Bot's open voice controller. | | `X-Hoody-Realm` | header | string | No | Per-request realm selector: `"global"` (not tied to a realm) or a 24-hex realm id (also accepted as `?realm=`). | | `realm` | query | string | No | The in:query alias of the `X-Hoody-Realm` header (read when the header is absent or empty). | ### Response Media type: none The WebSocket upgrade is accepted; the connection carries Twilio Media Streams frames. The 101 response carries no body. Media type: application/json ```json { "code": "bad_request", "message": "invalid request" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `bad_request` | Bad request | The request was malformed or carried invalid parameters. | Correct the request body or query parameters. | | `realm_scope_unsupported` | Realm scope unsupported | A per-request scope header this route does not take: a realm or computer header on a route without that dimension, or, on an agent pinned to one realm, a different realm. | Omit the header, or name the realm the agent is pinned to. | | `invalid_realm` | Invalid realm selector | The realm selector is malformed (not `""`/`"global"`/a 24-hex id). | Pass a valid realm selector. | Media type: application/json ```json { "code": "forbidden", "message": "Hoody Kit programs are reached through their URLs only" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `forbidden` | Forbidden (Source IP Guard) | Refused by the Source IP Guard: Hoody Kit programs are reached through their URLs only. | Call the agent through its URL, for example with `hoody agent …`. | Media type: application/json ```json { "code": "not_found", "message": "resource not found" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `not_found` | Not found | The requested resource does not exist. | Verify the path and identifier. | Media type: application/json ```json { "code": "realm_blocked", "message": "realm is blocked" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `realm_blocked` | Realm blocked | The requested realm is blocked or corrupt; the request fails closed before dispatch. | Select a valid, reachable realm. | | `bot_owned_elsewhere` | Bot served by another agent | Another agent process on this machine serves the Bot (`details.reason` `held`), or its owner lock cannot be taken on this filesystem (`details.reason` `lock_unavailable`). Nothing of the Bot was read or changed. The answer carries `Retry-After: 5`. `details.owner` names the serving process (`pid`, `started_at`) when known, else `null`. | Retry after `Retry-After` seconds, or reach the Bot through the agent that serves it. `GET /api/v1/agent/bots` lists such a Bot as a row with `owned_elsewhere: true`. | Media type: application/json ```json { "code": "rate_limited", "message": "request rate limit exceeded" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `rate_limited` | Too many requests | The per-client request rate limit was exceeded; the gateway throttled the request before dispatch. | Honor the `Retry-After` header and retry; reduce the request rate. | Media type: application/json ```json { "code": "internal_error", "message": "internal server error" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `internal_error` | Internal error | An unexpected error occurred while handling the request. | Retry; if persistent, inspect the daemon logs. | Media type: application/json ```json { "code": "service_unavailable", "message": "service unavailable" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `service_unavailable` | Service unavailable | The daemon could not service the request (too busy, or a per-client stream concurrency cap was hit). | Honor `Retry-After` and retry. | | `restriction_unknown` | Login restriction unknown | The agent cannot read yet which realms its login may serve (`GET /api/v1/agent/hoody/auth/status` reports restriction `"unknown"`), so it refuses realm-scoped requests until it can. Nothing was read or changed. | Retry after the restriction is readable. | ### SDK usage Twilio normally opens this connection; the SDK form is for tests. `client.agent.bots.connectTwilio` returns an unopened typed socket that is not an EventEmitter. Register the handlers, then call `await socket.connect()` and keep the returned socket. ```ts import { HoodyClient } from 'hoody-sdk'; const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' }); const socket = await client.agent.bots.connectTwilio('67e89abc123def456789abcd'); socket.onMedia((m) => console.log('media', m)); socket.onMark((m) => console.log('mark', m)); socket.onError((err) => console.error(err)); await socket.connect(); // later: socket.close(); ``` ```bash curl -i -N \ -H "Connection: Upgrade" \ -H "Upgrade: websocket" \ -H "Sec-WebSocket-Version: 13" \ -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \ "https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com/api/v1/agent/bots/67e89abc123def456789abcd/twilio" ``` ## Spoken message ### `POST /api/v1/agent/bots/{id}/messages/audio` Transcribes one recording (`wav` or `mp3`, at most 16 MiB and 600 seconds) and queues the transcript as a message to the Bot, like [`POST /api/v1/agent/bots/{id}/messages`](/api/agent/bots/#send-messages-and-follow-logs). Audio with no speech in it is answered `200 {"no_speech": true}` and nothing is queued or billed. With an `Idempotency-Key`, a retry with the same key and audio returns the same `message_id` without transcribing again (and without `transcript`); the same key with other audio, or one used through another door of the Bot (they share one set of client keys), is `422`, and a key starting with `"compat:"` or `"chan:"` is `400`. At most 4 recordings per Bot are transcribed at once (`503 speech_busy` past it); a request that repeats the `Idempotency-Key` of one still running waits for it (`429 speech_busy` when too many wait). `409 bot_deleted` or `bot_reset`: the Bot was deleted or reset before the recording was admitted. ### Request body The body is `multipart/form-data` with a required `file` part and an optional `language` part. | Name | In | Type | Required | Description | |------|-----|------|----------|-------------| | `file` | body | string | Yes | The binary recording: `wav` or `mp3`. | | `language` | body | string | No | The spoken language (for example `en`), when known. | ### Parameters | Name | In | Type | Required | Description | |------|-----|------|----------|-------------| | `id` | path | string | Yes | The bot id. | | `Idempotency-Key` | header | string | No | Opaque retry key (1-255 printable ASCII, no whitespace). A retry with the same key and audio returns the same `message_id` and queues nothing again; the same key with other audio, or one used through another door of the Bot, is `422`; a key starting with `"compat:"` or `"chan:"` is `400`. | | `X-Hoody-Realm` | header | string | No | Per-request realm selector: `"global"` (not tied to a realm) or a 24-hex realm id (also accepted as `?realm=`). | | `realm` | query | string | No | The in:query alias of the `X-Hoody-Realm` header (read when the header is absent or empty). | ### Response Media type: application/json ```json { "no_speech": true } ``` | Field | Type | Required | Description | |-------|------|----------|-------------| | `no_speech` | boolean | Yes | Always `true`. | Media type: application/json ```json { "message_id": "msg-6f1c0e2a-3b9d-4c51-9a7e-2d4b8c1f0a63", "state": "posted", "turn_id": "turn-2c7e4f8b-1a93-4d6e-b012-9c8e3f5a7d21", "transcript": "Hi there, can you summarize the report?" } ``` | Field | Type | Required | Description | |-------|------|----------|-------------| | `message_id` | string | Yes | The message's id; its log row carries it. | | `state` | string | Yes | `posted`: the Bot turn started (`turn_id`). `queued`: it waits for the Bot's current turn to end. `dropped`: the Bot dropped the message before posting it and will never post it; `reason` says why. | | `turn_id` | string | No | On `posted`: the Bot turn the message started. | | `reason` | string | No | On `dropped`: `access_revoked` (the chat-app sender is no longer allowed to send it) or `key_conflict` (the Bot could not deliver it). | | `transcript` | string | No | What was heard. Absent on a retry answered from its `Idempotency-Key`. | Media type: application/json ```json { "code": "bad_request", "message": "invalid request" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `bad_request` | Bad request | The request was malformed or carried invalid parameters, or the model lacks the capability (a speech model for a transcription, or the reverse). | Correct the request body or query parameters, or pick a model with the capability (`GET /api/v1/agent/audio/models?capability=tts` or `stt`). | | `realm_scope_unsupported` | Realm scope unsupported | A per-request scope header this route does not take: a realm or computer header on a route without that dimension, or, on an agent pinned to one realm, a different realm. | Omit the header, or name the realm the agent is pinned to. | | `invalid_realm` | Invalid realm selector | The realm selector is malformed (not `""`/`"global"`/a 24-hex id). | Pass a valid realm selector. | Media type: application/json ```json { "code": "forbidden", "message": "Hoody Kit programs are reached through their URLs only" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `forbidden` | Forbidden (Source IP Guard) | Refused by the Source IP Guard: Hoody Kit programs are reached through their URLs only. | Call the agent through its URL, for example with `hoody agent …`. | Media type: application/json ```json { "code": "not_found", "message": "resource not found" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `not_found` | Not found | The requested resource does not exist. | Verify the path and identifier. | Media type: application/json ```json { "code": "realm_blocked", "message": "realm is blocked" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `realm_blocked` | Realm blocked | The requested realm is blocked or corrupt; the request fails closed before dispatch. | Select a valid, reachable realm. | | `bot_deleted` | Bot deleted | The Bot was deleted while the recording was on its way; nothing was queued. A request for a Bot already gone is the ordinary `404 not_found`. | Do not retry: the Bot is gone. | | `bot_reset` | Bot reset | A reset of the Bot happened before the recording was admitted; nothing was queued. | Send it again if it is still wanted. | | `bot_owned_elsewhere` | Bot served by another agent | Another agent process on this machine serves the Bot (`details.reason` `held`), or its owner lock cannot be taken on this filesystem (`details.reason` `lock_unavailable`). Nothing of the Bot was read or changed. The answer carries `Retry-After: 5`. `details.owner` names the serving process (`pid`, `started_at`) when known, else `null`. | Retry after `Retry-After` seconds, or reach the Bot through the agent that serves it. `GET /api/v1/agent/bots` lists such a Bot as a row with `owned_elsewhere: true`. | Media type: application/json ```json { "code": "payload_too_large", "message": "The upload is larger than 16 MiB." } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `payload_too_large` | Payload too large | The upload is larger than 16 MiB. | Send a recording of at most 16 MiB. | | `audio_too_long` | Audio too long | The audio is longer than the route or the model takes (at most 600 seconds). | Send a shorter recording. | Media type: application/json ```json { "code": "unsupported_audio", "message": "this audio format is not accepted here" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `unsupported_audio` | Unsupported audio | The upload is not an audio file this route takes (`wav` or `mp3`; some models also take `webm`, `ogg`, `m4a` or `flac` on `POST /api/v1/agent/audio/transcriptions`). | Send `wav` or `mp3`. | Media type: application/json ```json { "code": "speech_not_configured", "message": "no speech model is set" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `model_unavailable` | Model unavailable | The speech model is not known or not offered. | Pick a model from `GET /api/v1/agent/audio/models`. | | `speech_not_configured` | Speech not configured | No speech model is set for this capability and none was named. | Name a model, or set a default speech model. | | `provider_refused` | Provider refused | The speech provider refused the input (its content rules or its own limits). | Change the input; `details.reason` says why when the provider said. | | `stt_suspect` | Transcript not trusted | The provider answered a transcript that is too short for the speech in the audio (empty though speech was heard, or only a few words for at least 8 seconds of speech), so it is not used. | Send the audio again, or use another model. | | `idempotency_key_reused` | Idempotency key reused | The `Idempotency-Key` already accepted a different message for this Bot. `details.message_id` names the message it accepted. | Use a fresh `Idempotency-Key` for a different message, or resend the identical message to get its receipt. | Media type: application/json The response carries a `Retry-After` header with seconds to wait before retrying. ```json { "code": "rate_limited", "message": "request rate limit exceeded" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `rate_limited` | Too many requests | The per-client request rate limit was exceeded; the gateway throttled the request before dispatch. | Honor the `Retry-After` header and retry; reduce the request rate. | | `speech_quota_exceeded` | Speech quota exceeded | The speech budget of this realm, Bot or agent is spent for now. | Honor `Retry-After`, or raise the budget. | | `idempotency_keys_exhausted` | Too many keys | The Bot remembers 4096 `Idempotency-Key`s at once; a new keyed message beyond that is refused and nothing is queued. | Send it later, or without a key. | | `speech_busy` | Too many waiting | Too many requests wait on one `Idempotency-Key`, or on the agent's spoken messages as a whole. | Honor `Retry-After` and retry. | Media type: application/json ```json { "code": "internal_error", "message": "internal server error" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `internal_error` | Internal error | An unexpected error occurred while handling the request. | Retry; if persistent, inspect the daemon logs. | Media type: application/json ```json { "code": "provider_error", "message": "the speech provider failed", "details": { "diag_id": "3f9a1c0b7e2d4a65" } } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `provider_auth` | Provider key refused | The speech provider refused the stored key. | Check the provider's key (`PUT /api/v1/agent/providers/{id}/auth/api-key`). | | `provider_error` | Provider error | The speech provider failed, or answered something unusable. `details.diag_id` names the failure in the agent's log. | Retry; if it persists, try another model. | Media type: application/json The response carries a `Retry-After` header with seconds to wait before retrying (busy, or the call cap). ```json { "code": "service_unavailable", "message": "service unavailable" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `service_unavailable` | Service unavailable | The daemon could not service the request (too busy, or a per-client stream concurrency cap was hit). | Honor `Retry-After` and retry. | | `restriction_unknown` | Login restriction unknown | The agent cannot read yet which realms its login may serve (`GET /api/v1/agent/hoody/auth/status` reports restriction `"unknown"`), so it refuses realm-scoped requests until it can. Nothing was read or changed. | Retry after the restriction is readable. | | `speech_unavailable` | Speech unavailable | Speech is not available here: the provider of the selected speech model has no API key (`details.reason` `no_credentials`), or the Hoody AI voice relay cannot serve it (a `details.reason` starting `relay_`, for example not set up or not reachable). | Set a provider key for a speech model. | | `speech_busy` | Speech busy | Too many speech calls are running on the agent. | Honor `Retry-After` and retry. | | `admission_write_failed` | Not saved | The message could not be saved, so it was not queued. | Honor `Retry-After` and send it again with the same `Idempotency-Key`. | Media type: application/json ```json { "code": "provider_timeout", "message": "the speech provider did not answer in time" } ``` | Error Code | Title | Description | Resolution | |------------|-------|-------------|------------| | `provider_timeout` | Provider timeout | The speech provider did not answer in time. | Retry, or send a shorter input. | ### SDK usage ```ts import { HoodyClient } from 'hoody-sdk'; const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' }); import { readFile } from 'node:fs/promises'; const file = await readFile('recording.wav'); const data = new FormData(); data.set('file', new Blob([new Uint8Array(file)], { type: 'audio/wav' }), 'recording.wav'); data.set('language', 'en'); await client.agent.bots.messages.sendAudio('67e89abc123def456789abcd', data, { idempotencyKey: 'spoken-message-001', agentRealm: 'global' }); ``` ```bash curl -X POST \ -F "file=@recording.wav;type=audio/wav" \ -F "language=en" \ "https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com/api/v1/agent/bots/67e89abc123def456789abcd/messages/audio" ```