Skip to content

Beta: these docs are still being updated. The next full update is on 18 November at 8 PM UTC. Docs in beta. Full update 18 Nov, 8 PM UTC.

Hoody.com

A Bot can be spoken to. POST /api/v1/agent/bots/{id}/messages/audio transcribes a recording and queues the transcript as the user’s message. Three WebSocket doors make a voice controller for one Bot at a time: /voice (a custom voice protocol with subprotocol hoody.voice.v1), /realtime (an OpenAI Realtime client), and /twilio (one phone call as a Twilio Media Stream). GET /api/v1/agent/bots/{id}/voice/check runs the same speech readiness check that /voice runs with require, and answers whether the Bot’s speech settings are ready without opening a connection.

In the Hub at hub.hoody.com, open Bots and select the Bot. Open Settings, choose Voice and press Check it works to check readiness. To speak to the Bot, press Talk in its composer, allow microphone access, then use Hold to talk.

Clients learn whether an agent supports /voice/check, speak=false and require from the bot_voice_check capability of GET /version.

The speech and transcription endpoints themselves live on the Audio page; text messages are on the Bot page. The terminal command hoody agent bots voice connect --id <bot> uses the /voice connection. See kit/agent/voice for its flags.

The base URL for every endpoint on this page is the same computer URL as every other agent page:

https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com

Checks whether a Bot can talk by voice before /voice is opened, the same way speech is resolved when it is used: which speech-to-text model and, unless speak=false, which text-to-speech model the Bot would use (its own setting, else the agent’s audio settings), that the model can do that, that the voice and language are ones it offers, and that the agent holds a key for its provider or can use the Hoody AI voice relay. The route answers 200 either way. ready says whether the configuration is ready; it is not a promise that a call to the speech provider will succeed.

The stt and tts stages each carry ok and model; when ok is false, also code, reason, message (what happened and what to do next), retryable and, when known, retry_after_secs and diag_id. tts is null when speak=false. max_utterance_secs is the longest utterance a voice connection takes (60, or the speech-to-text model’s limit when that is lower). When ready is false, diag_id names this check in the agent’s log.

The check bills nothing and takes at most 6 seconds. /voice with require makes the same check before its upgrade.

NameInTypeRequiredDescription
idpathstringYesThe bot id.
in_ratequeryintegerNoInput sample rate: 8000, 16000 (default), 24000 or 48000.
out_ratequeryintegerNoOutput sample rate: 8000, 16000, 24000 (default) or 48000.
in_formatquerystringNoInput encoding: pcm16 (16-bit little-endian, default), pcmu or pcma.
modequerystringNovad (default): the agent detects when the user starts and stops speaking. ptt (push to talk): the client sends ptt.down and ptt.up.
voicequerystringNoThe voice to speak in; omitted, the Bot’s own.
languagequerystringNoThe language the user speaks; omitted, the Bot’s own.
speakquerybooleanNofalse: check speech-to-text only, as for a voice connection opened with speak=false. Default true.
X-Hoody-RealmheaderstringNoPer-request realm selector: "global" (not tied to a realm) or a 24-hex realm id (also accepted as ?realm=).
realmquerystringNoThe in:query alias of the X-Hoody-Realm header (read when the header is absent or empty).

Media type: application/json

{
"diag_id": "9b2e47c01d5f8a36",
"max_utterance_secs": 60,
"ready": false,
"speak": true,
"stt": {
"code": "speech_not_configured",
"language": "auto",
"message": "No speech model is set. Set the Bot's speech.stt_model or speech.tts_model, the agent setting audio.stt_model or audio.tts_model, or set audio.relay_default back to true (the Hoody AI voices).",
"model": "",
"ok": false,
"reason": "not_configured",
"retryable": false
},
"tts": {
"model": "minimax/speech-2.8-turbo",
"ok": true,
"voice": "English_expressive_narrator"
}
}
FieldTypeRequiredDescription
readybooleanYesWhether every checked stage is ready.
speakbooleanYesThe speak value the check ran with.
sttobjectYesThe speech-to-text stage.
stt.okbooleanYesWhether this stage is ready.
stt.modelstringYesThe speech model spec that was checked; empty when none is set, or when the check could not run.
stt.languagestringNoThe language speech-to-text is asked for: a language code, or auto (empty when no language is set) for the model to detect it.
stt.codestringNoWhen ok is false: the error code (speech_not_configured, speech_unavailable, model_unavailable, bad_request, service_unavailable, …).
stt.reasonstringNoWhen ok is false: the machine reason (not_configured, no_credentials, relay_not_configured, unknown_model, not_an_stt_model, voice_not_found, language_not_supported, check_timeout, …).
stt.messagestringNoWhen ok is false: what happened and what to do next.
stt.retryablebooleanNoWhen ok is false: whether checking again later can succeed with nothing changed.
stt.diag_idstringNoWhen known: names the failure in the agent’s log.
stt.retry_after_secsintegerNoWhen known: seconds to wait before trying again.
ttsobjectYesThe text-to-speech stage; null with speak=false.
tts.okbooleanYesWhether this stage is ready.
tts.modelstringYesThe speech model spec that was checked; empty when none is set, or when the check could not run.
tts.voicestringNoThe voice a call would use.
tts.max_input_charsintegerNoWhen ok is true and known: the most characters of text one speech call takes with this model, for any of its languages and voices. Longer text is split before it is sent.
tts.codestringNoWhen ok is false: the error code (speech_not_configured, speech_unavailable, model_unavailable, bad_request, service_unavailable, …).
tts.reasonstringNoWhen ok is false: the machine reason (not_configured, no_credentials, relay_not_configured, unknown_model, not_a_tts_model, voice_not_found, language_not_supported, check_timeout, …).
tts.messagestringNoWhen ok is false: what happened and what to do next.
tts.retryablebooleanNoWhen ok is false: whether checking again later can succeed with nothing changed.
tts.diag_idstringNoWhen known: names the failure in the agent’s log.
tts.retry_after_secsintegerNoWhen known: seconds to wait before trying again.
max_utterance_secsintegerYesThe longest utterance a voice connection takes, in seconds: 60, or the speech-to-text model’s limit when that is lower.
diag_idstringNoWhen ready is false: names this check in the agent’s log.
import { HoodyClient } from 'hoody-sdk';
const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' });
await client.agent.bots.getVoiceCheck('67e89abc123def456789abcd');

Upgrades to a WebSocket with subprotocol hoody.voice.v1 that carries the user’s speech to the Bot and the Bot’s spoken replies back.

Binary frames from the client are input audio: one byte 0x00, a big-endian u32 sequence number, then 20 to 200 ms of audio in in_format at in_rate. A pcm16 frame of an odd byte length is not decoded: it counts as lost, and an utterance it falls in ends with transcript.final rejected frame_loss. Audio sent faster than 1.5 times real time over 2 seconds is also lost. Binary frames from the agent are output audio: one byte 0x01, a big-endian u32 out, a big-endian u32 sample offset, then 16-bit little-endian PCM at out_rate. Text frames are JSON objects with a type.

The client sends input.commit, ptt.down, ptt.up, output.cancel, playback.mark, own_gate.decide, own_gate.answer, text and ping; the agent sends session, vad.start, vad.end, transcript.partial, transcript.final, reply.provisional, reply.invalidate, reply.final, reply.diverged, output.start, output.end, output.stop, cancel.result, gate.announced, own_gate.pending, own_gate.resolved, busy, error, takeover and pong.

mode is fixed for the connection: vad (the default) or ptt. In ptt mode only audio between ptt.down and ptt.up counts; other audio is dropped. One utterance is at most 60 seconds, or the speech-to-text model’s limit when that is lower. A spoken message counts once the Bot’s log shows it.

Only replies to this connection’s own messages are spoken (and replies the Bot gives on its own, with speak_wakes); every reply’s text still arrives in reply.final. reply.final carries own: true when the turn answers a message sent on this connection, false when it does not, and no own when the connection cannot tell. An absent own means unknown, and you can then match turn_id in GET /api/v1/agent/bots/{id}/log. own is about this connection only: a message sent on an earlier connection is answered with own false.

With speak=false no speech is made: replies arrive as text only, every reply.final has spoken false, and nothing is billed for speech synthesis (speech-to-text is still billed); the session frame then shows speak false.

A speech failure is described by an error object {code, reason, message, diag_id, retryable, retry_after_secs}. Its message says what happened and what to do next; diag_id names the failure in the agent’s log. The error object appears:

  • on transcript.final when the transcription failed or was rejected as stt_suspect,
  • on reply.final when reason is tts_failed or tts_incomplete,
  • on an error frame with code service_unavailable when the speech service failed on an announcement (tts_failed or tts_incomplete, once per code and reason per connection).

Voice never answers a delegate’s approval or question: answer it on screen or by typing. The Bot’s own approvals and questions are answered with own_gate.decide and own_gate.answer; their outcome (applied, resolved, stale, unavailable, not_offered, no_pending_gate, stale_gate, gate_cancelled or decision_unconfirmed) arrives in an own_gate.resolved frame, never as an HTTP status.

One connection controls a Bot’s voice at a time: a second one is refused 409 voice_busy unless it asks takeover=true, which closes the first.

With require=stt, require=tts or require=stt,tts the agent first checks those speech settings as GET /api/v1/agent/bots/{id}/voice/check does: when one is not ready it answers 400 bad_request if every failure is a voice or language the speech model does not offer, otherwise 503 speech_unavailable, with the check’s result in details, and takes nothing over. Without require nothing is checked before the upgrade.

NameInTypeRequiredDescription
idpathstringYesThe bot id.
in_ratequeryintegerNoInput sample rate: 8000, 16000 (default), 24000 or 48000.
out_ratequeryintegerNoOutput sample rate: 8000, 16000, 24000 (default) or 48000.
in_formatquerystringNoInput encoding: pcm16 (16-bit little-endian, default), pcmu or pcma.
modequerystringNovad (default): the agent detects when the user starts and stops speaking. ptt (push to talk): the client sends ptt.down and ptt.up.
voicequerystringNoThe voice to speak in; omitted, the Bot’s own.
languagequerystringNoThe language the user speaks; omitted, the Bot’s own.
speak_wakesquerybooleanNotrue: also speak replies the Bot gives on its own; omitted, the Bot’s setting.
takeoverquerybooleanNotrue: take over from the connection that controls the Bot’s voice now.
speakquerybooleanNofalse: replies arrive as text only (reply.final); no speech is made and nothing is billed for speech synthesis. Speech-to-text still runs. Default true.
refusequerystringNoframe: when the agent refuses the connection, accept the upgrade, send one refused frame with the code, message, details and retry_after_secs of the refusal, then close with 4000 plus its HTTP status (4400, 4404, 4409, 4500 or 4503), so a client that cannot read an HTTP refusal of a WebSocket, such as a browser, learns why. Omitted: refusals are HTTP answers.
play_leadquerybooleanNotrue: the agent keeps at most 8 seconds of reply audio ahead of what playback.mark says was played, and a reply whose audio waits 5 seconds with no mark moving on stops speaking (output.stop and reply.final reason no_marks, spoken unknown, or partial when whole segments were heard; the turn goes on). Send playback.mark at least every 500 ms while playing. Omitted or false: reply audio is sent as fast as the connection takes it.
requirequerystringNostt, tts or stt,tts: check those speech settings before the upgrade and refuse it when they are not ready (400 bad_request for a voice or language the model does not offer, else 503 speech_unavailable). tts needs speak=true. Omitted: no check.
X-Hoody-RealmheaderstringNoPer-request realm selector: "global" (not tied to a realm) or a 24-hex realm id (also accepted as ?realm=).
realmquerystringNoThe in:query alias of the X-Hoody-Realm header (read when the header is absent or empty).

The connection closes with:

  • 1000 when the client closes it, after 10 minutes without a frame from the client (the agent’s own output does not count; send ping to stay connected), when another connection takes over, or when the Bot is deleted or suspended;
  • 1001 when the Bot is stopped, as when the agent shuts down;
  • 1002 when the client did not offer hoody.voice.v1;
  • 1008 for a malformed frame, more than 50 control frames a second, an input frame outside 20 to 200 ms, or a playback mark past the audio sent;
  • 1009 for a frame over 64 KiB;
  • 1011 when a write fails, the agent runs out of voice memory, or on an internal error.

A client that does not read gets no close frame: the connection just ends.

Media type: none

The voice connection is open. The 101 response carries no body.

client.agent.bots.connectVoice returns an unopened typed socket that is not an EventEmitter. Register the handlers, then call await socket.connect() and keep the returned socket.

import { HoodyClient } from 'hoody-sdk';
const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' });
const socket = await client.agent.bots.connectVoice('67e89abc123def456789abcd', { mode: 'vad', speak: true });
socket.onTranscriptFinal((m) => console.log('heard', m.utt, m.text));
socket.onReplyFinal((m) => console.log('reply', m.turn_id, m.text));
socket.onError((err) => console.error(err));
await socket.connect();
// send microphone frames with socket.sendInputAudio(seq, pcm); later: socket.close();

Upgrades to a WebSocket that speaks the OpenAI Realtime protocol (GA event names), so a Realtime client such as LiveKit Agents or Pipecat talks to the Bot. The client sends session.update, input_audio_buffer.append, .commit and .clear, conversation.item.create, .delete, .retrieve and .truncate, response.create and response.cancel; the agent answers with the matching server events.

Input audio is audio/pcm (24 kHz, 16-bit little-endian, the default) or audio/pcmu or audio/pcma (8 kHz); output audio follows the session’s output format. With server_vad the agent hears when the user starts and stops speaking and answers on its own; without turn detection the client commits and asks with response.create. semantic_vad runs the same detection; with no silence_duration_ms it ends a turn after 1200 ms of silence for eagerness low, 800 ms for medium or auto, and 500 ms for high.

A conversation.item.create is never undone: a user item reaches the Bot once, by its id. The Bot records every user item id it receives and never forgets one, so a client replaying its history on reconnect does not send an old item again; at the Bot’s limit (realtime_items on GET /api/v1/agent/bots/{id}) a new item is refused and the connection closed.

An event that would change an item already sent (deleting, truncating or editing a user item that waits for a response) closes the connection with 1008; so do a binary or non-JSON frame, a frame over 1 MiB, more than 50 control events a second, and input audio faster than 1.5 times real time. One connection controls a Bot’s voice at a time, over this door, the voice WebSocket or a phone call: another is refused 409 voice_busy unless it asks takeover=true, which closes the one before. A request that is not a WebSocket upgrade is refused 400 bad_request and takes nothing over.

NameInTypeRequiredDescription
idpathstringYesThe bot id.
takeoverquerybooleanNotrue: take over from the connection that controls the Bot’s voice now.
X-Hoody-RealmheaderstringNoPer-request realm selector: "global" (not tied to a realm) or a 24-hex realm id (also accepted as ?realm=).
realmquerystringNoThe in:query alias of the X-Hoody-Realm header (read when the header is absent or empty).

Media type: none

The Realtime connection is open. The 101 response carries no body.

client.agent.bots.connectRealtime returns an unopened typed socket that is not an EventEmitter. Register the handlers, then call await socket.connect() and keep the returned socket.

import { HoodyClient } from 'hoody-sdk';
const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' });
const socket = await client.agent.bots.connectRealtime('67e89abc123def456789abcd');
socket.onSessionCreated(() => console.log('session ready'));
socket.onResponseDone((m) => console.log('response done', m));
socket.onError((err) => console.error(err));
await socket.connect();
// later: socket.close();

The WebSocket URL for a TwiML <Connect> element containing a <Stream> child: one phone call becomes the Bot’s voice controller. Twilio’s stream events are taken as documented (connected, start, media, mark, stop, dtmf); audio is μ-law at 8 kHz, mono.

What the caller says is cut into utterances by voice activity detection; each one is transcribed in order and posted to the Bot as the user’s message. The Bot’s reply comes back as 20 ms media frames, one mark per spoken segment. When the caller speaks over the reply, its playback stops (a clear is sent). A stop event, or the call ending, drops the utterance still being spoken: it is never transcribed or posted; utterances already complete are still posted. DTMF digits are ignored. Gates are never announced or answered by voice.

The call takes over from another open voice controller only when the stream’s start event carries the custom parameter takeover=true (or the URL has ?takeover=true); otherwise a second controller is refused and the stream closes with code 1008.

NameInTypeRequiredDescription
idpathstringYesThe bot id.
takeoverquerybooleanNotrue: take over from the Bot’s open voice controller.
X-Hoody-RealmheaderstringNoPer-request realm selector: "global" (not tied to a realm) or a 24-hex realm id (also accepted as ?realm=).
realmquerystringNoThe in:query alias of the X-Hoody-Realm header (read when the header is absent or empty).

Media type: none

The WebSocket upgrade is accepted; the connection carries Twilio Media Streams frames. The 101 response carries no body.

Twilio normally opens this connection; the SDK form is for tests. client.agent.bots.connectTwilio returns an unopened typed socket that is not an EventEmitter. Register the handlers, then call await socket.connect() and keep the returned socket.

import { HoodyClient } from 'hoody-sdk';
const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' });
const socket = await client.agent.bots.connectTwilio('67e89abc123def456789abcd');
socket.onMedia((m) => console.log('media', m));
socket.onMark((m) => console.log('mark', m));
socket.onError((err) => console.error(err));
await socket.connect();
// later: socket.close();

POST /api/v1/agent/bots/{id}/messages/audio

Section titled “POST /api/v1/agent/bots/{id}/messages/audio”

Transcribes one recording (wav or mp3, at most 16 MiB and 600 seconds) and queues the transcript as a message to the Bot, like POST /api/v1/agent/bots/{id}/messages. Audio with no speech in it is answered 200 {"no_speech": true} and nothing is queued or billed.

With an Idempotency-Key, a retry with the same key and audio returns the same message_id without transcribing again (and without transcript); the same key with other audio, or one used through another door of the Bot (they share one set of client keys), is 422, and a key starting with "compat:" or "chan:" is 400. At most 4 recordings per Bot are transcribed at once (503 speech_busy past it); a request that repeats the Idempotency-Key of one still running waits for it (429 speech_busy when too many wait). 409 bot_deleted or bot_reset: the Bot was deleted or reset before the recording was admitted.

The body is multipart/form-data with a required file part and an optional language part.

NameInTypeRequiredDescription
filebodystringYesThe binary recording: wav or mp3.
languagebodystringNoThe spoken language (for example en), when known.
NameInTypeRequiredDescription
idpathstringYesThe bot id.
Idempotency-KeyheaderstringNoOpaque retry key (1-255 printable ASCII, no whitespace). A retry with the same key and audio returns the same message_id and queues nothing again; the same key with other audio, or one used through another door of the Bot, is 422; a key starting with "compat:" or "chan:" is 400.
X-Hoody-RealmheaderstringNoPer-request realm selector: "global" (not tied to a realm) or a 24-hex realm id (also accepted as ?realm=).
realmquerystringNoThe in:query alias of the X-Hoody-Realm header (read when the header is absent or empty).

Media type: application/json

{
"no_speech": true
}
FieldTypeRequiredDescription
no_speechbooleanYesAlways true.
import { HoodyClient } from 'hoody-sdk';
const client = new HoodyClient({ baseURL: 'https://{projectId}-{computerId}-agent-1.{server}.computers.hoody.com' });
import { readFile } from 'node:fs/promises';
const file = await readFile('recording.wav');
const data = new FormData();
data.set('file', new Blob([new Uint8Array(file)], { type: 'audio/wav' }), 'recording.wav');
data.set('language', 'en');
await client.agent.bots.messages.sendAudio('67e89abc123def456789abcd', data, { idempotencyKey: 'spoken-message-001', agentRealm: 'global' });