# hoody-display Subskill ## Overview ### Purpose `hoody-display` provides fully embedded desktop environments for GUI applications accessible via URL. It enables multiplayer, remote-controlled graphical interfaces where agents and users can interact with windows, screenshots, keyboard, and mouse inputs programmatically. ### When to Use - Capturing screenshots or thumbnails of running GUI applications - Automating mouse clicks, drags, and keyboard input in desktop apps - Managing windows (focus, move, resize, minimize, close) - Reading and writing clipboard contents - Building visual verification workflows with screenshot comparisons - Creating multiplayer desktop experiences where multiple agents share access ### Philosophy Fit Hoody-display embodies the core philosophy of fully embeddable, multiplayer desktop environments. Every GUI operation is exposed as a CLI command, enabling deterministic automation. The HTML5 Display client is accessible via URL, making desktop environments shareable and collaborative. ### Prerequisites - A running Hoody container with a display-enabled service - Container ID obtained from `hoody container list` or creation - Authentication via `hoody auth login`, `--token`, or `HOODY_TOKEN` env var ### Key Constraint Every command targeting a specific container requires `-c ` or the `HOODY_CONTAINER` environment variable. --- ## Common Workflows ### Workflow 1: Health Check and Service Verification Verify the display service is operational before performing any operations. ``` # Check service health hoody display health -c ``` Expected response structure: ``` { "status": "healthy", "service": "display", "version": "1.0.0", "uptime": 3600, "timestamp": "2025-01-15T10:30:00Z" } ``` ``` # Get display information including available screenshots hoody display info -c -o json ``` ### Workflow 2: Capture and Retrieve Screenshots Capture fresh screenshots or retrieve previously captured ones. ``` # Capture a new screenshot (returns image file) hoody display capture -c # Capture screenshot metadata only (no image download) hoody display capture-metadata -c -o json ``` Expected metadata response: ``` { "timestamp": 1705312200, "width": 1920, "height": 1080, "format": "png", "size": 245760 } ``` ``` # Retrieve the most recent screenshot hoody display latest -c # Get metadata for the most recent screenshot hoody display latest-metadata -c -o json # Retrieve a screenshot by specific timestamp hoody display by-timestamp 1705312200 -c # List all available screenshots hoody display list -c -o json ``` ### Workflow 3: Capture and Use Thumbnails Thumbnails are 320x180 scaled versions useful for visual verification without large downloads. ``` # Capture a new thumbnail hoody display capture -c # Retrieve the most recent thumbnail hoody display latest -c # Retrieve a thumbnail by timestamp hoody display by-timestamp 1705312200 -c ``` ### Workflow 4: Mouse Operations Perform precise mouse interactions with the display. ``` # Move cursor to absolute position hoody display move -c --x 500 --y 300 # Move cursor by relative offset hoody display move-relative -c --x 10 --y -20 # Get current cursor position hoody display location -c -o json ``` Expected location response: ``` { "x": 500, "y": 300 } ``` ``` # Single click at current position hoody display click -c # Double-click at current position hoody display double-click -c # Press and hold mouse button hoody display down -c # Release mouse button hoody display up -c # Scroll in a direction hoody display scroll -c --direction down ``` ### Workflow 5: Keyboard Operations Type text and send key combinations. ``` # Type a string of text hoody display type -c --text "Hello World" # Press key combination hoody display key -c --keys 'ctrl+c' # Press multiple key combinations hoody display key -c --keys 'ctrl+a' 'ctrl+c' # Hold a key down hoody display key-down -c --key Shift_L # Release a held key hoody display key-up -c --key Shift_L ``` ### Workflow 6: Clipboard Operations Read and write clipboard contents. ``` # Read clipboard text hoody display get -c -o json ``` Expected response: ``` { "text": "clipboard contents here" } ``` ``` # Write text to clipboard hoody display set -c --text "New clipboard content" ``` ### Workflow 7: Window Management List, focus, move, and manipulate windows. ``` # List all windows hoody display list -c -o json ``` Expected response: ``` [ { "windowId": "0x0400001", "name": "Text Editor", "geometry": { "x": 0, "y": 0, "width": 1024, "height": 768 } }, { "windowId": "0x0400002", "name": "Terminal", "geometry": { "x": 100, "y": 100, "width": 800, "height": 600 } } ] ``` ``` # Get the active window hoody display active -c -o json # Focus a specific window hoody display focus -c --window-id 0x0400001 # Move a window to new position hoody display move -c --window-id 0x0400001 --x 200 --y 150 # Resize a window hoody display resize -c --window-id 0x0400001 --width 1200 --height 900 # Minimize a window hoody display minimize -c --window-id 0x0400001 # Raise window to top hoody display raise -c --window-id 0x0400001 # Close a window hoody display close -c --window-id 0x0400001 # Search for windows by pattern hoody display search -c --pattern "Editor" # Get window properties hoody display properties 0x0400001 -c -o json # Get window geometry hoody display geometry 0x0400001 -c -o json # Get window title hoody display name 0x0400001 -c -o json ``` --- ## Advanced Operations ### Composite Action: Click-At Move cursor to a position and click in a single operation. ``` # Move to position (500, 300) and click hoody display click-at -c --x 500 --y 300 ``` ### Composite Action: Type-At Move cursor, click, and type text in one atomic operation. Ideal for filling form fields. ``` # Move to (200, 100), click, then type text hoody display type-at -c --x 200 --y 100 --text "username@example.com" ``` ### Composite Action: Drag Drag from one position to another. ``` # Drag from (100, 100) to (500, 300) hoody display drag -c --start-x 100 --start-y 100 --end-x 500 --end-y 300 ``` ### Composite Action: Select Select a range using click followed by shift-click. ``` # Select from (100, 50) to (400, 50) hoody display select -c --start-x 100 --start-y 50 --end-x 400 --end-y 50 ``` ### Action Sequences with Batch Execute a sequence of multiple actions in a single command. ``` # Execute a batch of actions hoody display batch -c --actions '[ {"type": "move", "x": 500, "y": 300}, {"type": "click"}, {"type": "type", "text": "Hello"}, {"type": "key", "keys": ["Return"]} ]' ``` ### Single Action with Optional Screenshot Execute one action and optionally capture a screenshot for verification. ``` # Perform action and capture screenshot hoody display act -c --action '{"type": "click", "x": 500, "y": 300}' --screenshot ``` ### Wait with Optional Screenshot Wait for a specified duration, optionally capturing a screenshot at the end. ``` # Wait 2 seconds and capture screenshot hoody display wait -c --duration 2000 --screenshot ``` ### Emergency Reset Release all held inputs when something goes wrong. Use this for error recovery. ``` # Emergency release all inputs hoody display reset -c ``` ### Get Display Dimensions ``` # Get display geometry hoody display geometry -c -o json ``` Expected response: ``` { "width": 1920, "height": 1080 } ``` ### Access HTML5 Display Client Open the web-based display client for visual interaction. ``` # Access the display client URL hoody display access -c -o json ``` The returned URL follows the Hoody proxy routing pattern: `https://{projectId}-{containerId}-display-{serviceId}.{node}.containers.hoody.com` ### Error Recovery Pattern When automation gets stuck, follow this recovery sequence: ``` # 1. Reset all inputs hoody display reset -c # 2. Verify service is healthy hoody display health -c # 3. Capture screenshot to see current state hoody display capture -c # 4. Identify active window hoody display active -c -o json ``` ### Performance Considerations - Use thumbnails instead of full screenshots for frequent visual checks - Use `capture-metadata` when you only need screenshot dimensions/timing - Prefer `click-at` and `type-at` composite actions over multiple separate commands - Use `batch` for sequences of related actions to reduce round trips - The `wait` action with screenshot combines delay and verification efficiently - Retrieve `latest` or `latest-metadata` instead of capturing new screenshots when the display state has not changed --- ## Quick Reference ### Essential Commands | Operation | Command | |-----------|---------| | Health check | `hoody display health -c ` | | Screenshot | `hoody display capture -c ` | | Latest screenshot | `hoody display latest -c ` | | List windows | `hoody display list -c -o json` | | Click at position | `hoody display click-at -c --x X --y Y` | | Type text | `hoody display type -c --text "text"` | | Key combo | `hoody display key -c --keys 'ctrl+c'` | | Read clipboard | `hoody display get -c -o json` | | Focus window | `hoody display focus -c --window-id WID` | | Reset inputs | `hoody display reset -c ` | ### Common Flags | Flag | Purpose | |------|---------| | `-c ` | Target container (required for all commands) | | `-o json` | Machine-readable JSON output | | `-t ` | API authentication token | ### Response Formats **Health**: status, service, version, uptime, timestamp **Screenshot metadata**: timestamp, width, height, format, size **Window list**: Array of windowId, name, geometry objects **Mouse location**: x, y coordinates **Clipboard**: text string ### Schema-Required Fields | Endpoint | Required Fields | |----------|-----------------| | `mouse/move` | `x: integer`, `y: integer` | | `mouse/move-relative` | `x: integer`, `y: integer` | | `mouse/scroll` | `direction: string` | | `keyboard/type` | `text: string` | | `keyboard/key` | `keys: array` | | `keyboard/key-down` | `key: string` | | `keyboard/key-up` | `key: string` | | `clipboard` (POST) | `text: string` | | `window/focus` | `windowId: string` | | `window/move` | `windowId: string`, `x: integer`, `y: integer` | | `screenshot/{timestamp}` | `timestamp: string` (path parameter) | | `thumbnail/{timestamp}` | `timestamp: string` (path parameter) | | `window/{windowId}/properties` | `windowId: string` (path parameter) |