<!-- source: openspec/current/robot-operator.md -->

---
title: Robot operator surface platform
status: proposed
owner: pavel
audience: [pavel, ai, engineer]
tldr: Three-layer architecture for bespoke robot operator surfaces — Gateway adapters, branded UI Shell, AI-Companion layer. Project engagements, not SaaS. Deferred pending pilot evidence.
---

# Robot Operator Surface Platform

> Reusable three-layer architecture for bespoke robot operator surfaces: Gateway adapters, branded UI Shell templates, and an AI-Companion narrative layer. Delivered as project engagements, not a SaaS product.

---

## Status

**Deferred — architecture defined, no implementation in flight.** Phase 1 (this spec) and Phase 2 (Foxglove panel kit scaffold) landed in RAP-1151. Phase 3 (outreach), Phase 4 (demo stand), and Phase 5 (archive) depend on hardware / pilot evidence.

Research foundation: [`openspec/_methodology/research/robot-operator-interfaces.md`](../_methodology/research/robot-operator-interfaces.md) and [`openspec/_methodology/research/robot-operator-icp-shortlist.md`](../_methodology/research/robot-operator-icp-shortlist.md).

---

## Goals

1. Define a reusable three-layer architecture the studio can quote and deliver against without re-deriving it per pilot.
2. Enumerate the integration transports the Gateway layer must support to cover ~70% of plausible pilot inquiries (six MVP transports; six on-demand).
3. Lock the seven UI template names and their interaction contracts, so quotes reference a known catalogue rather than a hand-rolled list each time.
4. Place the AI-Companion narrative layer as the defensible differentiator — the layer Foxglove / Formant / InOrbit / Encord explicitly do not build — wired to Muse's existing narrative APIs.
5. Establish package boundaries (`packages/operator-shell/`) consistent with the studio's existing boundary model (`canvas`, `muse`, `forge`).

---

## Three-layer architecture

```
                ┌──────────────────────────────────────┐
                │  Layer 3 — AI-Companion (narrative)  │
                │  - explains intent in human terms    │
                │  - proposes interventions            │
                │  - generates post-mortems            │
                │  Backed by: @rapoport-studio/muse    │
                └──────────────────┬───────────────────┘
                                   │ subscribes to topics + state
                ┌──────────────────▼───────────────────┐
                │  Layer 2 — UI Shell (templates)      │
                │  Seven templates listed in § Templates│
                │  React 19 + Tailwind + linkable-ui   │
                │  Backed by: packages/operator-shell  │
                └──────────────────┬───────────────────┘
                                   │ topic / state / command interface
                ┌──────────────────▼───────────────────┐
                │  Layer 1 — Gateway (I/O)             │
                │  Adapters per transport (§ Transport) │
                │  WebSocket bridge + state cache      │
                │  Backed by: packages/operator-shell  │
                │  (gateway subpath) or per-pilot      │
                │  custom service on Cloudflare Workers│
                └──────────────────┬───────────────────┘
                                   │ rosbridge / Foxglove WS /
                                   │ VDA 5050 / MAVLink / gRPC / MQTT
                ┌──────────────────▼───────────────────┐
                │  Robot / fleet                       │
                └──────────────────────────────────────┘
```

Each layer is independently versionable. A pilot may choose to use only layers 1+2 (no AI-companion) or only layer 3 (companion as a panel inside Foxglove). The studio's headline product is "all three layers, branded for the OEM."

The transport adapters expose a unified internal shape: `{ topics: TopicSubscription[], commands: CommandClient, state: StateSnapshot }`. Templates consume this shape regardless of underlying transport — swapping ROS 2 for MAVLink is a Gateway-config change, not a template rewrite.

### Deployment shapes

- **Cloud-hosted pilot** — Cloudflare Worker bridges WebSocket → backend; suitable for read-mostly operator dashboards over the public internet.
- **On-prem pilot** — adapter runs as a Node service on the OEM's internal network, with a WebSocket up-link to the studio-hosted UI Shell.
- **Hybrid** — Gateway lives at OEM, AI-Companion runs in studio cloud, UI Shell loads in operator's browser.

---

## Transport matrix

Research scan covered 12 transports. Studio MVP must speak six; the remaining six are "on demand" — added when a pilot requires them. Source: [`robot-operator-interfaces.md`](../_methodology/research/robot-operator-interfaces.md) § 3.

### MVP (six transports — must speak from day one)

| Transport | When used | Web compatibility | Maturity |
|---|---|---|---|
| **Foxglove Bridge (C++ WS) + MCAP recordings** | Modern ROS 2 robots; the studio default. Outperforms rosbridge. | First-class WebSocket + JSON + Protobuf + ROS 2 .msg/.idl | Mature, rising |
| **VDA 5050 v3.0 over MQTT** | AMR / AGV fleet operators (logistics OEM). Critical for ICP B. | Browser via MQTT-over-WebSocket (HiveMQ / EMQX) | Industry standard, v3.0 published March 2026 |
| **MAVLink** | Drones — Parrot Anafi, ArduPilot/PX4 stack, civilian operators | UDP/TCP plus WebSocket tunnels | Very mature (since 2009) |
| **DJI Cloud API (MQTT + HTTPS + WS)** | DJI Matrice + Dock customers wanting a branded alternative to FlightHub 2 | Native browser | Mature; v3+ in 2026 |
| **Spot SDK (gRPC over TLS)** | Boston Dynamics Spot pilots | Needs gRPC-Web proxy or BFF | Mature, well-documented |
| **HTTPS REST adapter (URScript REST, ABB Robot Web Services, ANYmal API)** | Cobots and quadrupeds with vendor REST surfaces; one studio adapter wraps all three with per-vendor schemas | Native browser | Each mature |

### On-demand (add per pilot)

| Transport | When triggered |
|---|---|
| **rosbridge_suite (JSON over WS)** | Legacy ROS 1 deployments. Foxglove Bridge is preferred for any new project. |
| **KUKA KRL / KUKA System Software** | KUKA cobot customer specifically. |
| **Anduril Lattice SDK** | Defense-adjacent civilian pilots — only after EU/US entity is established. |
| **OPC UA** | When operator surface must talk to MES / SCADA on the factory floor. |
| **Generic MQTT (non-VDA 5050)** | IoT-fleet telemetry from service / cleaning robots not bound by VDA 5050. |
| **CycloneDDS direct (Unitree native middleware)** | Only if the pilot bypasses the SDK wrappers — usually not needed. |

---

## Seven UI templates

| # | Template | Primary user | Core interaction |
|---|---|---|---|
| 1 | **Operator Console** | Remote operator handling autonomy-fallback | Map + video + telemetry overlay + one-click takeover + intent narration (Layer 3) |
| 2 | **Fleet Dashboard** | Fleet manager / mid-tier ops | Robot grid, status badges, alerts queue, KPIs |
| 3 | **Task / Mission Scheduler** | Business-side ops user | Drag-and-drop missions, recurrence, recharge windows; vertical-specific templates |
| 4 | **Teleop Pad** | Operator during full manual mode | Gamepad / browser-keyboard control + latency-aware video + safety interlocks |
| 5 | **Skill Composer** | Robot programmer / advanced operator | Chain primitives (pickup → orient → place); apps-for-robots model |
| 6 | **Telemetry Replay + AI Post-mortem** | Post-incident analyst, support | MCAP / bag replay scrubber + LLM-generated incident summary (Layer 3) |
| 7 | **Approval / Handoff UI** | Operator supervising autonomous behaviour | LLM proposes intent → operator approves or overrides; logs every decision for training data |

Templates 1, 6, and 7 are the studio's **lead offerings** — least overlap with Foxglove, most use of Layer 3, highest defensibility. Templates 2–5 are commodity-adjacent and serve as up-sells once a pilot is in flight.

### Template contract

Each template is a React 19 component exported from `packages/operator-shell/templates/<name>/`. Contract:

- **Props**: `{ gateway: GatewayClient, companion?: CompanionClient, theme: OperatorTheme, locale: Locale }`.
- **State**: URL-driven via the studio's linkable-ui system (`openspec/current/linkable-ui.md`). Open dialogs, selected robot, panel layout — all reachable by URL.
- **Theming**: OEM brand tokens via `OperatorTheme`. Layered on top of `@rapoport-studio/ui` tokens; the OEM can override without forking.
- **A11y**: keyboard-first interaction. Operators wear gloves, work tired, look at multi-monitor. No mouse-only paths.

---

## AI-Companion narrative API

Layer 3 is **not** a new package. It is a read-only subpath off `@rapoport-studio/muse`: `@rapoport-studio/muse/companion`. Exports:

- `narrate(stateSnapshot): Promise<NarrationStream>` — explains current robot state in 1–2 sentences, streams updates as state changes.
- `propose(stateSnapshot, capabilities): Promise<InterventionProposal[]>` — suggests operator actions; each proposal carries a confidence score and a rationale.
- `postMortem(replaySnapshot): Promise<PostMortemDoc>` — generates a structured incident report from a telemetry replay window.

The `muse/companion` subpath is added to the boundary table in `CLAUDE.md` § Package boundaries as a Muse carve-out (after `muse/mirror` and `forge/core`). It uses the same `ANTHROPIC_API_KEY` secret and the same Muse runtime — no new auth path.

### Hard constraint — retrieval-only, no free-form generation

The companion is **retrieval-only over structured state**. It does not generate claims; it summarises events that exist in the studio's structured event store (Supabase append-only `operator_events` table, populated by Gateway adapters in real time). Every clause emitted by the companion must carry a citation — `{ event_id, timestamp }` references — that the operator UI renders as inline anchors clickable back to the underlying telemetry.

Failure mode this prevents: "Robot 12 finished the medication round" when it actually got stuck halfway. The companion has no path to assert "finished" unless an `event:round_complete` row exists with the matching robot ID and timestamp.

This is a **constraint, not a feature flag**. The companion runtime refuses to emit text without at least one event citation per sentence. Implementation: a post-LLM validator pass that strips uncited claims before display, plus a system prompt that pre-loads the constraint and the event-store schema.

Source: [`robot-operator-interfaces.md`](../_methodology/research/robot-operator-interfaces.md) § 6, risk R5 (AI companion trust calibration).

---

## Package boundaries

New row in `CLAUDE.md` boundary table:

| Package | May import |
|---|---|
| `@rapoport-studio/operator-shell` (deferred) | `@rapoport-studio/ui`, `@rapoport-studio/muse/companion`, transport SDKs (`roslibjs`, `@foxglove/ws-protocol`, `mqtt`, `mavlink2rest-client`, `bosdyn-grpc-web`) |

Forbidden imports: `db`, `forge`, `apps/web`, `apps/studio`. The shell is consumed **by** apps; it does not consume them.

The `muse/companion` carve-out follows the existing pattern (`forge/core` is consumed read-only by `apps/studio` and `packages/muse`). Boundary enforcement via ESLint `no-restricted-imports` lands when the package goes active, deferred until first pilot signs (per design § 7).

---

## ICP segments

Three named segments, each addressable by the studio with its current stack and team size. Full outreach short-list: [`robot-operator-icp-shortlist.md`](../_methodology/research/robot-operator-icp-shortlist.md).

### ICP A — Pre-Series-B humanoid startups (EU / IL)

- **Buyer**: CTO or VP Engineering at a humanoid-robot startup, founded 2022–2025, headcount 15–60.
- **Pain**: operator companion UI for pilot deployments at customer sites. They have ROS 2 / proprietary stack, no UX team.
- **Price band**: $80k–$200k for Operator Console + Post-mortem template, 10–14 week engagement.
- **Studio fit**: AI-companion layer differentiates against "we built it in-house with React"; bespoke branding plus reusable Operator Console template gives studio margin.

### ICP B — AMR fleet operators in logistics

- **Buyer**: Head of Robotics Ops at a 3PL or distribution-center operator, OR an AMR vendor selling to such operators.
- **Pain**: VDA 5050-compliant fleet dashboard, branded for the operator (their floor manager sees their logo, not the vendor's).
- **Price band**: $50k–$120k for Fleet Dashboard + Task Scheduler, 8–10 weeks.
- **Studio fit**: Cloudflare Workers + Supabase fits warehouse-scale fleets at low op-ex; VDA 5050 standard means adapter once, deploy many.

### ICP C — Vertical-specific OEMs (agro, cleaning, medical)

- **Buyer**: VP Product at a single-purpose robotics OEM (e.g., autonomous orchard sprayer, hospital service robot, commercial cleaning fleet).
- **Pain**: custom operator surface that speaks their vertical's vocabulary (rows / wards / floors), branded for their customer.
- **Price band**: $30k–$80k for Operator Console + Telemetry Replay, 6–8 weeks per vertical.
- **Studio fit**: studio's product-shaping muscle (verticalised content, branded surfaces) transfers cleanly.

Geographic anchor: EU + Israel + UK first. North America and APAC deferred until the studio has two delivered pilots as references.

Named first-wave outreach targets (by operator-UI gap signal): **ARX Robotics, Stark, PHINXT Robotics, Energy Robotics, Capra Robotics**. Honest gaps: Israel under-represented (defense robotics jumped to mega-rounds; Series-A-band IL robotics are largely software-only); France has no Seed/A-stage humanoid that surfaced.

---

## Open questions

These do not block this spec but must be resolved before the first pilot ships. OQ-1 and OQ-3 closed during Phase 0 research; the remaining six carry forward.

| # | Question | Status |
|---|---|---|
| OQ-1 | Demo hardware decision | **Closed.** Buy a single Unitree Go2 ($1.6k–$3k) as known-good test rig; Gazebo + ROS 2 for everything else; pilots provide remote access or recorded MCAP files. |
| OQ-2 | Does `muse/companion` need streaming output to feel responsive in Operator Console (Template 1)? | Open — benchmarks needed once a real telemetry stream is wired. |
| OQ-3 | Operator-action audit log | **Closed.** Reuse the studio's existing audit substrate from RAP-828 (append-only Supabase event store + operator-facing review surface). |
| OQ-4 | AI-companion fallback when Anthropic connection is degraded | Open — fallback to last-good-narration vs "companion offline" indicator. UX call. |
| OQ-5 | Pricing for Israel-headquartered pilots (Israeli VAT + contracting) | Open — defer to legal once the first IL lead surfaces. |
| OQ-6 | Productize vs stay services-led | Open — defer until 5 closed pilots; until then services-led with substrate harvest. |
| OQ-7 | Defense-adjacent exposure | Open — decline defense pilots until studio has an EU/US entity with the right clearances; pursue civilian critical-infrastructure instead. |
| OQ-8 | Unitree geopolitical dependency | Open — build the substrate Unitree-friendly but not Unitree-locked; ANYbotics / Spot / Apptronik are the path-of-escape. |
