OrbitPilot

Documentation

How to use OrbitPilot

Getting started, use cases, Agent Skills, MCP, mobile pairing, and API integration.

Back to home

How to use OrbitPilot

OrbitPilot is a cloud control plane that connects AI agents to registered Android devices. Agents send goals and structured automation steps; the cloud schedules work, enforces safety policies, and coordinates AutoPilot on each device to perform taps, text entry, scrolling, and screenshots.

Getting started

  1. Create an account at /register or sign in at /login.
  2. Register a device in the console under Devices. Copy the one-time device token (shown only once).
  3. Install AutoPilot on your Android phone (see Mobile (AutoPilot) below).
  4. Create an API key under Settings -> API Keys with the scopes your agent needs.
  5. Connect your agent via SKILLS.md, MCP, or direct HTTP calls to POST /v1/tasks.

The console (Dashboard) is the operator surface for devices, tasks, schedules, confirmations, and API keys. External agents own planning; OrbitPilot owns execution, scheduling, observability, and guardrails.

Use cases

ScenarioWhat happens
Immediate runAn agent submits a task; OrbitPilot picks an online device; AutoPilot executes steps; screenshots and status stream back; the agent receives completion or a confirmation prompt.
Scheduled jobYou persist a schedule (cron or interval); the worker wakes the job, allocates a device, runs with logging, and surfaces failures via the console or webhooks.
Human confirmationBefore high-risk actions (public posts, DMs, etc.), the task pauses; operators approve or reject from the console; execution resumes or stops.
Ops and monitoringOperators use the Dashboard for device inventory, task timelines, screenshot history, and pending confirmations without writing code.

MVP focuses on Android first. Payment and purchase flows are blocked by policy.

Dashboard console

After sign-in, open /app/overview for a summary of devices and tasks.

AreaPurpose
DevicesRegister phones, view online/busy state, revoke tokens
TasksCreate, inspect, and cancel automation runs
SchedulesManage recurring or one-shot scheduled jobs
ConfirmationsApprove or reject paused high-risk steps
SettingsAPI keys, webhooks, profile

The UI supports English and Chinese (locale switcher in the header) and light / dark themes.

Agent Skills

Skills are markdown instructions bundled for AI clients (Cursor, Claude Code, Codex, etc.). OrbitPilot publishes:

  • Repository root SKILLS.md — full HTTP and WebSocket contract for agents
  • .agents/skills/orbitpilot-api/ — focused integration skill for API keys, tasks, devices, and webhooks

Point your agent at SKILLS.md when it needs to list devices, create tasks, handle webhooks, or pair AutoPilot over WebSocket (/ws).

Typical flow for an external planner:

  1. Read device list (GET /v1/devices) with devices:read
  2. Submit structured steps (POST /v1/tasks) with tasks:write
  3. Poll task status or subscribe via webhooks
  4. Handle confirmation_required events in the console or via webhook payload

Request minimum scopes on each API key. Dashboard summary requires both devices:read and tasks:read.

MCP server

The @orbitpilot/mcp-server package exposes OrbitPilot HTTP APIs as MCP tools (prefixed orbitpilot_, e.g. orbitpilot_list_devices, orbitpilot_create_task). Use MCP when your client (Cursor, VS Code, Windsurf) speaks Model Context Protocol instead of raw REST.

Modes

ModeWhen to useAuthentication
stdioLocal development; Cursor spawns a subprocessORBITPILOT_API_KEY in server env
HTTPShared or production MCP behind HTTPSPer-request Authorization: Bearer opk_... or X-Api-Key

Local stdio (from repo root):

ORBITPILOT_API_KEY=opk_xxx pnpm mcp:dev

Local HTTP:

pnpm mcp:dev:http
# Default: http://127.0.0.1:5151

Cursor (remote HTTP) — in ~/.cursor/mcp.json or project .cursor/mcp.json:

{
  "mcpServers": {
    "orbitpilot": {
      "url": "https://mcp.example.com",
      "headers": {
        "Authorization": "Bearer opk_your_key_here"
      }
    }
  }
}

Production MCP listens on 127.0.0.1:5151 inside the stack; place nginx (or similar) in front and set MCP_PUBLIC_URL. See packages/mcp-server/README.md for VS Code, Windsurf, and Docker notes.

Mobile (AutoPilot)

AutoPilot is the Android app that maintains a WebSocket to OrbitPilot Cloud and executes UI actions through the accessibility service.

Requirements

  • Android 8.0+ (physical device or emulator)
  • Device registered in the console with a valid device token
  • OrbitPilot accessibility service enabled when prompted
  • Notifications allowed on Android 13+ so the foreground session can stay alive

Install and pair

  1. Download the APK from the landing page (Android app section) when NEXT_PUBLIC_AUTOPILOT_APK_URL is configured, or build from apps/autopilot-android.
  2. Open AutoPilot and enter your WebSocket URL (e.g. wss://api.example.com/ws) and device token.
  3. Confirm the device appears online in Devices in the console.

AutoPilot reports heartbeat and busy/idle state so schedulers do not overload a single phone.

HTTP API and webhooks

  • All REST routes live under /v1 (e.g. POST /v1/tasks, GET /v1/devices).
  • Machine auth uses API keys (opk_...) via Authorization: Bearer or X-Api-Key.
  • Browser session auth uses cookies from POST /v1/auth/login.

Webhooks: configure endpoints in Settings -> Webhooks. Verify signatures with HMAC over the exact JSON body using your webhook secret and headers documented in SKILLS.md. Never log raw opk_ keys, device_token, or webhook secrets.

OpenAPI/Swagger is available on the API host at /v1/docs when enabled in your deployment.

Security practices

  • Treat device tokens and API keys as one-time or rotatable secrets.
  • Use human confirmation for publishing and outbound messages.
  • Block payment-related actions in MVP; review orbit_risk on task steps.
  • Grant the smallest API key scope set required for each integration.

Next steps