Crobot Architecture and System Guide

Source-review edition based on the September 15, 2026 task workspace and available repository snapshot. Includes external factual corrections beyond PR #175. Production configuration and correspondence to the full private GitHub diff were not independently verified.

Crobot is Coframe's autonomous coding platform. It receives development tasks from GitHub, Slack, a web dashboard, or external orchestrators via a REST API. Each task executes within its own hardened, isolated sandbox running on Google Kubernetes Engine (GKE) with gVisor (runsc). Inside the sandbox, an autonomous agent inspects code, runs local tests and builds, verifies visual changes using a headless browser, commits focused adjustments, pushes a git branch, and opens a GitHub Pull Request.

This document describes the current architecture of Crobot, detailing how components interact, where state lives, how authentication and trust boundaries are enforced, how the task lifecycle progresses from request to pull request, and how external agents can automate and monitor tasks programmatically.

Table of Contents

1. System Architecture Overview
2. Component Responsibilities
   • Entry Points and Ingress
   • Gateway Control Plane
   • Execution Sandboxes
   • Persistence and Storage
   • External Services
3. Architecture Diagram
4. Task Lifecycle and Execution Flow
   • Persisted Task Statuses
   • End-to-End Lifecycle Sequence
5. Task Identity, Sessions, and State Persistence
   • Task IDs vs. Session IDs
   • Sub-Agent Hierarchy
   • Persistence Across Sleep, Release, and Revival
6. Authentication, Security, and Trust Boundaries
   • Ingress Authentication and Scoping
   • Sandbox Hardening and Isolation
   • Credential Delegation and Token Isolation
7. External-Agent API Workflow (/api/v1)
   • Authentication
   • Task Creation
   • Polling and Status Inspection
   • Event Streaming (SSE)
   • Inspecting the Transcript
   • Answering Questions and Sending Follow-Ups
   • Stopping a Task
   • Client-Side Filtering and Acceptance Criteria
8. Operational Resilience and Failure Recovery
   • Cluster Capacity and Slot Waits
   • Pod Losses, Evictions, and OOM
   • Model Provider Failures and Fallbacks
   • Poisoned Thought Signatures
   • Stalled Sub-Agents
   • Idle Reaping vs. Question Retention
9. Source Code Map

System Architecture Overview

Crobot is organized into two primary tiers:

1. Centralized Gateway (gateway/): A Node.js/TypeScript service built on Hono that acts as the ingress controller, API gateway, task orchestrator, event relay, and Kubernetes controller. The gateway runs as the crobot-gateway deployment and service inside the crobot namespace (configured by GATEWAY_NAMESPACE, defaulting to crobot in gateway/src/config.ts). The gateway handles webhooks, authenticates callers, tracks usage and infrastructure costs, coordinates pull request reviews and CI checks, and manages the lifecycle of sandbox pods and volumes.
2. Ephemeral Execution Sandboxes (sandbox/): Per-task, isolated container environments executing under the gVisor (runsc) container runtime inside the crobot-tasks namespace. Each sandbox mounts a dedicated Kubernetes PersistentVolumeClaim (PVC), boots an OpenCode server daemon (opencode serve), syncs team knowledge and secrets, and executes shell tools, compilers, test suites, git commands, and browser QA scripts.

The gateway does not execute arbitrary user repository code or compilers itself; all repository modifications, test runs, and git pushes take place exclusively within the untrusted execution sandboxes. Conversely, sandboxes do not hold long-lived GitHub App private keys or Jarvis admin credentials; they request GitHub credentials on demand from the gateway. GitHub App mode returns short-lived installation tokens; token mode returns the configured PAT. These routes authenticate sandbox requests with a per-task bearer token.

Component Responsibilities

Entry Points and Ingress

Crobot accepts work from four main ingress surfaces, all routed through the gateway (gateway/src/routes.ts):

• GitHub App Webhooks (POST /webhooks/github): Validates HMAC-SHA256 signatures (x-hub-signature-256) against GITHUB_WEBHOOK_SECRET. Handles issue mentions, comment triggers (@coframe), pull request review feedback, and check_run failures (gateway/src/github.ts, gateway/src/pr-feedback.ts, gateway/src/ci-fix.ts).
• Slack Events & Interactivity (POST /webhooks/slack, POST /webhooks/slack/interactions): Validates Slack signatures (x-slack-signature) against SLACK_SIGNING_SECRET. Supports direct messages, channel thread mentions, assistant thread view, and Block Kit interactive repository pickers (gateway/src/slack.ts, gateway/src/slack-installs.ts).
• Web UI (ui/): A React/Vite single-page application served by the gateway. Allows team members and customers to initiate tasks, view real-time streaming tool outputs, answer pending questions, inspect usage and cost ledgers, and manage repository settings (ui/src/App.tsx, ui/src/CrobotPage.tsx).
• Programmatic API (/api/v1/*): An authenticated REST API designed for automated pipelines, external scripts, and agent orchestrators (gateway/src/routes.ts).

Gateway Control Plane

The gateway runs as the crobot-gateway service in the crobot namespace and coordinates task execution across several modular controllers:

• Task Service (gateway/src/tasks.ts): Manages Kubernetes resources for tasks (PVC, Secret, Job) in the crobot-tasks namespace. Implements create, ensureRunning, sendPrompt, stop, reap, and volume archival/revival.
• Standby Pool (gateway/src/standby.ts): Maintains a pool of pre-warmed sandbox pods for frequently used repositories. Standbys pre-clone the default branch and boot OpenCode ahead of time, allowing new tasks to start in seconds rather than waiting for cold pod scheduling and repository cloning.
• Relay Event Loop (gateway/src/relay.ts): Connects to the sandbox's OpenCode /event SSE stream. Tracks turn activity, captures live token and dollar costs, extracts pull request URLs from model replies, delivers interim progress messages to Slack/GitHub, handles model retry loops, and dispatches completion webhooks.
• Pull Request & CI Automation (gateway/src/pulls.ts, gateway/src/pr-feedback.ts, gateway/src/ci-watch.ts, gateway/src/ci-fix.ts): Discovers pull requests opened by sandboxes, injects session link footers into PR descriptions, batches review comments into follow-up turns, polls CI check status, and prompts sandboxes to fix failing CI checks when permitted by repository settings.
• Settings, Knowledge & Guides (gateway/src/settings.ts, gateway/src/guides.ts): Stores team knowledge notes, repository-specific environment setup/maintenance commands, and Coframe guide playbooks in Kubernetes ConfigMaps in namespace crobot.
• Usage & Cost Ledger (gateway/src/usage.ts, gateway/src/infra.ts): Aggregates AI token usage, model costs, and GCP node/disk infrastructure hours into monthly ConfigMaps.

Execution Sandboxes

Sandboxes execute in the crobot-tasks namespace under GKE Standard:

• Container Environment (sandbox/Dockerfile, sandbox/entrypoint.sh): Built on Debian with Node.js, Python, git, build essentials, and headless Google Chrome for Testing. Sandboxes run as non-root (uid: 1000, crobot).
• OpenCode Daemon (opencode serve): Listens on port 4096 inside the pod, protected by a per-task password. It exposes REST and SSE endpoints for session management, tool invocation, and event streaming.
• Sandbox Helper Shims (sandbox/bin/):
  • gh: Wrapper around the real GitHub CLI that obtains a token from the gateway via crobot-github-token; token lifetime and scope depend on GitHub App versus PAT mode.
  • git-credential-crobot: Git credential helper providing GitHub tokens dynamically for git fetch and git push.
  • crobot-sync: Fetches repository knowledge, team secrets, and environment commands from GET /internal/sandboxes/:name/context.
  • crobot-screenshot & crobot-video: Uploads browser verification artifacts to the gateway blob store and outputs formatted Markdown tables and GIF previews.
  • crobot-ci: Queries the gateway to print CI failure annotations and logs for the active PR.
  • crobot-suggest-knowledge: Submits proposed knowledge notes back to the gateway for team approval.
• Browser QA Tooling: Employs agent-browser (Vercel agent browser CLI) to drive headless Chrome under gVisor. Sandboxes record WebM video walkthroughs via ffmpeg, which crobot-video trims and optimizes into inline GIF previews.

Persistence and Storage

Crobot utilizes Kubernetes-native primitives in namespace crobot / crobot-tasks and Google Cloud Storage (GCS) for persistence:

• Task Disks (PVCs): Each active task owns a 20Gi standard-rwo PersistentVolumeClaim in namespace crobot-tasks formatted with ext4. The PVC is mounted at /workspace and stores git repositories, package caches (/workspace/.cache), and OpenCode session databases (/workspace/.opencode-data).
• Task Secrets: An Opaque Kubernetes Secret in namespace crobot-tasks holds the OpenCode HTTP basic auth password (OPENCODE_SERVER_PASSWORD), the internal gateway communication token (CROBOT_TASK_TOKEN), and optional user delegation keys (CROBOT_JARVIS_KEY).
• Archived Tasks (TaskArchive, gateway/src/task-archive.ts): When a task's PVC is released to conserve disk quota, task metadata (annotations, status, PR URLs, token totals) is preserved in a ConfigMap in namespace crobot labeled crobot/archived-task: <taskId>.
• Archived Transcripts (TranscriptArchive, gateway/src/transcripts.ts): Session message histories are compressed (gzip) and chunked across ConfigMaps named crobot-transcript-<taskId>[-<chunk>] in namespace crobot. Tool outputs and text fields are safely trimmed to fit within Kubernetes object size boundaries.
• Blob Storage (gateway/src/blobs.ts): Large attachments, screenshots, and video recordings are stored in a GCS bucket (or ConfigMap fallback) and served publicly via random 128-bit unguessable URLs (/shots/<taskId>/<random>.<ext>).

External Services

• GitHub: Source code repositories, webhooks, pull requests, commit checks, and Actions job logs. Authenticated via a GitHub App (coframe[bot]) in App mode, or via personal access token (GITHUB_TOKEN) in token mode.
• Slack: Incoming mentions, interactive pickers, and live status updates via the Slack Events and Web APIs.
• Jarvis (Coframe Core): Authenticates client apps, resolves user organizations and roles, provides Coframe project mappings, and mints acting-user credentials.
• LLM Providers:
  • Provider API keys such as OPENROUTER_API_KEY, ANTHROPIC_API_KEY and OPENAI_API_KEY enter through crobot-provider-keys. Vertex AI authenticates via the sandbox service account's Workload Identity and the GKE metadata server.
  • The active default and allowed models are discovered at runtime via GET /api/v1/models. While source code defaults fall back to google-vertex/gemini-3.8-flash in gateway/src/config.ts, live deployments configure defaults dynamically via environment variables (MODEL_DEFAULT).

Architecture Diagram

The standalone architecture overview diagram is located at docs/architecture.svg:

Text Alternative and Component Flow

1. Ingress: External callers (GitHub Webhooks, Slack API, Web UI, Programmatic API /api/v1) send requests to the Gateway in namespace crobot.
2. Authentication & Validation: The Gateway verifies signatures (HMAC-SHA256 for GitHub and Slack) or tokens (Jarvis JWT, Jarvis API key, Crobot API key) and validates organization scope against Jarvis.
3. Orchestration: The Gateway TaskService either claims an existing ready sandbox from the Standby Pool or provisions a new Kubernetes PVC, Secret, and Job in namespace crobot-tasks. The task creation endpoint returns 201 Created asynchronously.
4. Execution Startup:
   • The Sandbox container boots under gVisor (runsc), mounts /workspace from the PVC, runs initial crobot-sync, and starts opencode serve on port 4096.
   • The Gateway detects the sandbox is healthy, calls OpenCode to create the session, and uses the OpenCode shell endpoint to fetch origin, check out the task working branch crobot/<taskId>, and run repository maintenance commands.
5. Relay & Control: The Gateway Relay connects to the sandbox's /event SSE stream, sends the user prompt, streams live progress, updates Slack/GitHub status lines, and monitors token costs.
6. Tool Operations:
   • Git operations call git-credential-crobot -> gateway /internal/sandboxes/:name/github-token to obtain repository credentials (short-lived installation tokens in App mode, or the configured PAT in token mode).
   • Browser QA runs via agent-browser and captures screenshots/video sent to gateway /internal/sandboxes/:name/screenshots, which stores them in GCS.
   • The agent creates a pull request via gh pr create.
7. Completion & Archival: Turn completion drives idle handling and transcript capture; a PR URL may already exist and is not proof of successful completion. The gateway separately manages PR links and footers. The task service releases eligible PVCs after the configured retention period, or on PR closure when enabled, and moves metadata to TaskArchive.

Task Lifecycle and Execution Flow

Persisted Task Statuses

A task record (TaskRecord, defined in gateway/src/manifests.ts) persists exactly one of five statuses:

In addition to status, the task detail endpoint (GET /api/v1/tasks/:id) returns running: boolean, which indicates whether runningPod() finds a Running or Pending pod with a pod IP. This is not a readiness or model-activity check (gateway/src/tasks.ts). Thus, a task may have status: "idle" with running: true (sandbox is awake and warm) or status: "idle" with running: false (sandbox is sleeping to save memory/CPU).

When an agent invokes the question tool, the relay sets the task's statusDetail annotation with the prefix "Waiting for your answer". The task remains in status: "running". This signals the UI and external callers that input is needed and prevents the idle reaper from terminating the sandbox pod.

End-to-End Lifecycle Sequence

sequenceDiagram
    autonumber
    actor Caller as Caller / User / API
    participant GW as Gateway (Namespace: crobot)
    participant K8s as Kubernetes (GKE / gVisor)
    participant SB as Sandbox Pod (Namespace: crobot-tasks)
    participant GH as GitHub API / Remote

    Caller->>GW: POST /api/v1/tasks (repo, prompt)
    alt Standby Sandbox Available
        GW->>K8s: Claim standby PVC & Secret (adopt)
    else Cold Provisioning
        GW->>K8s: Create PVC (20Gi), Secret, and Job
    end
    GW-->>Caller: 201 Created (task record returned asynchronously)

    K8s->>SB: Boot pod (gVisor runtime, uid 1000)
    SB->>GW: GET /internal/sandboxes/:name/context (crobot-sync)
    GW-->>SB: Knowledge notes, secrets, env scripts
    SB->>SB: Start opencode serve on port 4096

    GW->>SB: Poll until healthy (waitHealthy)
    GW->>SB: Create Session (createSession)
    GW->>SB: Run branch setup through session shell (/session/:sessionId/shell)
    GW->>SB: Run repo maintenance commands (/session/:sessionId/shell)
    GW->>SB: Deliver initial prompt asynchronously (/session/:sessionId/prompt_async)

    loop Agent Execution Turn
        SB->>GW: SSE Events (tool execution, text delta, tokens)
        GW->>GW: Relay updates (cost ledger, Slack/GH status)
        opt Git Push & Pull Request
            SB->>GW: GET /internal/sandboxes/:name/github-token
            GW-->>SB: GitHub credential (App token or configured PAT)
            SB->>GH: git push origin crobot/taskId
            SB->>GH: gh pr create --base baseBranch
        end
        opt Clarification Question
            SB-->>GW: question.asked event
            GW->>GW: Set statusDetail: "Waiting for your answer"
            Caller->>GW: POST /api/v1/tasks/:id/prompt (or question reply route)
            GW->>SB: replyQuestion (structured answer)
        end
    end

    SB-->>GW: session.idle / turn complete (reply text + PR URL)
    GW->>GW: Extract PR URL, annotate task status="idle"
    GW->>GH: Inject session link footer into PR description
    GW->>K8s: Save compressed transcript ConfigMap
    opt Webhook Configured
        GW->>Caller: POST webhookUrl (event: "task.completed")
    end

    opt Idle Timeout (Source default: 30 min)
        GW->>K8s: Delete Job (pod sleep, PVC preserved)
    end
    opt Configured Retention Expired / PR Closed (when enabled)
        GW->>K8s: Delete PVC & Secret, move meta to TaskArchive (status="archived")
    end
    opt Follow-Up on Archived Task
        Caller->>GW: POST /api/v1/tasks/:id/prompt
        GW->>K8s: Revive: claim standby / create PVC, clone & checkout origin branch
        GW->>SB: New session + resume preface quoting archived transcript
    end

Task Identity, Sessions, and State Persistence

Task IDs vs. Session IDs

• Task ID (taskId): The immutable identifier representing the entire unit of work from request to pull request resolution. Generated at inception (gateway/src/taskId.ts):
  • API requests: api-<12 alnum chars> (e.g., api-ms2lt3akrcna)
  • UI requests: ui-<12 alnum chars>
  • GitHub issues/PRs: gh-<owner>-<repo>-i<number> or gh-<owner>-<repo>-pr<number>
  • Slack threads: slack-<channel>-<timestamp> Task IDs are attached to Kubernetes labels (crobot/task: <id>), annotations, transcript ConfigMaps, and blob paths.
• OpenCode Session ID (sessionId): An internal session identifier generated by the opencode daemon within the sandbox pod. A single task maintains one primary OpenCode session per sandbox lifetime. If a sandbox volume is released and later revived, a new OpenCode session ID is created on the fresh sandbox.

Sub-Agent Hierarchy

When the primary agent uses the task tool to spawn sub-agents (e.g., for codebase exploration or parallel research), OpenCode creates child sessions with parentID set to the primary session ID.

• The Gateway Relay discovers descendant sessions via gateway/src/opencode.ts.
• Token usage and financial costs across all descendant sessions are aggregated into the root task's total (gateway/src/usage.ts).
• Sub-agent transcripts are captured alongside the main transcript in ConfigMap storage (gateway/src/transcripts.ts) and can be viewed via GET /api/v1/tasks/:id/subagents/:sessionId.

Persistence Across Sleep, Release, and Revival

Authentication, Security, and Trust Boundaries

[External Callers / Webhooks]
           |
     HTTPS / Ingress
           v
+-----------------------------------------------------------+
| Gateway Control Plane (crobot namespace)                  |
|  - Validates JWT / HMAC / API Keys                        |
|  - Holds GitHub App Private Key & Master Secrets          |
|  - Supplies App installation tokens or a configured PAT  |
+-----------------------------------------------------------+
           | Gateway -> sandbox: HTTP port 4096, Basic auth
           | Sandbox -> gateway: HTTP port 8080, task bearer token
           v
+-----------------------------------------------------------+
| Sandbox Pod (crobot-tasks namespace, gVisor runsc)        |
|  - Security: runAsNonRoot (uid 1000), 16Gi ephemeral layer|
|  - NetworkPolicy: Public egress (Cloud NAT), DNS, Gateway |
|  - Provider Secrets: envFrom crobot-provider-keys         |
|  - Workspace: /workspace (20Gi ext4 PVC)                  |
+-----------------------------------------------------------+

Ingress Authentication and Scoping

Callers authenticate to the Gateway via one of three methods:

1. Jarvis JWT: Verified using JARVIS_JWT_SECRET (HS256).
2. Jarvis API Key (jrv_...): Validated upstream against Jarvis /api/auth/me.
3. Crobot API Key: Configured in gateway environment (CROBOT_API_KEY / CROBOT_API_KEYS). Grants administrative API access. The header X-Crobot-User: user@coframe.com is honored specifically when authenticating via a configured Crobot API key to attribute task ownership. It does not allow arbitrary user impersonation when using standard user tokens.

Organization Scoping & Multi-Tenancy:

• Non-team callers must identify their Jarvis organization using x-crobot-org (or the supported org query parameter). The gateway verifies membership and role against Jarvis (admin, editor, approver, viewer).
• Callers may only access repositories connected to their organization.
• Customer Views: taskForViewer removes model and usage fields from non-team task records; messageForCustomer removes selected model, provider and usage fields from message metadata on routes that apply it. This is not a universal redaction guarantee: /api/v1/models still returns the default model identifier, with only that model in the non-team allowed list (gateway/src/routes.ts).

The /internal/sandboxes/* handlers check the sandbox task bearer token. Their name does not establish a cluster-only network boundary: the checked-in gateway Ingress forwards / to the same service. Additional production routing restrictions were not verified (gateway/src/routes.ts, infra/k8s/gateway/ingress.yaml).

Sandbox Hardening and Isolation

Sandboxes run untrusted code and execute user commands. They are isolated using defense-in-depth:

• gVisor Runtime (runtimeClassName: "gvisor"): Intercepts and virtualizes Linux kernel syscalls in userspace (runsc), mitigating container breakout vulnerabilities.
• Unprivileged Execution: Pod security context mandates runAsNonRoot: true, runAsUser: 1000, runAsGroup: 1000, fsGroup: 1000.
• Ephemeral Storage Budget: The container writable layer has an explicit request and limit of 16Gi ephemeral storage (gateway/src/manifests.ts). Package manager caches are directed to /workspace/.cache on the persistent disk to prevent ephemeral exhaustion.
• NetworkPolicy (infra/k8s/base/sandbox-networkpolicy.yaml):
  • Restricts ingress to port 4096 from the crobot namespace only.
  • Permits egress to UDP/TCP port 53 (cluster DNS and NodeLocal DNSCache).
  • Permits egress to 169.254.169.254/32 (GKE metadata server for Workload Identity tokens).
  • Permits egress to port 8080 on pods in namespace crobot (the gateway internal service).
  • Permits egress to the public internet (0.0.0.0/0 via Cloud NAT), explicitly excluding private RFC 1918 networks, CGNAT ranges, and the cluster Services CIDR.

Credential Delegation and Token Isolation

• Provider Keys: Provider API keys such as OPENROUTER_API_KEY, ANTHROPIC_API_KEY and OPENAI_API_KEY enter through crobot-provider-keys (envFrom: [{ secretRef: { name: "crobot-provider-keys" } }]). Vertex AI authenticates via the sandbox service account's Workload Identity and the GKE metadata server.
• Jarvis Delegation Isolation: When a user runs a task from Jarvis, a personal delegation key (CROBOT_JARVIS_KEY) is stored in the task's Secret for the gateway to act as that user against Jarvis Core (e.g., creating Coframe metrics). This key is never injected into the sandbox pod's container environment; only the gateway reads it.
• GitHub Tokens: Sandboxes do not store GitHub App private keys. The sandbox CLI shims query the gateway at GET /internal/sandboxes/:name/github-token using CROBOT_TASK_TOKEN. In GitHub App mode, the gateway mints a short-lived GitHub App installation access token scoped strictly to the task's repositories, expiring in 1 hour. If the gateway is configured in token mode (GITHUB_AUTH_MODE=token), the gateway returns the configured GITHUB_TOKEN.

External-Agent API Workflow (/api/v1)

The programmatic API allows external agents (such as orchestrator agents or CI runners) to manage Crobot tasks end-to-end.

Authentication

Pass a personal Jarvis API key or a configured Crobot API key via Authorization: Bearer or X-API-Key:

export API_KEY="jrv_your_personal_jarvis_key"
export CROBOT_BASE_URL="https://crobot.coframe.com"

Note: If using a shared CROBOT_API_KEY, you can attribute task creation to a specific email using X-Crobot-User: user@coframe.com.

Task Creation

Submit a new task via POST /api/v1/tasks. The endpoint creates the Kubernetes record and returns 201 Created immediately while execution starts in the background:

curl -s -X POST "$CROBOT_BASE_URL/api/v1/tasks" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "repo": "Coframe/jarvis",
    "prompt": "Fix null pointer in user profile component and verify test suite",
    "baseBranch": "main",
    "webhookUrl": "https://my-service.com/api/crobot-webhook"
  }'

Response (201 Created):

{
  "id": "api-ms2lt3akrcna",
  "source": "api",
  "repo": "Coframe/jarvis",
  "baseBranch": "main",
  "title": "Fix null pointer in user profile component",
  "status": "starting",
  "createdAt": "2026-09-15T12:00:00.000Z",
  "taskUrl": "https://crobot.coframe.com/tasks/api-ms2lt3akrcna"
}

(Note: running is omitted in the creation response).

Polling and Status Inspection

Inspect task status via GET /api/v1/tasks/:id. The detail endpoint adds running: boolean to indicate whether a sandbox pod is alive:

curl -s "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna" \
  -H "Authorization: Bearer $API_KEY"

Response (200 OK):

{
  "id": "api-ms2lt3akrcna",
  "status": "idle",
  "running": true,
  "prUrl": "https://github.com/Coframe/jarvis/pull/2450",
  "prs": [
    {
      "repo": "Coframe/jarvis",
      "number": 2450,
      "url": "https://github.com/Coframe/jarvis/pull/2450"
    }
  ],
  "cost": 0.42,
  "tokens": {
    "input": 12500,
    "output": 3200,
    "reasoning": 850,
    "cacheRead": 4000,
    "cacheWrite": 0
  },
  "lastActive": "2026-09-15T12:08:30.000Z"
}

Note: If statusDetail starts with "Waiting for your answer", the agent is waiting on human input while status remains "running".

Monitoring Caveat: The persisted running annotation can be stale after a completed turn. An external controller can refine it with /opencode/session/status and the latest session messages. An idle session-status map plus a completed final assistant reply is evidence that the turn ended; an empty map alone can occur during startup. A pending question/permission still takes precedence.

Event Streaming (SSE)

For live real-time observation, external agents can stream execution events via Server-Sent Events (SSE):

curl -N "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/events" \
  -H "Authorization: Bearer $API_KEY"

Note: The sandbox pod must be running. If the sandbox is sleeping, this endpoint returns 409 Conflict.

Inspecting the Transcript

Read the compressed transcript snapshot of the last completed turn:

curl -s "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/transcript" \
  -H "Authorization: Bearer $API_KEY"

Note: This endpoint serves a possibly trimmed snapshot archived at turn completion. If called during the very first turn before an archive exists, it returns 404 Not Found.

Answering Questions and Sending Follow-Ups

1. Simple Prompt Answer: Sending a text prompt to /prompt will automatically apply that text to all currently pending questions on the task:

   curl -s -X POST "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/prompt" \
     -H "Authorization: Bearer $API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Use option A and continue the implementation."}'

2. Structured Answers for Distinct Questions: When multiple questions are pending and need distinct answers, interact with OpenCode via the reverse proxy:

   • Query pending questions:

     curl -s "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/opencode/question" \
       -H "Authorization: Bearer $API_KEY"

   • Reply with distinct answers:

     curl -s -X POST "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/opencode/question/<requestId>/reply" \
       -H "Authorization: Bearer $API_KEY" \
       -H "Content-Type: application/json" \
       -d '{"answers": [["First answer"], ["Second answer"]]}'

   • Permission requests are handled separately via the task proxy (ui/src/opencode.ts):
     • Query pending permissions:

       curl -s "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/opencode/api/session/<sessionId>/permission" \
         -H "Authorization: Bearer $API_KEY"

     • Reply to a permission request (valid values for reply are "once", "always", or "reject"):

       curl -s -X POST "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/opencode/api/session/<sessionId>/permission/<requestId>/reply" \
         -H "Authorization: Bearer $API_KEY" \
         -H "Content-Type: application/json" \
         -d '{"reply": "once"}'

Stopping a Task

Interrupt an active turn without destroying the volume:

curl -s -X POST "$CROBOT_BASE_URL/api/v1/tasks/api-ms2lt3akrcna/stop" \
  -H "Authorization: Bearer $API_KEY"

Client-Side Filtering and Acceptance Criteria

1. Task Retrieval & Filtering: The GET /api/v1/tasks endpoint does not support cursor-based pagination. It accepts repo, status, source, experiment, and limit. Note that limit is evaluated on the server before client-side filtering. External agents needing to inspect their own tasks should retrieve without a narrow limit, filter by repository, or look up known task IDs directly.
2. Independent Acceptance Verification: A task state of status: "idle" combined with a prUrl means the record is idle and a PR is associated with it. An explicit stop or a previously opened PR can produce the same combination; inspect the latest turn separately. It does not guarantee that the solution passes all user acceptance criteria or tests. Orchestrators must independently inspect the PR diff, evaluate CI check results, and verify functionality.

Operational Resilience and Failure Recovery

Cluster Capacity and Slot Waits

• When nodes in the GKE cluster are fully occupied, a new sandbox pod cannot schedule (PodScheduled=False).
• The gateway detects this condition (gateway/src/tasks.ts) and updates statusDetail to: "Waiting for a sandbox slot: every node is full (N min so far)".
• Standby Eviction: After 30 seconds of waiting, the gateway instructs the standby pool (gateway/src/standby.ts) to retire an idle pre-warmed standby (StandbyPool.makeRoom) to free node capacity.
• The task waits up to capacityWaitMs (default 60 minutes) for a node slot before failing. The initial prompt text is preserved in the PVC annotation crobot/request and dispatched once the pod starts.

Pod Losses, Evictions, and OOM

• If a pod crashes due to out-of-memory or node eviction, the Kubernetes Job automatically retries (up to backoffLimit: 3).
• All package manager caches (npm, uv, pip, Playwright) are placed on the persistent volume at /workspace/.cache to prevent exhausting the container's 16Gi ephemeral storage.
• The gateway monitors pod loss reasons via Kubernetes status (podLoss). It counts losses in crobot/pod-losses. If a pod fails twice in a single turn (MAX_POD_LOSSES = 2), the task fails with the exact diagnostic message rather than continuing to loop.
• Upon pod recovery, the relay replays the request with RESUME_PREFACE, informing the agent that its sandbox was restarted.

Model Provider Failures and Fallbacks

• If a model call fails due to rate limits (429) or transient provider outages (500, 502, 503, socket timeouts), gateway/src/relay.ts uses the configured PROVIDER_RETRY_DELAYS_MS schedule. The source configuration defaults to 20s and 60s; the older 30s/90s/180s constant is only a fallback when no schedule is supplied. Deployment overrides and OpenCode’s own retry behavior can differ (gateway/src/config.ts).
• If the primary provider remains unavailable after retries, the relay can make one additional attempt on the first allowed fallback different from the current model, selected from MODEL_FALLBACKS (or the allowed-model list when no fallback list is configured). It does not iterate indefinitely through every provider. Available models should be queried from GET /api/v1/models.

Poisoned Thought Signatures

• Gemini models require function-call thought signatures to be echoed back verbatim. Occasionally, third-party proxies mangle reasoning_details, causing subsequent turns to fail permanently with HTTP 400 (Corrupted thought signature).
• The relay classifies this as a poisoned conversation. It calculates the earliestBrokenRequest across the transcript, rolls back the broken request, and replays it using the selected allowed fallback when one exists. The provider is configuration-dependent; Vertex AI is not guaranteed (gateway/src/relay.ts).

Stalled Sub-Agents

• If a sub-agent session becomes stuck (e.g., waiting for interactive input or hanging on an external API), the idle reaper aborts sub-sessions that have made no progress for stallMinutes (default 45 minutes). The aborted session reports an error; subsequent progress depends on how the remaining agent turn handles it.

Idle Reaping vs. Question Retention

• Pods idle longer than idleMinutes (source default 30 minutes in gateway/src/config.ts, configurable per environment) have their Kubernetes Job deleted to free CPU/RAM. The PVC and disk contents remain intact.
• Question Exception: If an agent asks a question (statusDetail begins with "Waiting for your answer"), the reaper uses a 6-hour inactivity threshold (WAITING_MAX_MS = 6 * 3600_000) instead of the normal idle threshold. This is not a guaranteed pod lifetime: the busy-session check can retain the pod longer, and other lifecycle limits still apply.

Source Code Map