<!-- source: openspec/current/forge/verifier.md -->

---
title: Forge — Verifier
status: stable
owner: pavel
audience: [pavel, ai, engineer]
tldr: Verification contract for forge build — five checkpoint gates, build outcome enum, failure modes, retry policy, and known gaps.
---
# Forge — verifier

> Part of openspec/current/forge/ — split from forge.md on 2026-05-12

---

## Checkpoint model (verification gates)

`forge build` walks **five programmatic checkpoints** and exits at a permanent post-PR boundary. The five programmatic gates default to stop-and-acknowledge; `--auto-*` flags skip specific gates. The post-PR boundary is **not** a programmatic gate — the orchestrator records a `completed` event and returns after PR creation; merging is performed by a human, not by Forge.

| # | Stage | Type | Default | Skip flag |
|---|---|---|---|---|
| 1 | After `forge plan` (plan written, awaiting approval) | programmatic gate | stop | `--auto-approve-plan` (RAP-403); `--auto-plan` alias auto-passes this gate AND skips architect generation |
| 2 | After migrations applied (DDL changes) | programmatic gate | stop | `--auto-migrations` |
| 2.5 | After spec-acceptance deterministic checks (see § Spec-acceptance gate) | programmatic gate | stop | `--auto-spec-acceptance` |
| 3 | After code written, before commit | programmatic gate | optional | default skip; `--review-code` adds the gate |
| 4 | Before `git push` | programmatic gate | stop | `--auto-push` |
| 5 | Before `gh pr create` | programmatic gate | stop | `--auto-pr` |
| 6 | After PR created, before merge | **exit point** (behavioural boundary, not a gate) | orchestrator returns; PR open, awaiting human review | n/a — Forge does not merge |

#### Plan auto-invocation (RAP-404)

`forge build <key>` auto-invokes `runPlan` when `~/.forge/plans/<key>.md` is missing. Pre-RAP-404 the orchestrator threw "No plan found" (RAP-402); the new behaviour mirrors the inline pattern from `commands/epic/orchestrator.ts` (epic driver already did this lazy-plan dance, lifting it to `forge build` itself eliminates the split).

`--require-plan` opts back into the legacy strict gate: when set, a missing plan throws the RAP-402 error instead of auto-invoking. Operators who want to enforce a human "OK to plan, OK to build" hand-off before the architect spends Anthropic budget pass `--require-plan`.

The auto-run plan's cost is accumulated under `state.costByRole.architect` (see § Cost surface under split `--auto-plan` flags) — same as if the operator had run `forge plan <key>` manually first.

#### `--auto-plan` flag split (RAP-403)

Pre-RAP-403, `--auto-plan` did two unrelated things: (a) skipped the AI plan-generation phase (architect role) and reused the cached plan, AND (b) auto-passed the `plan-acceptance` checkpoint at runtime. Two operator intents bound to one flag — memory `forge_architect_budget_cap` recorded 47 of 49 recent builds skipping architect entirely as a side effect.

RAP-403 introduces two flags:

| Flag | Effect |
|---|---|
| `--skip-architect` | Skip the AI plan-generation phase. Use the cached plan markdown as-is. |
| `--auto-approve-plan` | Auto-pass the `plan-acceptance` checkpoint at runtime. |
| `--auto-plan` (back-compat alias) | Implies BOTH `--skip-architect` AND `--auto-approve-plan`. Deprecated; `--help` carries the note. |

Operators can now opt into either half independently: `--auto-approve-plan` alone re-runs architect (paying its budget) and still auto-passes the gate, useful when the issue body has changed since the last cached plan. `--skip-architect` alone reuses the cached plan but stops at the gate for human review.

Programmatic-gate implementation lives in `commands/build/checkpoints.ts`. Each of the five gates records a structured event into `BuildState.checkpointHistory[]` so `forge status` can replay the timeline. The orchestrator (`commands/build/orchestrator.ts`) calls `await checkpoint(name, summary)` for stages 1–5; it does **not** call `checkpoint('pre-merge', ...)`. The "pre-merge checkpoint" name elsewhere in this document and in `packages/muse/src/builder-prompt.ts` captures the *behavioural rule* — Forge stops, the PR sits open until a human merges — not a programmatic pause. RAP-48 Phase 3.A finding #9 prompted this distinction; the prior wording ("six ordered checkpoints… stop-and-acknowledge at each one") implied a sixth programmatic gate that does not exist.

The post-PR boundary is a deliberate human-in-the-loop on accepting work into `main`. Merging individual build PRs is owned by Pavel (or whoever reviews the PR), never auto-triggered by `forge build` itself. Auto-merge at the `forge build` level remains out of scope. At the `forge epic` level, `--auto-merge` (RAP-408) enables GitHub auto-merge for sub-issue PRs labelled `auto-merge-safe`, but only when explicitly opted in — see `commands.md § Epic conductor`.

---

## Build outcome

`BuildState.outcome` (RAP-392 A4) is the authoritative end-of-run signal. `currentCheckpoint` survives as an in-progress cursor (still useful for `forge status` and the future `forge resume`); `outcome` is what an honest consumer — `forge epic`, the studio Phase 2 UI, an external babysitter loop — reads to answer "did this build ship".

| Outcome | When written | Notes |
|---|---|---|
| `in-progress` | At `runBuild` start | Reset on every successful start; preserved on the orchestrator's `finally` releaseBuildLock path |
| `completed-with-pr` | After `GitHubClient.createPullRequest` succeeds | `isShippedOutcome(outcome) === true` — the only outcome the epic driver advances past |
| `completed-no-pr` | After `pr-creation` checkpoint passes but the user declines or `createPullRequest` warns (no `--auto-pr`) | The dry-run path also terminates here |
| `aborted-no-changes` | When `git diff --cached --quiet` exits 0 after `git add -A` AND `git rev-list main..HEAD --count` is 0 (RAP-418) | Builder produced no tracked changes AND the branch has no commits ahead of base. When the branch is ahead of base, the orchestrator treats the run as Builder-self-committed and proceeds to push + PR. |
| `aborted-at-checkpoint` | When `enterCheckpoint` returns `proceed: false` (human said "no") | Survives the catch-block's `'failed-builder'` overwrite |
| `failed-commit` | `git commit` exit non-zero | Branch left in whatever state `git add -A` reached |
| `failed-push` | `git push -u origin <branch>` exit non-zero | Worktree has the commit but the remote does not |
| `failed-pr` | `createPullRequest` throws under `--auto-pr` | Push succeeded; PR call did not (auth, branch protection, GitHub 5xx) |
| `failed-builder` | Any other exception in the try-block (`runBuild` catch) | Generic bucket for verification gates, adapter blocked, SDK transport failures |
| `cancelled` | `AbortController.signal.aborted` OR `CancelledByExternalSignal` | `forge cancel <key>` from any process |

`outcome` is optional on the `BuildState` type only so pre-RAP-392 state files load without error; new writes always populate it. Callers should treat `state.outcome` as the authoritative signal and consult `state.currentCheckpoint` only when they need to know **where** the run got stuck.

Why this lives in the orchestrator rather than in the agent's prompt: the CI drift guard (`tools/build-openspec-bundle.mjs --check`) is fail-closed, and dogfood runs kept tripping it because the agent edited a spec, committed, and forgot to regen. A deterministic post-step removes the foot-gun without expanding the prompt.

---

## Failure modes

When `forge build` encounters an unexpected condition (Anthropic SDK error, Supabase migration failure, GitHub API failure, plan ambiguity, drift between spec and live state), Forge follows a strict **stop-write-comment-exit** pattern:

1. Stop immediately. No auto-recovery attempts in v1.
2. Write `.forge/errors/<RAP-NN>-error.md` with full context (last command, error message, attempted action, checkpoint reached, cost so far).
3. Post a Linear comment on the issue with a one-line summary plus a pointer to the local error file.
4. `process.exit(1)`.

Pavel investigates manually. Auto-recovery (retry with different approach, alternative path, fallback model) is deferred to v2 once we observe what failure modes actually occur in practice. The `~/.forge/state/<RAP>.json` file is **not cleaned up** on failure — it's left in `failed` state so `forge status RAP-NN` can render the timeline and `forge cancel` (when shipped) can clean up if Pavel chooses.

`packages/forge/src/core/failure.ts` exposes `reportFailure(issueKey, context)` which encapsulates the four-step pattern.

**Known specific modes (observed in dogfood):**

- **Verification gates failed during build.** When `Plan.verificationGates[]` declares commands like `pnpm typecheck` / `pnpm lint` / `pnpm test`, the build orchestrator pairs each gate with the SDK subprocess's recorded Bash invocation and inspects the captured output for stable failure markers (`commands/build/verification.ts § parseVerificationResults` + `FAILURE_MARKERS`). Marker matching is **content-based**, not `is_error`-based — recorded commands routinely pipe through `tail`/`head`, masking the real exit code (RAP-48 Phase 3.A finding #12, shipped via PR #385). When one or more gates fail, the orchestrator throws `VerificationFailedError` with the format `"Verification gates failed: <gate1>, <gate2>, … (<N>/<M>)."` and routes through the standard stop-write-comment-exit path above. First observed in real-issue dogfood — Track D Cycle 1 (RAP-167, 2026-05-09): the plan declared 8 gates, the build wrote clean ROADMAP.md edits, but `pnpm typecheck` / `pnpm lint` / `pnpm test` failed (3/8) and the orchestrator correctly aborted before push. Operator response: read `.forge/errors/<RAP>-error.md` for the failed-gate evidence (~600 char tail per gate); fix root cause (a working-tree contaminant, an upstream regression on `main`, or a real defect in the build's diff); re-run `forge build` from the same checkpoint, or finish the work manually if the diff is small. The gate is doing its job — it refuses to push code that wouldn't survive CI.

---

## Retry policy

### Missing-plan error contract (RAP-402)

`runBuild` checks `existsSync(planPath)` at entry. When the plan markdown is absent and `--require-plan` (RAP-404) is set, the orchestrator throws with the message:

```
No plan found at <planPath>. Run `forge plan <issueKey>` first.
```

The pre-RAP-402 wording said "No approved plan found", implying a plan-approval state that does not exist (no frontmatter status, no `forge plan approve` command). The honest message above is the source of truth.

Without `--require-plan` — the default after RAP-404 — `runBuild` auto-invokes `runPlan` instead of throwing; see § Checkpoint model.

### Cost surface under split `--auto-plan` flags (RAP-403)

`state.costByRole.architect` is the architect-role accumulator. `--skip-architect` (RAP-403) bypasses the architect entirely; under that flag `costByRole.architect` stays `0` for the run. `--auto-approve-plan` alone (without `--skip-architect`) still runs the architect — `costByRole.architect` reflects its spend even though the operator never sees the plan-acceptance prompt. The legacy alias `--auto-plan` zeroes the architect spend, same as `--skip-architect`. The `ForgeConfig.budgets.architect` cap (when configured) is checked against `costByRole.architect` before the architect call, so `--skip-architect` makes the cap an inert dead-code guard for runs that pass it.

---

## Test coverage

`packages/forge/src/__tests__/` — **~80 test files** as of 2026-05-11 (verify with `find packages/forge/src -name '*.test.ts' | wc -l`). The suite has grown materially during Builder-mode shipping (Waves 1-3 added ~50 files: build orchestrator, checkpoints, cancel sentinel/guard/orchestrator, plan orchestrator/builder/render, inspect/inbox/epic orchestrators, finalize, merge-poller, gate-validator, golden-studio, worktree, migration-files, persistence-adapter, retry, json-retry, verifier, reviewer, status-orchestrator, build-checkpoints, build-retry, build-subprocess, build-registry, finding-hash, lessons-parallel, pr-precheck, report, render, anthropic, lock, plus the long-standing audit/spec/estimate/review/state/config/lessons/auth/usage/baseline coverage). Skipped suites concentrate in `__tests__/modules.test.ts` (module-registry on-disk assertions deferred until the consumer-owned registry refactor lands).

### What's tested

- Config resolution + validation (`config.test.ts`)
- Project-context lifecycle: load, cache, invalidate (`project-context.test.ts`)
- LinearClient parsing — issue keys, GraphQL responses (`linear-client.test.ts`)
- JSON-fence stripping helpers (`core-json.test.ts`)
- Agent batching and grouping (`orchestrator.test.ts`)
- Verifier weight math (`verifier.test.ts`)
- Audit Linear-integration formatting (`linear-integration.test.ts`)
- Audit report generation (`report.test.ts`)
- Glob escape for Next.js dynamic segments (`utils.test.ts`)
- Builder-mode state machine: checkpoint transitions (`build-checkpoints.test.ts`), gate validation (`gate-validator.test.ts`), retry-on-gate-failure (`build-retry.test.ts`), build orchestrator dispatch (`build-orchestrator.test.ts`)
- Cross-process cancel: sentinel detection (`cancel-sentinel.test.ts`), cancel guard (`cancel-guard.test.ts`), cancel orchestrator (`cancel-orchestrator.test.ts`)
- Per-issue worktree isolation: setup + adoption (`worktree.test.ts`)
- Global build lock (`lock.test.ts`)
- Lessons-learned parallel-safe append (`lessons-parallel.test.ts`)
- Plan generation: builder (`plan-builder.test.ts`), orchestrator (`plan-orchestrator.test.ts`), rendering (`plan-render.test.ts`)
- Inspect orchestrator (`inspect-orchestrator.test.ts`)
- Inbox orchestrator (`inbox-orchestrator.test.ts`)
- Epic orchestrator + topo-sort (`epic-orchestrator.test.ts`, `epic-topo-sort.test.ts`)
- Status orchestrator (`status-orchestrator.test.ts`)
- Finalize (commit/push/PR) flow (`finalize.test.ts`)
- Merge poller (`merge-poller.test.ts`)
- Finding hash dedup (`finding-hash.test.ts`)
- Persistence adapter shims + Builder hooks (`persistence-adapter.test.ts`, `persistence-adapter.build.test.ts`)
- Per-role budget caps (`budget.test.ts`)
- Per-role model resolution (`anthropic.test.ts`)
- Auth identity loader (`auth.test.ts`)
- BuildMetrics frozen schema (`metrics.test.ts`)
- Render pipeline (`render.test.ts`)
- Mirror-export deterministic tar.gz (`mirror-export.test.ts`)
- Spec --fix proposal generation (`spec-fix.test.ts`)
- Spec checker baseline (`baseline.test.ts`)
- Init / config (`init-forge.test.ts`, `config.test.ts`, `state.test.ts`)
- JSON-retry billing instrumentation (`retry.test.ts`, `json-retry.test.ts`)

### What's not tested (by unit tests)

- End-to-end `forge audit` run against fixtures with real Anthropic (would require live keys)
- Real Linear GraphQL responses end-to-end (would require live keys)
- Real Supabase persistence end-to-end (would require live keys)
- **`forge review --orchestrate` (orchestrated review)** — code path not yet shipped; see § Known gaps item 8. The `reviewer.test.ts` + `verifier.test.ts` files cover single-mode review and verifier unit logic, not orchestration.

`pnpm --filter @rapoport-studio/forge test:coverage` runs vitest with `@vitest/coverage-v8` and enforces 60% thresholds across lines, functions, branches, and statements. Configuration lives in `packages/forge/vitest.config.ts`. HTML + text + JSON-summary reports land in `packages/forge/coverage/` (gitignored). CI (`check` job in `.github/workflows/ci.yml`) runs the coverage step after the baseline test step and uploads `packages/forge/coverage/` as a `forge-coverage` artifact (14-day retention). The job fails if any threshold is not met.

Ratcheting the threshold above 60% is deferred to a future issue once coverage baseline stabilises across added test files.

---

## Known gaps

The path between the package's current state and "stable on multiple consumers" is concrete. One open item remains as of 2026-05-11 after this stabilization arc closes:

8. **Orchestrated review is specified but unimplemented.** `commands/review/orchestrator.ts`, `commands/review/agents/*`, `commands/review/prompts/{security,a11y,design-system,architect,qa,spec-fidelity,_shared,_verifier}.md`, and the shared `core/orchestration.ts` primitive are described in `commands.md § Review orchestration` but not yet shipped. Single-mode `forge review` (no flag) works today; `--orchestrate`, `--pr`, and `--post` will fail until the implementation lands. Tracked in Linear [RAP-AI-111](https://linear.app/rapoport-studio-slr/issue/RAP-AI-111). The `audit/orchestrator.ts` parallel-fanout pattern (Promise.all + Promise.allSettled at `commands/audit/orchestrator.ts:77,189`) is the reference implementation to mirror.

(Items 1-3 closed by changes that landed before May 2026. Items 4-7 closed under [RAP-161](https://linear.app/rapoport-studio-slr/issue/RAP-161) / [RAP-165](https://linear.app/rapoport-studio-slr/issue/RAP-165) / [RAP-166](https://linear.app/rapoport-studio-slr/issue/RAP-166) / dead-code removal. The crossed-out entries previously listed here have been retired by `sync-forge-spec-to-code` per D-FORGE-SYNC-2 — section is now scope-current.)

---

## Spec-acceptance gate (applied 2026-05-14 from change `add-spec-acceptance-gate`)

A sixth programmatic checkpoint inserted between `migrations` (#2) and `code-review` (#3). It runs deterministic assertions from `openspec/changes/<slug>/verification.md` — the fourth artefact produced by `muse draft` — before the AI reviewer sees the diff.

### Runtime flow

```
forge build <RAP-NN>
    …migrations checkpoint…
    └─ Phase 5.5: spec-acceptance (NEW)
            ├─ core/acceptance.ts:parseVerificationMd()    — prose → AssertionSpec[]
            ├─ core/acceptance.ts:runDeterministicChecks() — shell → AssertionResult[]
            └─ commands/build/reviewer.ts:buildUserMessage()
                    + injects ## Acceptance criteria section
                    + passes deterministic-check results as ground truth
```

Implementation: `packages/forge/src/core/acceptance.ts`. Design source-of-truth: `openspec/archive/2026-05-14-add-spec-acceptance-gate/design.md`.

### Behaviour

- **If `verification.md` is absent** — legacy-mode: gate is skipped entirely. All 179+ archived changes and any active change without `verification.md` continue to build without retroactive injection.
- **If `verification.md` is present** — `parseVerificationMd()` extracts `## Acceptance` assertions; `runDeterministicChecks()` runs them. Any `'failed'` assertion throws `SpecAcceptanceFailedError` → `aborted-at-checkpoint` outcome at checkpoint `spec-acceptance`. Reviewer's verdict is recorded but does not ungate a deterministic red.
- **`reviewer-confirms` assertions** — not run by the deterministic checker. Surfaced to the AI reviewer as a mandatory checklist item; reviewer must address in `evidence[]`. Reviewer-confirms lines do not trigger `aborted-at-checkpoint` by themselves.

### Six assertion shapes

| # | Shape | Surface syntax | Checker |
|---|---|---|---|
| 1 | `file-exists` | `Done when: [ -f <path> ]` | `fs.existsSync(path)` |
| 2 | `grep-returns-N` | `Done when: grep -c '<marker>' <path> returns <N>` | `execFileSync('grep', ['-c', marker, path])` → compare |
| 3 | `test-passes` | `Done when: pnpm <cmd>` | spawn pnpm, exit code 0; output scanned for `FAILURE_MARKERS` |
| 4 | `PR-exists-for-branch` | `Done when: gh pr list --head <branch> returns ≥<N>` | `execFileSync('gh', ['pr', 'list', '--head', branch, '--json', 'number'])` → count |
| 5 | `state-field-equals` | `Done when: jq -r '<path>' <state.json> returns '<value>'` | `execFileSync('jq', ...)` → compare |
| 6 | `reviewer-confirms` | `Reviewer confirms: <claim>` | NOT run deterministically — surfaced to AI reviewer |

**Parser resilience** (mirrors `commands/build/verification.ts:104-130`): unrecognised lines → `parse-error`, not thrown. Empty `## Acceptance` section → `[]`, treated as hardened-prompt-only mode.

### `BuildState.checkpointHistory` entry

The orchestrator pushes one entry for the spec-acceptance checkpoint:

```ts
{
  name: 'spec-acceptance',
  status: 'passed' | 'failed',
  summary: '<N>/<M> deterministic assertions passed',
  timestamp: '<ISO>',
  details?: string[]  // raw assertion lines that failed
}
```

### AI reviewer context-isolation guarantee (applied 2026-05-14 from Phase 2 of change `add-spec-acceptance-gate`)

Full audit: `openspec/_methodology/research/forge-reviewer-isolation-2026-05.md`.

**Pre-fix state:** `buildUserMessage()` (`reviewer.ts:259-276`, pre-RAP-740) unconditionally included `## SDK transcript excerpt (tail)` — the Builder's raw JSONL tool-call history (Bash commands, file reads, edits, intermediate SDK reasoning) — in every non-degraded reviewer call. The reviewer could confirm the builder's own assessment rather than independently evaluating the diff.

**Fix applied (RAP-740):** The raw excerpt section was removed. Replaced with a single summary line: `## Builder activity: <N> tool calls, <M> file edits`. Counts are derived by running `parseToolCallsFromTranscript()` (already used by `metrics.ts`) against the JSONL — `tool_use` events are counted but their content is not forwarded.

**Isolation guarantee post-fix.** The reviewer's input in non-degraded mode contains:

| Section | Content |
|---|---|
| `## Build context` | Issue key, wave, branch, checkpoint, cost |
| `## Builder activity: N tool calls, M file edits` | Counts only — no content, no reasoning |
| `## Acceptance criteria` (when `verification.md` present) | Raw `## Acceptance` section + deterministic check results table |
| `## Verification gates` | Gate names + pass/fail status + evidence tail on failure |
| `## Baseline failures` | Pre-existing-on-main gates (when applicable) |
| `## Git diff against build base` | Full diff (up to 24 000 chars) |
| `## Pre-existing-failure heuristic` | File-intersection hint (when applicable) |

The reviewer's ground truth is the diff and gate results only. It cannot see the builder's reasoning, intermediate Bash commands, or file-read content.

**Degraded mode** (`transcriptPath === null`): the `## Degraded mode` section is unchanged and explains the dead-subprocess condition. In that case the reviewer still has no builder reasoning — it has even less information.

---

## Per-role model tiering, retry-on-gate, AI reviewer + drift metrics (applied 2026-05-10 from change `improve-forge-model-tiering-and-verifier`)

Five-phase epic ([RAP-213](https://linear.app/rapoport-studio-slr/issue/RAP-213)) implementing the three-role model architecture (Architect = Opus 4.7, Builder = Sonnet 4.6 by config, Reviewer = Haiku 4.5 with Sonnet 4.6 escalation — RAP-217 Phase 5 routed escalation to Sonnet rather than Opus because the reviewer is a structured-JSON classifier and Sonnet handles that reliably for ~5× less cost), build retry-on-gate-failure with one focused retry, per-role cost guards (`ForgeConfig.budgets` + `BuildState.costByRole`), AI reviewer with degraded modes (dead-subprocess, pre-existing-failure detection), and the frozen v1 `BuildMetrics` schema at `core/metrics.ts` as a Maestro/Muse-consumable contract for spec-vs-code drift learning.

Bonus work shipped under the same epic: cancel state-machine guard ([RAP-218](https://linear.app/rapoport-studio-slr/issue/RAP-218)) preventing `completed → cancelled` corruption, and the global build lock at `~/.forge/locks/build.lock` ([RAP-224](https://linear.app/rapoport-studio-slr/issue/RAP-224)) preventing multi-session collisions on the parent `.git` (the dead-subprocess pattern that motivated the epic in the first place).

Authoritative proposal/design/tasks: `openspec/archive/2026-05-10-improve-forge-model-tiering-and-verifier/` — this marker is pointer-only.

Phase issues: [RAP-214](https://linear.app/rapoport-studio-slr/issue/RAP-214) ([#519](https://github.com/rapoport-studio/rapoport.studio/pull/519)) · [RAP-215](https://linear.app/rapoport-studio-slr/issue/RAP-215) ([#524](https://github.com/rapoport-studio/rapoport.studio/pull/524)) · [RAP-216](https://linear.app/rapoport-studio-slr/issue/RAP-216) ([#530](https://github.com/rapoport-studio/rapoport.studio/pull/530)) · [RAP-217](https://linear.app/rapoport-studio-slr/issue/RAP-217) ([#533](https://github.com/rapoport-studio/rapoport.studio/pull/533)). Bonus: [RAP-218](https://linear.app/rapoport-studio-slr/issue/RAP-218) ([#517](https://github.com/rapoport-studio/rapoport.studio/pull/517)) · [RAP-224](https://linear.app/rapoport-studio-slr/issue/RAP-224) ([#522](https://github.com/rapoport-studio/rapoport.studio/pull/522)).
