<!-- source: openspec/current/forge-audit.md -->

---
title: Forge — Audit watchman and calibration falsifier
status: stable
owner: pavel
audience: [pavel, ai, engineer]
tldr: Capability spec for the OpenSpec audit calibration watchman — thresholds, event schemas, and falsifier semantics.
---
# Forge — Audit watchman and calibration falsifier

> Capability spec for the OpenSpec audit calibration watchman.
> Source of truth for thresholds, event schemas, and falsifier semantics.

## Overview

The Forge audit pipeline runs an OpenSpec audit before every `forge build`
dispatch. The **watchman** monitors whether the audit panel is correctly
calibrated by tracking two kinds of miscalibration:

- **False-positive (over-blocking):** the panel fires BLOCK on changes that
  are actually fine, forcing repeated `--ignore-audit` overrides.
- **False-negative (under-finding):** the panel consistently finds nothing
  real, meaning critical drift goes undetected.

## Two-sided falsifier

The falsifier is the decision component that consumes observed rates and
returns a `FalsifierVerdict`.

### Verdicts

| Verdict | Meaning |
|---|---|
| `MISCALIBRATED_FALSE_POSITIVE` | Panel is over-strict; too many real-good changes are being blocked or dismissed. |
| `MISCALIBRATED_FALSE_NEGATIVE` | Panel is too lenient; most audits find nothing of substance. |
| `OK` | All rates are within acceptable calibration bounds. |
| *(insufficient data)* | Fewer than `CALIBRATION_WINDOW` samples; returns `OK` until enough history exists. |

### Calibration window

All rate computations use a **sliding window of `CALIBRATION_WINDOW = 10`**
builds or audit reports. The falsifier does not fire until both data sources
(JSONL events and `.forge/audits/` reports) have at least 10 records.

### Thresholds

| Constant | Value | Triggers |
|---|---|---|
| `BLOCK_OVERRIDE_THRESHOLD` | 0.30 (30%) | If > 30% of the last 10 builds had a BLOCK verdict overridden via `--ignore-audit` → `MISCALIBRATED_FALSE_POSITIVE` |
| `DISMISSAL_RATE_THRESHOLD` | 0.60 (60%) | If > 60% of all findings across the last 10 audit reports were dismissed as false positives by the verifier → `MISCALIBRATED_FALSE_POSITIVE` |
| under-finding threshold | 0.70 (70%) | If fewer than 70% of the last 10 audit reports contained at least one verified finding → `MISCALIBRATED_FALSE_NEGATIVE` |

The under-finding threshold is derived as `1 - BLOCK_OVERRIDE_THRESHOLD`,
keeping the two-sided bounds symmetric around 50%.

### Precedence

False-positive checks run before the false-negative check. If a panel is
simultaneously over-blocking AND under-finding (unlikely in practice),
`MISCALIBRATED_FALSE_POSITIVE` is reported as the more actionable signal.

## Rate computation

### Block-override rate

Reads JSONL events from `~/.rapoport/events/YYYY-MM-DD.jsonl`:

- **Denominator:** last `CALIBRATION_WINDOW` `run.started` events with
  `cli='forge'` and `payload.command='build'`.
- **Numerator:** builds in that window whose `linearId` appears in an
  `audit_override` event (written by the build orchestrator when
  `--ignore-audit` bypasses a BLOCK verdict).

`blockOverrideRate = overrideCount / windowBuilds`

### Dismissal rate

Reads `.forge/audits/openspec-*-*.json` report files:

- **Denominator:** total findings across the last `CALIBRATION_WINDOW` reports.
- **Numerator:** findings where the verifier marked `verified: false`
  (false-positive dismissals).

`dismissalRate = dismissedFindings / totalFindings`

### Finding rate

Reads the same `.forge/audits/` report files:

- **Denominator:** count of reports in the last `CALIBRATION_WINDOW`.
- **Numerator:** reports containing at least one finding with `verified: true`.

`findingRate = reportsWithRealFindings / windowReports`

## Event schemas

### `audit_override` (JSONL)

Written to `~/.rapoport/events/YYYY-MM-DD.jsonl` by the build orchestrator
when `--ignore-audit[=<reason>]` is passed and the audit gate verdict is BLOCK.

```jsonc
{
  "event": "audit_override",
  "ts": "2026-06-04T12:00:00.000Z",
  "issueKey": "RAP-1279",
  "changeSlug": "false-positive-watchman",
  "reason": "calibration in progress"
}
```

## Source location

```
packages/forge/src/audit/
├── constants.ts     — threshold constants and CALIBRATION_WINDOW
├── events.ts        — JSONL reader and .forge/audits/ reader
├── falsifier.ts     — pure two-sided falsifier function
├── watchman.ts      — rate computation helpers + runWatchman() entry point
└── __tests__/
    ├── falsifier.test.ts
    └── watchman.test.ts
```

## Implementation notes

- All rate computation functions are pure (accept arrays, return numbers) so
  tests can inject synthetic fixtures without filesystem access.
- `runWatchman()` is the only impure entry point; it reads from disk and
  delegates to the pure helpers.
- The falsifier does not emit events or modify any state — it only reads and
  classifies. Downstream consumers (CLI output, CI summaries) are responsible
  for acting on the verdict.
- The `--ignore-audit` escape hatch is intentionally preserved: operators
  can still override BLOCK verdicts when needed. The watchman's role is to
  detect when overrides are happening *too often*, not to prevent individual
  overrides.
