Prompt Ops

On first invocation, read references/orchestrator.md and follow its welcome protocol.

Treat prompts as code: source-controlled, diffable, semver’d, and gated on an eval suite the same way application code is gated on tests. A prompt is the single most behavior-defining string in an LLM app — and the one most teams edit live, in a console, with no version history and no way to tell whether “it got better” is real or vibes. This skill makes a prompt change a reviewable diff with a measured score delta attached.

This is not a tri-agent harness. It’s an operations layer the model-facing skills build on: oc-claude-api owns the request surface (model routing, caching, tool wiring), oc-agent-forge owns agent topology, oc-rag-forge owns retrieval. Prompt Ops owns the part underneath all three — the prompt text, the eval datasets that score it, and the regression gate that stops a “small wording tweak” from quietly tanking quality on a migration.

opchain dogfoods this skill on itself. The worked example referenced throughout is prompts/opchain-eval/ — opchain’s own eval set (inputs.jsonl, expected.jsonl, eval.yaml), published in Sprint 3 as the canonical /oc-prompt eval artifact. Wherever this doc says “the eval set”, that directory is the live instance. It is flat and holds only eval files (plus a README) at the top level (no eval/ subdir, no vX.Y.Z/prompt.md, no CHANGELOG.md, no baseline.json yet), because the “prompt” under test is opchain’s routing surfaces (skill description: frontmatter + the orchestrator routing table), not a prompt.md.

Model facts come from oc-claude-api / the claude-api skill, not memory. Judge-model choice, model IDs, and Batch-API economics in this skill are sourced there. Current models: Fable 5 (claude-fable-5), Opus 4.8 (claude-opus-4-8), Sonnet 4.6 (claude-sonnet-4-6), Haiku 4.5 (claude-haiku-4-5). When a prompt is pinned to a model, pin it to one of these exact IDs.

/oc-prompt — Command Reference

PROMPT OPS COMMANDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  PROMPT-AS-CODE
  /oc-prompt              Version / organize a prompt set (prompt-as-code layout)
  /oc-prompt diff         Diff two prompt versions + their eval-score deltas

  EVALUATION
  /oc-prompt eval         Run a prompt version against an eval dataset → scorecard
  /oc-prompt goldset      Build or extend the eval goldset (inputs/expected/rubric)
  /oc-prompt judge        Configure / calibrate the LLM-as-judge grader

  REGRESSION / DRIFT
  /oc-prompt regress      Re-run the eval suite and gate on score regression
  /oc-prompt baseline     Freeze the current scores as the regression baseline
  /oc-prompt drift        Re-run the frozen baseline to detect prompt/model drift

  UTILITIES
  /checkpoint             Show checkpoint status

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Type any command to begin. /oc-prompt to see this again.

Executable evaluation commands

Run these commands from the opchain source checkout with dependencies installed, or from the unpacked local runtime artifact. A skills-only ZIP or the Claude plugin does not include these npm commands. Keep the runtime in its own directory; pass dataset and output paths for the target project explicitly.

The runner-backed forms are intentionally narrow. With the coordinator-provided oc-prompt alias, use these exact forms:

npm run oc-prompt -- run <dataset-dir> --adapter http-json --endpoint <url> --api-key-env <ENV_NAME> --out <result.json>
npm run oc-prompt -- baseline <dataset-dir> --result <result.json> --out <baseline.json>
npm run oc-prompt -- regress <dataset-dir> --result <result.json> --baseline <baseline.json>

They implement /oc-prompt eval, /oc-prompt baseline, and /oc-prompt regress respectively. Missing adapter configuration, credentials, a judge verdict, or an interrupted call writes a BLOCKED result; no provider is chosen implicitly. /oc-prompt, diff, goldset, judge, and drift remain assistant-driven modes in this release.

How This Skill Fits the Build Pipeline

oc-claude-api (model routing) ──pins the model an eval runs against──┐
                                                                     │
oc-prompt-ops owns:                                                  ▼
   prompts/<name>/<version>/prompt.md        ┌────────────────────────────┐
   prompts/<name>/eval/inputs.jsonl     ────►│  /oc-prompt eval           │
   prompts/<name>/eval/expected.jsonl        │  run prompt × goldset       │
   prompts/<name>/eval/eval.yaml             │  grade → scorecard          │
                                             └─────────────┬──────────────┘
                                                           │ score delta
                                       ┌───────────────────┴────────────────┐
                                       ▼                                     ▼
                          /oc-prompt regress (PR gate)         /oc-prompt drift (scheduled)
                          block merge on score drop            flag model/prompt drift
                                       │
              ┌────────────────────────┼────────────────────────┐
              ▼                        ▼                         ▼
        oc-agent-forge          oc-rag-forge             oc-claude-api migrate
        (prompts it ships)      (gen-prompt eval)        (eval-gate the diff PR)

The skill’s output is an evaluated, versioned prompt plus a regression suite. oc-agent-forge (trajectory fixtures) and oc-rag-forge (retrieval goldset) run their own suites side by side with this one; what they bring here is any single prompt they ship (an agent’s system prompt, RAG’s generation prompt), registered and scored like any other. oc-claude-api migrate gates its model-migration diff on /oc-prompt drift showing no regression on the new model.

Principle 1: Prompt-as-Code

A prompt that lives only in a SaaS console or interpolated inside a function is an undiffable, unversioned production dependency. Prompt Ops moves it into the repo and treats it like any other source artifact.

The convention (full layout in references/prompt-versioning.md):

prompts/
└── <prompt-name>/
    ├── CHANGELOG.md            # one entry per version, with the eval delta
    ├── v1.0.0/
    │   └── prompt.md           # the prompt text — the only source of truth
    ├── v1.1.0/
    │   └── prompt.md
    └── eval/
        ├── inputs.jsonl        # eval inputs (one JSON object per line)
        ├── expected.jsonl      # expected outputs / grading targets, keyed by id
        └── eval.yaml           # rubric: which grader, thresholds, judge config

Four rules make it “code”:

1. Source-controlled. The prompt text lives in prompt.md, reviewed in a PR like any other change. No live console edits — those are invisible to git and to the eval gate.
2. Diffable. A prompt change is a line diff. /oc-prompt diff v1.0.0 v1.1.0 shows the text delta and the score delta side by side, so a reviewer sees both what changed and what it did.
3. Lockstep-versioned. The prompt and the eval set version together. A prompt bump that needs new eval cases bumps both in the same commit, so a version is always evaluable against its own contemporaneous goldset.
4. Semver’d by behavior. PATCH = typo/format fix, no score movement expected. MINOR = capability added, scores should hold or rise. MAJOR = behavior changed such that the old expected outputs no longer apply (the goldset itself changes). See references/prompt-versioning.md.

The model the prompt runs against is part of the version contract: a prompt pinned to claude-opus-4-8 is a different evaluable artifact than the same text on claude-haiku-4-5. The pinned model ID comes from oc-claude-api’s routing decision and is recorded in eval.yaml.

Principle 2: Eval Datasets (the goldset)

You cannot tell whether a prompt change helped without a fixed set of cases to score it on. That set — the goldset — is three files. (Full format and a small worked example in references/eval-datasets.md; the live instance is prompts/opchain-eval/.)

inputs.jsonl — one JSON object per line, each an eval case with a stable id and the input the prompt runs on:

{"id": "route-001", "input": "build me a recipe app"}
{"id": "route-007", "input": "set up a RAG pipeline over our internal docs"}

expected.jsonl — the grading target for each id. mode selects the grader; the rest of the object is that grader’s config:

{"id": "route-001", "expect": {"mode": "contains", "all": ["oc-app-architect", "/oc-discover"]}}
{"id": "route-007", "expect": {"mode": "contains", "all": ["oc-rag-forge", "/oc-rag"]}}

eval.yaml — the suite config: the default grader, the judge config for llm_judge cases, and the pass/regression thresholds:

prompt: opchain-routing
model: claude-opus-4-8         # the model the prompt-under-test runs on (from oc-claude-api)
grading:
  default_mode: contains       # exact | contains | llm_judge (a case may override via its own mode)
  judge:
    model: claude-opus-4-8     # judge ≥ capability of the model under test
    output_format: json_schema # force a {score, reason} verdict — not free text
thresholds:
  pass_rate: 0.90              # ≥ 90% of cases must pass
  regression_epsilon: 0.05     # a pass_rate drop > this vs baseline fails the gate
cost:
  cost_per_eval: null          # measured by oc-cost-ops /oc-cost attribute (null until first costed run)

Grading modes

LLM-as-judge rules (detail in references/eval-datasets.md):

• The judge model should be at least as capable as the model under test — default to claude-opus-4-8; never grade Opus output with Haiku.
• Force a structured verdict via output_config.format (a {score, reason} JSON schema) so the grade is parseable and the gate is deterministic, not a free-text “looks good”.
• The judge is itself a prompt — it lives under prompts/<name>/eval/ and is versioned and calibrated like any other. Re-run /oc-prompt judge to re-calibrate against a hand-graded sample when the judge drifts.

Avoiding train/eval leakage

The goldset is a held-out test set. If you tune a prompt by staring at the eval cases and editing until they pass, you’ve overfit — the scores are memorised, not earned. Keep a few-shot/example pool separate from the goldset, never copy a goldset case into the prompt’s examples, and grow the goldset from real production failures, not from cases you already know the prompt handles.

Principle 3: Regression Gating

Once a prompt has a goldset and a passing score, freeze that score as a baseline and gate every future prompt change on not regressing below it.

npm run oc-prompt -- baseline prompts/model-routing/eval --result result.json --out baseline.json
... edit prompts/model-routing/v1.2.0/prompt.md ...
npm run oc-prompt -- run prompts/model-routing/eval --adapter http-json --endpoint <url> --api-key-env PROVIDER_KEY --out result.json
npm run oc-prompt -- regress prompts/model-routing/eval --result result.json --baseline baseline.json

The regress gate (run on every PR that touches prompts/; /oc-prompt regress is agent-driven — no CI runner script ships with this skill):

PROMPT REGRESSION — model-routing  v1.1.0 → v1.2.0
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
case mode        baseline   new     Δ
pass_rate        0.93       0.95    +0.02   ✓
route-001 judge  1.0        1.0      0.00   ✓
route-014 judge  1.0        0.0     -1.00   ✗  REGRESSION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
VERDICT: FAIL — route-014 regressed (long-context routing now picks Haiku).
aggregate +0.02 but a per-case regression beyond epsilon blocks the merge.

Two failure conditions, both blocking:

• Aggregate regression: pass_rate drops more than regression_epsilon below baseline.
• Per-case regression: any individual case that passed at baseline now fails — even if the aggregate went up. A change that lifts the average while breaking a previously-working case made that case worse, and the table names it.

On a deliberate behavior change (a MAJOR prompt bump), the baseline is re-frozen with /oc-prompt baseline after the new expected outputs land — you don’t bypass the gate, you move it forward intentionally with a CHANGELOG entry recording the new scores.

Principle 4: Drift Detection

Regression gating catches your prompt edits. Drift is the score moving when you didn’t touch the prompt — because the model under it changed, or the world the prompt describes did. (Full playbook in references/drift-detection.md.)

Two drift sources:

• Model drift. The same prompt scores differently on a new model version. A prompt tuned for claude-opus-4-7 may over- or under-trigger on claude-opus-4-8 (more narration, different tool-reach defaults). /oc-prompt drift re-runs the frozen baseline goldset against the newly-pinned model and reports the per-case delta — this is the exact signal oc-claude-api migrate gates its migration diff on.
• Prompt drift. The prompt text didn’t change but its dependencies did — a referenced doc, a tool name, an enum value — so cases that used to pass now fail. The goldset catches this on the scheduled /oc-prompt drift run before a user does.

/oc-prompt drift runs on a schedule (or on a model-version bump), compares the live score to the frozen baseline, and:

The principle: a score change is never silent. Either you changed the prompt (a diff, gated) or something underneath it changed (drift, detected) — there is no third category.

/oc-prompt diff — prompt + eval delta

diff is the reviewer’s view of a prompt change. It shows the text diff and the eval-score diff in one place, so “I reworded the system prompt” always arrives with “and here’s what it did to the scores”:

/oc-prompt diff model-routing@v1.1.0 model-routing@v1.2.0
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 prompt.md
- Route long-context tasks to a high-capability model.
+ Route tasks over 200k tokens to claude-opus-4-8 or claude-fable-5.

 eval scorecard (goldset: 28 cases)
   pass_rate   0.93 → 0.95   +0.02
   regressions 1   (route-014: long-context → Haiku)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

A prompt diff with no attached eval delta is incomplete — diff runs the eval if a scorecard for either version is stale. Versioning + diffing detail lives in references/prompt-versioning.md.

Cost: oc-cost-ops collaboration (v1.6)

Running an eval suite costs tokens — every case is at least one model call, and llm_judge grading doubles that (the call under test plus the judge call). For large goldsets, route eval runs through the Batch API (50% of standard price, sourced from oc-claude-api) — eval is latency-tolerant by definition, so batch is the right default.

eval.yaml carries cost_per_eval / budget_per_eval / regression_pct, owned by oc-cost-ops (shipped v1.6). cost_per_eval is measured from real token counts by /oc-cost attribute — it stays null until the first costed run (Cost Ops attributes, it does not estimate). oc-cost-ops runs its cost-regression gate, /oc-cost gate, beside this skill’s score gate, /oc-prompt regress; this skill does not invoke it. Both are agent-driven PR-time checks, so a prompt change that holds quality but triples token cost gets a FAIL verdict on the PR alongside the score (see oc-cost-ops/references/budget-gates.md).

# eval.yaml — not the checkpoint's top-level `cost` block (that one is oc-cost-ops's)
cost:
  cost_per_eval: null    # measured by oc-cost-ops /oc-cost attribute (null until first costed run)

Boundaries (what oc-prompt-ops does NOT own)

Prompt Ops owns the prompt artifact and its evidence — the versioned text, the goldset, the baseline, the regression and drift gates. Siblings route, run, deploy, and cost it.

Cross-Skill Integration

Checkpoint Integration

The shared checkpoint schema, write rules and resume protocol live in references/checkpoint-protocol.md, bundled with this skill. This section adds only what is specific to oc-prompt-ops.

Checkpoint Location

{project-dir}/.checkpoints/oc-prompt-ops.checkpoint.json

When to Write

skill_state

{
  "prompt": "model-routing",
  "version": "1.2.0",
  "pinned_model": "claude-opus-4-8",
  "goldset": {
    "path": "prompts/opchain-eval",
    "cases": 28,
    "grading_modes": ["contains", "llm_judge"],
    "judge_model": "claude-opus-4-8"
  },
  "baseline": {
    "frozen_at_version": "1.1.0",
    "pass_rate": 0.93
  },
  "last_eval": {
    "version": "1.2.0",
    "pass_rate": 0.95,
    "regressions": ["route-014"],
    "verdict": "FAIL"
  },
  "last_drift": {
    "against_model": "claude-opus-4-8",
    "from_model": "claude-opus-4-7",
    "delta": 0.01,
    "verdict": "PASS"
  }
}

Cross-Skill Reads

Principles

1. A prompt is code. Source-controlled, diffable, reviewed in a PR, gated on an eval suite. Live console edits are invisible to git and to the gate — they don’t happen here.
2. No score claim without a goldset. “It got better” is a measurement against a fixed set of held-out cases, or it’s a vibe. Every prompt ships with inputs.jsonl / expected.jsonl / eval.yaml.
3. Version prompt and eval in lockstep. A version is always evaluable against its own contemporaneous goldset. Bump both in one commit.
4. Semver by behavior. PATCH = no expected score movement, MINOR = scores hold or rise, MAJOR = expected outputs change. The bump kind tells the reviewer what the eval should show.
5. A diff carries its score delta. /oc-prompt diff shows the text change and what it did to the scores, together. A prompt change reviewed without its eval delta is reviewed blind.
6. Gate on per-case regression, not just the average. A change that lifts the aggregate while breaking a previously-passing case made that case worse. Name the case and block the merge.
7. Pin the model — it’s part of the version. The same text on a different model is a different evaluable artifact. The pinned ID comes from oc-claude-api and lives in eval.yaml.
8. Judge with a model at least as capable, and force a structured verdict. Default the judge to claude-opus-4-8; never grade Opus output with Haiku; make the grade a {score, reason} JSON object, not free text.
9. No train/eval leakage. The goldset is held out. Don’t copy goldset cases into the prompt’s examples, and grow the goldset from real production failures, not from cases the prompt already passes.
10. Every score change is explained. Either you changed the prompt (a gated diff) or something underneath changed (detected drift). There is no silent third category — and that is the whole point of the harness.