Monitoring Ops

On first invocation, read references/orchestrator.md and follow its welcome protocol.

Post-deployment observability skill. Deploy-ops ships the code; oc-monitoring-ops watches it run. Covers five domains: uptime monitoring, error tracking, structured logging, alerting pipelines, and incident response.

This skill does NOT build features or deploy code — it instruments what’s already deployed and establishes the feedback loop that catches problems before users report them.

/oc-monitor — Command Reference

MONITORING OPS COMMANDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  SETUP
  /oc-monitor setup         Guided observability stack setup for a project
  /oc-monitor stack         Recommend monitoring tools for the stack
  /oc-monitor instrument    Add structured logging + error capture to codebase

  OBSERVE
  /oc-monitor health        Live health check — hit endpoints, report status
  /oc-monitor errors        Check error tracking service for recent issues
  /oc-monitor uptime        Show uptime status and recent incidents
  /oc-monitor metrics       Key metrics snapshot (latency, error rate, throughput)
  /oc-monitor status        Current monitoring state from checkpoint

  ALERT
  /oc-monitor alerts        Design or audit alerting rules
  /oc-monitor oncall        Set up on-call rotation and escalation
  /oc-monitor slo           Define or review SLOs/SLIs/error budgets

  RESPOND
  /oc-monitor incident      Start or review an incident response
  /oc-monitor runbook       Generate or update operational runbooks
  /oc-monitor postmortem    Structured post-incident review

  REPORT
  /oc-monitor dashboard     Design an ops monitoring dashboard (routes to oc-dash-forge)
  /oc-monitor report        Generate weekly/monthly ops report
  /oc-monitor audit         Full observability maturity assessment
  /oc-monitor compare       Compare two monitoring snapshots (drift detection)

  SESSION
  /checkpoint            Show checkpoint status
  /checkpoint show       Display full checkpoint JSON
  /checkpoint reset      Archive and restart

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Type any command to begin. /oc-monitor to see this again.

Session Persistence (Checkpoint Protocol)

Checkpoint: {project-dir}/.checkpoints/oc-monitoring-ops.checkpoint.json

Resume on Start

When any /oc-monitor command is invoked:

1. Check for checkpoint
2. If exists: show tier, tools, SLO status, active incidents, maturity grade, next action
3. Ask: “Continue, restart, or show full checkpoint?”
4. On continue: load context, resume from next_actions[0]

What’s Tracked

Monitoring-ops uniquely stores runtime state alongside pipeline progress — the last health check result, active incident count, and SLO budget consumption. This means the checkpoint serves double duty: session persistence AND operational snapshot.

How This Skill Fits the Pipeline

oc-reverse-spec → oc-app-architect → oc-git-ops → oc-deploy-ops → MONITORING-OPS
                                                            │
                                              ┌─────────────┤
                                              │             │
                                         oc-security-auditor   oc-scale-ops
                                         (detection/        (perf
                                          response input)    budgets)

oc-deploy-ops ships it, oc-monitoring-ops watches it. The handoff:

1. oc-deploy-ops completes production promotion
2. oc-deploy-ops runs health check (basic HTTP 200 verification)
3. If oc-monitoring-ops checkpoint exists: oc-monitoring-ops takes over ongoing observation
4. If not: oc-deploy-ops suggests /oc-monitor setup for the project

oc-deploy-ops’s health check is a one-shot verification. oc-monitoring-ops provides continuous observation, error aggregation, alerting, and incident coordination.

Cross-Skill Connections

Observability Maturity Model

Every project maps to a maturity tier. The setup wizard determines the tier from the project’s scale, sensitivity, and infrastructure.

Default for aidops-scale apps: T0 or T1. Don’t overengineer monitoring for a 2-user app. The setup wizard auto-detects the appropriate tier.

Phase 1: Setup (/oc-monitor setup)

Guided setup that instruments a project for observability. Reads existing config (oc-deploy-ops checkpoint, wrangler.toml, package.json) to avoid asking questions with known answers.

Setup Wizard

Use ask_user_input for remaining unknowns:

1. What tier? (auto-detect from oc-scale-ops/oc-deploy-ops context, confirm)
2. Error tracking provider? (Sentry, LogRocket, Highlight, BetterStack, or console-only)
3. Uptime monitoring? (BetterStack, UptimeRobot, Checkly, or CF Health Checks)
4. Alerting channel? (Telegram, Slack, Discord, email, PagerDuty)
5. Status page? (BetterStack, Instatus, or none)

Setup Deliverables

Health Endpoint Contract

Every monitored app needs a GET /api/health endpoint. This is the shared contract between oc-deploy-ops (one-shot check), oc-monitoring-ops (continuous check), and external uptime monitors.

Response shape:

{
  "status": "ok | degraded | unhealthy",
  "checks": { "database": "ok", "kv": "ok" },
  "version": "1.2.3",
  "timestamp": "2026-04-23T10:00:00Z"
}

Rules:

• Returns 200 when all checks pass, 503 when any check is “down”
• Each dependency (DB, KV, external API) gets its own named check
• Include deployed version for deploy-correlation
• Include per-check latency at T2+ (for oc-scale-ops integration)

Read references/instrumentation.md for full implementation patterns per stack (Workers, Next.js, FastAPI), including structured logger, request monitoring middleware, and Sentry transport for Workers.

Phase 2: Observe

Health Check (/oc-monitor health)

Goes deeper than oc-deploy-ops’s one-shot 200 check:

1. HTTP status — hit /api/health, verify 200
2. Per-dependency status — parse response body, check each named service (DB, KV, etc.)
3. Latency profile — 5 samples, report avg and max
4. Certificate expiry — TLS cert days remaining
5. Version correlation — compare deployed version to last-known-good

Output uses the Health Check Report format from references/output-templates.md.

Error Tracking (/oc-monitor errors)

Checks the configured error tracking service. Adapts to what’s available:

Metrics Snapshot (/oc-monitor metrics)

Aggregate key metrics from available sources:

PRODUCTION METRICS — [project]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Uptime (30d):    99.87%  (2h 17m total downtime)
  Error rate:      0.3%    (12 errors / 4,200 requests today)
  p50 latency:     42ms
  p95 latency:     187ms
  p99 latency:     890ms
  Active errors:   3 unresolved (1 HIGH, 2 LOW)
  Last deploy:     2 days ago (v1.2.3)
  Next cert expiry: 47 days
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Status (/oc-monitor status)

Reads the checkpoint and displays the current monitoring state without hitting any live endpoints. This is a checkpoint read, not a health check.

MONITORING STATUS — [project]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Tier:          T1 (Foundations)
  Maturity:      C (50%)
  Tools:         Sentry + BetterStack + Telegram
  SLOs:          99% avail / 1s p95 / 1% errors
  Budget:        86% remaining (24 days left in window)
  Last health:   2 min ago — ✅ ok (42ms)
  Incidents:     0 active
  Runbooks:      3 of 6 alert-mapped
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Phase 3: Alert (/oc-monitor alerts)

Alert Design Principles

1. Every alert must have an action. If you can’t do anything when it fires, it’s noise, not an alert. Remove it.
2. Two thresholds: warn and critical. Warn = investigate soon. Critical = act now.
3. Alert on symptoms, not causes. “Error rate > 5%” is better than “database connection pool exhausted” — the symptom catches more failure modes.
4. Pair every alert with a runbook. The alert says “something’s wrong.” The runbook says “here’s what to do.”

Standard Alert Set (T1+)

SLO Definition (/oc-monitor slo)

Service Level Objectives quantify “good enough.” They prevent over-engineering reliability for apps that don’t need five nines.

Error budget = 100% − SLO. If your SLO is 99% availability, you have 7.3 hours of downtime per month before you violate it. Spend that budget on deploys and feature velocity, not perfection.

Read references/alerting-patterns.md for channel setup (Telegram, Slack, PagerDuty), alert transport code, and SLO burn-rate alerting math.

Phase 4: Respond (/oc-monitor incident)

Read references/incident-response.md for full incident template, runbook template, and post-mortem template.

Incident Lifecycle

ALERT FIRES
    │
    ▼
ACKNOWLEDGE (who's looking at it?)
    │
    ▼
DIAGNOSE (what's broken? use runbook decision tree)
    │
    ├──► QUICK FIX (restart, rollback, scale) ─────┐
    │                                               │
    ├──► FIX FORWARD (patch + deploy) ──────────────┤
    │                                               │
    └──► ESCALATE (beyond current expertise) ───────┤
                                                    ▼
                                      MITIGATE (service restored)
                                                    │
                                                    ▼
                                      RESOLVE (root cause addressed)
                                                    │
                                                    ▼
                                      POST-MORTEM (blameless, action items)

Key Files

Observability Audit (/oc-monitor audit)

Full maturity assessment. Scores each domain and recommends the next upgrade.

Audit Domains

Tier-adaptive grading: At T0, incident response and alerting carry reduced weight (a solo dev doesn’t need PagerDuty). At T2+, all five domains are equally critical. The audit grades against the project’s declared tier, not against T3 universally.

Maturity Scoring

See references/output-templates.md for full audit report format, ops report template, and metrics snapshot format.

Dashboard Routing (/oc-monitor dashboard)

When the user wants a monitoring dashboard, route to oc-dash-forge with ops archetype pre-selected:

Detected monitoring dashboard request. Routing to oc-dash-forge with ops archetype.

oc-dash-forge will produce:
  - High-density ops layout (dark mode, tight tiles)
  - Real-time KPI tiles (error rate, latency, throughput, uptime)
  - Incident table, event feed, throughput chart
  - Working React prototype with mock data

Proceeding to /oc-data-forge with ops context...

Package the following context for oc-dash-forge:

• Archetype: ops (pre-selected, skip interview)
• Data sources: health endpoint, error tracking, uptime service, structured logs
• KPIs: error rate, p95 latency, uptime %, active incidents, deploy recency
• Refresh cadence: near-real-time (30s-2min)

Ops Report (/oc-monitor report)

Generate a periodic operations summary covering availability, performance, errors, SLO status, deploys, and incidents. Cadence depends on tier: T1 = monthly, T2+ = weekly.

The report pulls from: oc-monitoring-ops checkpoint (SLO budget, incident count), oc-deploy-ops checkpoint (deploy count, rollbacks), and error tracking service (error counts, top issues).

See references/output-templates.md for the full report template.

Snapshot Comparison (/oc-monitor compare)

Compares two monitoring snapshots to track observability posture over time. Mirrors oc-security-auditor’s /oc-security compare pattern.

Input: two checkpoint timestamps or dates. If one argument, compares against current state.

Diffs: maturity grade, domain scores, SLO budget consumption, incident count, runbook coverage. Highlights regressions prominently. Uses the comparison format from references/output-templates.md.

File Structure

project-dir/
├── .checkpoints/
│   └── oc-monitoring-ops.checkpoint.json
├── .monitoring.json                 # Tool config (URLs, check IDs, channels)
├── monitoring/
│   ├── config.md                    # Strategy: what's monitored, thresholds, tools
│   ├── runbooks/
│   │   ├── downtime.md
│   │   ├── error-spike.md
│   │   ├── deploy-failure.md
│   │   ├── latency.md
│   │   ├── cert-renewal.md
│   │   └── error-budget.md
│   ├── incidents/
│   │   └── INC-YYYY-MM-DD-NNN.md
│   └── postmortems/
│       └── PM-YYYY-MM-DD-NNN.md
├── src/
│   ├── lib/
│   │   ├── logger.ts                # Structured logging utility
│   │   ├── alerting.ts              # Alert dispatch (Telegram/Slack/PagerDuty)
│   │   └── sentry.ts               # Error tracking transport (Workers)
│   └── middleware/
│       └── monitoring.ts            # Request tracking, timing, error capture

Checkpoint Integration

Location

{project-dir}/.checkpoints/oc-monitoring-ops.checkpoint.json

When to Write

progress_table

[
  { "id": "setup",      "label": "Observability setup",    "status": "not_started" },
  { "id": "logging",    "label": "Structured logging",     "status": "not_started" },
  { "id": "errors",     "label": "Error tracking",         "status": "not_started" },
  { "id": "uptime",     "label": "Uptime monitoring",      "status": "not_started" },
  { "id": "alerts",     "label": "Alerting pipeline",      "status": "not_started" },
  { "id": "slos",       "label": "SLO definition",         "status": "not_started" },
  { "id": "runbooks",   "label": "Operational runbooks",   "status": "not_started" },
  { "id": "dashboard",  "label": "Monitoring dashboard",   "status": "not_started" }
]

skill_state

{
  "tier": "T1",
  "tools": {
    "error_tracking": "sentry",
    "uptime": "betterstack",
    "alerting": "telegram",
    "logging": "structured-json"
  },
  "slos": {
    "availability": "99%",
    "latency_p95": "1s",
    "error_rate": "1%"
  },
  "last_health_check": {
    "at": "2026-04-22T10:00:00Z",
    "status": "ok",
    "latency_ms": 42
  },
  "active_incidents": 0,
  "runbook_count": 3,
  "maturity_grade": "C"
}

Cross-Skill Reads

Tool Recommendations by Stack

Cloudflare Workers (aidops default)

Vercel / Next.js

General (any stack)

PM-Tool MCP Integration (v1.3+)

oc-monitoring-ops opens incident tickets in the PM tool when alerts fire, and back-references them through resolution.

The runtime contract — concrete tool names, retry policy, idempotency markers, the pm_deferred_actions[] schema, and the extended state vocabulary (resolved-pending-postmortem) — lives in oc-integrations-engineer/references/pm-mcp-protocol.md. All MCP calls below honour that contract; this section says only how oc-monitoring-ops shapes incidents and per-event updates.

On alert fire

1. Look up the alert in the runbook registry (configured per alert id; see runbooks/).

2. Compose incident description, prefixed with the idempotency marker per protocol §3:

   <!-- opchain:oc-monitoring-ops:incident-fired:<alert-event-id> -->
   
   Alert: {alert-name} ({severity})
   Fired at: {iso-timestamp}
   Service: {service}
   Symptoms: {first-three-symptoms-from-alert-payload}
   Runbook: {runbook-url}
   On-call: {on-call-engineer-from-pagerduty}
   Recent deploys: {list-of-deploys-in-last-2h-from-deploy-ops-checkpoint}

3. Pre-create check: call the registry-resolved list_issues tool (Linear: mcp__claude_ai_Linear__list_issues; GitHub: mcp__mcp-server-github__list_issues) filtered to the configured project + the incident issue type from pm.yaml.issue_types, description-text query for the marker. If a match exists, reuse the existing incident id (mid-burst alert dedupe).

4. Otherwise call the registry-resolved create_issue tool (Linear: mcp__claude_ai_Linear__save_issue with no id; GitHub: mcp__mcp-server-github__issue_write action=create) with:

   • issue_type from pm.yaml.issue_types.incident (default “Incident”).
   • priority from alert severity mapping.
   • labels from runbook (incident, service:<name>, severity:<level>), merged with pm.yaml.labels_default.
   • parent / blocked-by relation to the most recent deploy ticket if one is open (likely culprit) — read from oc-deploy-ops.checkpoint.json skill_state.pm.deploy_tickets[].

5. Record incident id in oc-monitoring-ops.checkpoint.json skill_state.pm.incidents[] with the correlating Sentry / PagerDuty event id.

Per-event updates during the incident

Each row uses a unique idempotency marker. Pre-write check via list_comments (Linear) or issue_read (GitHub) before every add_comment. State strings come from pm.yaml.states / pm.yaml.states.extended — never hard-coded.

Postmortem back-reference

When the postmortem is written, oc-monitoring-ops appends an action-item sub-ticket per remediation item, parent = the incident ticket. Each remediation sub-ticket carries marker <!-- opchain:oc-monitoring-ops:remediation:<incident-id>:<item-n> --> in its description and is created via the create_issue tool with the pre-create check pattern above. Each remediation sub-ticket is assigned to the owning team’s default assignee (from pm.yaml.remediation_owners map), with a target close date.

Alert hygiene

If an alert fires more than N times in 24h (default 5; configurable per alert), oc-monitoring-ops adds a noisy-alert label (per protocol Appendix A — labels, not state) to the incident and surfaces a tuning recommendation rather than spamming new tickets — same incident gets new comments, not new tickets. The pre-create check in step 3 above ensures this naturally for in-window alerts.

/oc-monitor --retry-pm flush

Invokes the protocol §4 flush against oc-monitoring-ops.checkpoint.json pm_deferred_actions[]. Filter to skill: "oc-monitoring-ops" and retriable: true. Critical alerts should NOT depend on flush succeeding — Telegram / PagerDuty fire regardless; the flush is reconciliation only.

Failure modes

• MCP unavailable → alert fires through Telegram / PagerDuty as always; intended PM-MCP write is deferred per protocol §4. Operator visibility is unaffected.
• PM provider rate-limits during a major incident burst → batch into a single rollup ticket via the marker dedupe in step 3 (the same alert-event-id naturally folds into one incident); separate comments per alert event with marker <!-- opchain:oc-monitoring-ops:burst-event:<incident-id>:<event-n> -->.
• 403 on incident creation → defer with retriable: false; surface to the on-call channel as a side message; never silently widen scope.

Principles

1. Monitor symptoms, not implementations. Alert on error rate, not on which specific database query failed. Symptoms catch failures you didn’t predict.
2. Every alert needs a runbook. An alert without instructions is just noise.
3. Error budgets over perfection. 99% uptime is fine for aidops-scale apps. That’s 7 hours of acceptable downtime per month. Use that budget for shipping.
4. Structured logs are non-negotiable. console.log("error") tells you nothing in production. JSON with request ID, user context, and timestamps tells you everything.
5. Proportional investment. T0 for personal apps, T3 for revenue-bearing products. Don’t build PagerDuty rotation for a 2-user fitness tracker.
6. Post-mortems are investments, not punishments. Every incident you learn from is one that won’t repeat. Blameless or don’t bother.
7. Observability is continuous. Setup is phase 1, not the finish line. Audit quarterly, refine alerts monthly, run health checks daily.