Field Notes — 001

The Tool That Admits What It Doesn't Know

2026-07-20 · 7 min read · v0.8.0 release

Most infrastructure tools tell you what's broken. CrisisMode was built to tell you what to do about it — and, as of v0.8.0, what's about to break. But the design decision I want to talk about is a stranger one: we spent a lot of engineering effort making sure this tool refuses to answer questions it can't actually answer.

Where this started

Getting an app to production has never been easier. Maybe you built yours with an AI pair over a weekend — Next.js on a serverless platform, a managed Postgres, a Redis cache. It works. People are using it. Congratulations, sincerely: you're now running production infrastructure.

Here's what nobody tells you at that moment: the skills that got you to production are not the skills that keep you there. When something breaks at 3am, the platform dashboard says "unhealthy," the logs are a firehose, and every action you might take carries a question you can't answer: will this make it worse? Seasoned SREs know this feeling too — they've just been burned enough times to move slowly. The worst moment to improvise is exactly the moment you're forced to.

CrisisMode exists for that moment. The premise: during an incident, the cost of a wrong action is at its highest, so safety can't be a feature you add — it has to be the frame everything else hangs inside.

Agents propose. The framework disposes.

CrisisMode ships recovery agents for the systems that actually take apps down: PostgreSQL replication, Redis memory pressure, Kafka brokers, etcd consensus, Kubernetes cascades, DNS, TLS expiry, stuck migrations, queue backlogs, bad deploys. Each agent diagnoses from live evidence and produces a recovery plan — a linear, human-readable sequence of steps.

But agents never touch your systems directly. Every plan passes through a safety kernel that enforces the rules an exhausted human at 3am might skip:

State capture before mutation. Any risky step must snapshot what it's about to change. No snapshot, no execution.
A rollback strategy, always. Plans without one are rejected by the validator, not frowned at in code review.
Declared blast radius. Every action states what it can affect, and the framework holds it to that.
Five escalation levels. From observe (read-only) through diagnose, suggest, repair-safe, and repair-destructive — and the destructive levels always keep a human in the loop. There is deliberately no mechanism to skip approval under pressure.

If you're newer to ops: this means you can run crisismode scan or crisismode diagnose with zero risk — they only read. If you're a veteran: yes, dry-run is the default, mutations are opt-in per escalation level, and every execution leaves an immutable forensic record.

Speaking SRE — literally

Incident response is a team sport with existing tooling, so CrisisMode also speaks the emerging SRE incident-bundle format: pipe an evidence bundle to crisismode bundle respond and it returns ranked root-cause hypotheses and policy-gated recovery actions as a structured response. Your incident tooling collects the evidence; CrisisMode reasons over it. It's scored against the open compatibility benchmark for that ecosystem — we publish how we do, including where we fall short. (That last part is a theme. Hold that thought.)

v0.8.0: asking the question before the incident

Everything above is reactive — it assumes the pager already went off. The newest capability flips the timeline. crisismode readiness answers one question: will this stack break under load?

crisismode readiness — sample output

$ crisismode readiness Scale readiness: at-risk (score 80/100) 4 rules evaluated, 1 could not run ⚠ Connection headroom [at_risk] 72 of 100 connections in use (72%) Fix: close idle connections or add a connection pooler before traffic grows ✓ Long-running transactions [ready] ✓ Missing indexes [ready] · Slow queries [unknown] could not run: pg_stat_statements is not available — enable it with CREATE EXTENSION pg_stat_statements Capacity ceilings (upper bounds — real capacity is lower): Database concurrent queries: at most 100 connections [declared: max_connections = 100 (declared)] Could not assess: db-throughput — mean query time unavailable (pg_stat_statements absent or empty)

It checks the failure modes that quietly kill working apps at scale: connection headroom (are you creeping toward your database's connection limit?), tables big enough that a missing index will hurt, transactions left idle mid-work, queries that are slow on average, and the serverless-plus-direct-connection trap that bites nearly everyone deploying functions against a small Postgres. Every finding comes with a plain-English explanation and a concrete fix — you don't need to know what "idle in transaction" means to act on the report, and if you do know, the exact thresholds are documented and quoted from the source constants.

Then it goes further and computes your capacity ceilings — how much load the stack can take. And this is where the design gets opinionated, because capacity numbers are exactly where tools love to lie.

The honesty contract

Every ceiling CrisisMode reports is labeled with its evidence class: declared (a limit your system is configured with, like max_connections = 100), measured (derived from observed data on your live system), or typical (a cited range for systems like yours — explicitly flagged as not a measurement). Throughput bounds come from Little's law applied to your actual mean query time, presented as an "at most" with the practical wall called out: systems degrade well before their ceiling.

The contract has teeth in four places:

A check that can't run reports unknown with the reason — it never guesses.
Unknowns never move your score. A tool that punished missing data would teach itself to fabricate.
A ceiling that can't be computed is omitted with an explanation, never estimated silently.
If the database is unreachable, you get an honest report saying exactly that — carrying the real connection error, not a fabricated "all clear" or a crash.

And we don't just claim this — we attack it. CrisisMode's torture harness runs twenty scenarios against real infrastructure: real Postgres, real injected failures, real assertions on the output. Two of them exist specifically to torture the readiness honesty contract: one injects genuine at-risk conditions and demands detection (with a false-alarm guard — a healthy stack must read ready), and one stops the database mid-run and verifies CrisisMode admits what it can no longer see.

The week we shipped this, the harness caught a regression in our own fix for a bug the harness had caught. That's the point. Trust isn't built by always having an answer — it's built by knowing which answers you can bet on.

One more thing for the AI-native crowd

If an AI coding agent helped you build your app, it can help you run it: crisismode mcp exposes eight read-only diagnosis tools over the Model Context Protocol — scan, diagnose, readiness, and more. Ask Claude or Cursor "will my stack survive launch?" and it can go find out, with real data. The MCP surface never mutates infrastructure, by design.

Get started

v0.8.0 ships as signed single binaries for Linux and macOS — no runtime to install. Two commands to a first opinion about your stack:

install

curl -fsSL https://github.com/trs-80/crisismode/releases/latest/download/crisismode-darwin-arm64 -o crisismode chmod +x crisismode ./crisismode scan # what's wrong right now? ./crisismode readiness # what breaks under load?

It's open source, Apache 2.0. If a report ever tells you something that turns out to be fabricated, that's the highest-severity bug we recognize — file it.