failbook
GitHub

The model failed twice. Don't retry blind.

failbook is an open library of real failure cases. Two failures in a row usually mean a hidden wrong assumption in the request or the workflow — not in the model. Search cases of similar questions, confirm with the user, retry with the rephrased request. Only human-error cases are admitted, so they never expire with the next model release.

failbook search "<your situation>" Get the skill / CLI

The flow: rule of two

  1. 1

    After 2 failures on the same task — before attempt 3 — search failbook with the current situation.

  2. 2

    Each case shows how others phrased it, the human-side root cause, and the rephrased request — confirm with the user.

  3. 3

    Retry with the confirmed phrasing. Hit a case that isn't in the library? Contribute it via PR.

The case library

17 human-error cases and growing via PRs

coding Editing template structure with bare closing tags as anchors silently eats the wrong block

After an AI edits a template/component, build is green but the new UI element never renders at runtime.

The scene

“the build passes but the dialog/component I asked for doesn't show up, why is the render broken?”

What happened

The edit applied without errors; compiler/type-check/build all green. At runtime the added element is missing or lands in the wrong branch.

Why blind retries keep failing

The model re-applies similar edits or tweaks the new component's code, assuming the insertion position was correct because the build passed. The real problem: the edit anchor was three bare closing tags, text that appears many times; fuzzy matching landed on a different block and consumed structural closers, so the new element sits inside a dead branch and the template's real close was swallowed — which the tolerant compiler still accepts.

Root cause (human side)

Human accepted 'build green' as proof of 'structure right'. Template closers are non-unique text and must never anchor a structural edit; compiler tolerance turns structural damage into a silent pass.

Rephrase the task like this

For structural template edits: anchor on unique long context including a content line (never bare closing tags), and after the edit verify at runtime — open the page and confirm the element actually renders — not just the build.

Countermeasures

  1. Re-do the edit with an anchor that includes unique content lines around the target position
  2. Search the file for orphaned/duplicated closers to confirm structure is intact
  3. Runtime-verify: actually open the affected view and check the element renders where expected

How to verify the fix

The element is visible in a real browser/render at the intended position — not merely a passing build.

ops Production asset directories overwritten without a ledger check — style drift found only by a human, weeks late

An AI-generated asset batch replaced live files directly; no manifest, no comparison, drift discovered late and the whole batch was scrapped.

The scene

“an agent updated our image assets directly in the production folder and they look off — how do we prevent untracked overwrites?”

What happened

Live assets silently replaced by a new batch with no record of prompt source or approval. Ledger says the old version is current. The drift (overdetailed style, inconsistent palette) is invisible when reviewing single images and surfaces only when a human compares categories side by side.

Why blind retries keep failing

The model regenerates 'better' images one by one. But the process had no gate: writing to the live directory required no ledger reconciliation, and single-image review cannot detect cross-asset style drift — so the same failure recurs regardless of image quality.

Root cause (human side)

Human allowed direct writes to the production asset directory with no 'reconcile against the ledger' step, and reviewed replacements one-by-one instead of comparing same-category assets before/after.

Rephrase the task like this

Institute the gate: anything appearing in a production asset directory must reconcile with the ledger (run manifest: prompt, source, approver) — unrecorded files get archived first, never overwritten. Before replacing a category, produce a before/after contact sheet for human approval. Always keep the previous batch archived so rollback is one copy.

Countermeasures

  1. On discovery of unrecorded assets: archive them as-is with a run manifest before any disposition
  2. Require a before/after contact sheet of the same category for approval — never single-image review
  3. Keep every superseded batch archived (restorable in one step)
  4. Restrict write access to production asset dirs to the ledger-updating pipeline

How to verify the fix

Ledger, manifest, and live directory agree one-to-one; the contact sheet for the last replacement exists and is approved; rollback of the previous batch succeeds.

automation Helper steps that navigate away silently invalidate every check that runs afterwards

Script reports success but nothing was actually submitted; points/money not consumed, no new records appear.

The scene

“my browser automation says the task was submitted successfully but nothing happened on the site, can you make the selector more reliable?”

What happened

Automation logs a success toast/submission, but the platform shows no consumption and no new record. All intermediate checks passed.

Why blind retries keep failing

The model keeps fixing selectors and waits, because the reported symptom points there. The real break is earlier: an auxiliary step (e.g. capturing a 'before' baseline) navigated to another page, so every later step — including the submit click and its verification — ran on the wrong page where empty states were misread as success.

Root cause (human side)

Human assumed intermediate helper steps may freely navigate; the page the checks run on was never asserted. Success was judged by a single weak UI signal ('input box cleared') that the platform's new UI no longer exhibits.

Rephrase the task like this

Restructure the flow: do all navigating helper steps first, return to the work page, assert the URL/frame before every critical step, and verify submission by a server-side effect (API response with status code, or new record added after the pre-captured baseline) — never by a UI cosmetic signal.

Countermeasures

  1. Move all navigation-heavy helper steps (baseline capture) before the input/submit phase; return to the work page afterwards
  2. Assert current URL/page identity before input, before submit, and before each verification
  3. Replace cosmetic success signals (input cleared, toast) with a server-side effect: listen for the submit API response, or diff records against the pre-captured baseline
  4. After any platform UI change, re-run the whole chain once with a cheap sample before batch runs

How to verify the fix

One paid/consuming sample run: the server-side effect (API ret code or new baseline-diff record) must be observable, not just the script's own success log.

data Bulk UPDATE with WHERE describing the target state matches zero rows, silently

A migration/backfill 'runs successfully' but changes nothing; zero affected rows looks like success.

The scene

“the update query executed without errors but when I check, none of the rows are marked — did it even run?”

What happened

UPDATE completes with no error; verification shows the column unchanged for the intended rows. Command output showed nothing alarming.

Why blind retries keep failing

The model tweaks syntax or permissions. The logic bug survives every retry: the WHERE clause was written as the value being SET (select rows that already have it) instead of the rows that still need the change — so the statement matches zero rows, and databases treat 0 updated rows as a perfectly successful statement.

Root cause (human side)

Human wrote the WHERE clause from the goal ('set flag=1') instead of from the selection ('where flag is not yet 1'). Zero-row success is indistinguishable from success unless affected-row counts are checked.

Rephrase the task like this

Write the WHERE to describe the to-be-changed set (e.g. `WHERE COALESCE(flag,0)=0` when setting flag=1), run it, then verify with a count query: affected rows match expectation and a recount of the target state returns zero remaining.

Countermeasures

  1. Restate the WHERE as 'rows still needing the change', not 'rows with the goal state'
  2. Run inside a transaction; check affected row count before committing
  3. After commit, run a count of rows still matching the old state — must be 0

How to verify the fix

Affected-rows number equals the pre-counted expectation, and the post-count of unconverted rows is zero.

data Sequential string replacements corrupt each other: the previous step's output matches the next step's search

Batch version bumps via chained replace produce mixed versions pointing at non-existent artifacts.

The scene

“I replaced the version strings in several passes and now some entries point to a version that was never released — how did that happen?”

What happened

After a chain of replacements (v1→v2, then v2→v3, ...), some entries that should have become the final version were rewritten twice — referencing a version that never existed; other data files that also embed versions were missed entirely.

Why blind retries keep failing

The model re-runs the 'missed' replacement or fixes individual entries. The mechanism: sequential str.replace calls share one corpus — the output of step N is input to step N+1, so anything matching later patterns gets hit again. Retry keeps operating on an already-corrupted corpus without a structural check.

Root cause (human side)

Human applied multi-pattern edits as a chain of naive replaces without realizing the intermediate outputs join the match set of subsequent passes, and verified by spot-reading instead of structurally parsing the result.

Rephrase the task like this

Apply all substitutions in ONE pass with a single regex alternation (or a function that maps each match exactly once), then verify by parsing the result as data (JSON parse + compare every version field against the expected value) — not by eyeballing a few lines. Keep in mind which files embed the same value; update and check all of them.

Countermeasures

  1. Replace the chain with one regex alternation pass (match-once semantics)
  2. Structurally verify: parse the file and assert every field equals the expected value
  3. Enumerate every file carrying the value (data copies exist in more than one place) and repeat there
  4. Assert no reference points to an artifact version that was never published

How to verify the fix

Structural check over the whole corpus: every version field equals the target, zero occurrences of intermediate versions, all referenced artifacts exist.

debugging 'The click does nothing' is two different silent failures — capture the network to tell them apart

An automation click on a web app shows zero reaction; selector fixes never help because the click wasn't the problem.

The scene

“clicking the button via automation shows absolutely no reaction — no toast, no navigation, nothing. Is my selector wrong?”

What happened

Click via automation: no toast, no navigation, no visible request. Switching to a stealth browser driver makes the click 'work' — but still no result appears.

Why blind retries keep failing

The model keeps refining selectors or timing. The symptom actually has two stacked layers: (1) the driver being detected as a bot, so the click handler never runs; (2) after the click fires, a business gateway (insufficient credits) silently aborts the flow without any UI feedback. Both look identical from the outside — 'nothing happened'.

Root cause (human side)

Human judged the failure from UI appearance only. Without capturing the network layer, 'request never sent' (bot detection) and 'request sent but flow silently stopped' (business gate) are indistinguishable — and they have completely different fixes.

Rephrase the task like this

Capture network activity around the click and compare: if the expected request appears but the flow stops afterward, it is a business gate — check the balance/eligibility API. If no request appears at all, it is the automation layer being detected — switch to a stealth driver setup. Do not touch selectors based on this symptom alone.

Countermeasures

  1. Record requests before/after the click; diff for the expected POST
  2. Request present + downstream stops → call the business/eligibility API directly and act on its answer
  3. No request → change the browser-control layer (stealth profile), not the page interaction
  4. Verify success by the business result (record created), never by absence of error

How to verify the fix

The expected request is observed AND the business result (e.g. new generation record) appears; the diagnosis names the exact blocking layer.

coding Copied 'working' parsing code fails silently because the input shape was never tested

Login/callback or URL handling copied from another project passes silently — the feature is dead with zero errors.

The scene

“I reused the same auth callback code from our other app but the login never completes and there are no errors, what's different?”

What happened

The external side works (redirect arrives), but the app behaves as if the callback never happened. No exception, no log line.

Why blind retries keep failing

The model inspects the network/redirect and re-checks the copied code 'since it worked over there'. The real issue: URL parsers treat non-standard schemes differently than http — the first path segment becomes the host — so the copied assertion compares against a path that never occurs, and the mismatch branch returns silently.

Root cause (human side)

Human treated 'reference implementation' as 'verified for my input shape'. Parsing rules for custom schemes were assumed identical to http URLs, and the failure branch had no logging, making the dead path invisible.

Rephrase the task like this

Before reusing parsing code, write one test with the exact input shape it will see in this project (e.g. `new URL('myapp://auth/callback?code=x')` — inspect host and pathname). Add a log line to every early-return branch so dead paths are observable.

Countermeasures

  1. Print/inspect how the URL actually parses (host vs pathname) for the custom scheme
  2. Normalize before comparing: reconstruct the path as host + pathname, then assert
  3. Log every silent return in callback handling with the raw input
  4. Keep the shape test as a permanent regression test

How to verify the fix

The shape test passes and an end-to-end login round-trip lands the user in a logged-in state; killing the handler deliberately shows up in logs.

deploy Deploy script dies on a non-critical step before the actual deploy runs

Release script aborts mid-way on an auxiliary step (backup push); production never got the update and nobody noticed.

The scene

“our publish script sometimes just stops halfway and the site isn't updated, how should we handle the failing git step?”

What happened

Script exits with an error after the build; the deploy command never executed. Output looks like a normal failure of the auxiliary step.

Why blind retries keep failing

The model tries to fix the failing auxiliary step (git auth, network), because that is what errored. But that step failing is expected sometimes; the design flaw is ordering — the critical deploy was placed after a step allowed to fail, under set -e, so any auxiliary hiccup silently cancels the release.

Root cause (human side)

Human ordered the pipeline as build → backup → deploy and let set -e treat every step as fatal. The deploy is the only step that must run; backup is optional by nature.

Rephrase the task like this

Reorder the release script so the deploy command runs first (or is guaranteed regardless of auxiliary failures); make the backup push non-fatal with `|| warn`, and track a pending-backup note when it fails.

Countermeasures

  1. Put the deploy command at the top of the critical path
  2. Demote auxiliary steps (git backup push) to best-effort: `git push ... || echo WARN backup pending`
  3. Log a follow-up item when a backup is skipped so it can be pushed later manually

How to verify the fix

Simulate the auxiliary failure (disconnect network for git) and confirm the site still deploys and the warning is logged.

deploy Deploy succeeded but the site still shows old content — data changes were never built

Editing data files and running the publish script ships the old build; the script's success message hides it.

The scene

“we updated the data files and published successfully, why does the live site still show the old content?”

What happened

Publish output says success; production serves previous content. Data edits exist locally but never reached the deployed output.

Why blind retries keep failing

The model re-checks the data files and re-runs publish. But publish only uploads the prebuilt output directory — the data-to-output build step lives outside the publish script, so unless it is run explicitly, uploads faithfully ship the stale build. 'Publish succeeded' is true and useless.

Root cause (human side)

Human assumed publish implies build. The pipeline had a gap: content generation was a manual, unlisted step, and nothing verified the live result after upload.

Rephrase the task like this

Treat 'publish' as build + upload + verify: always run the build step before the publish script, and after deploying, fetch the live URL and check the new content is actually present (following redirects).

Countermeasures

  1. Chain the commands: build step && publish script (never publish alone after data edits)
  2. After deploy, curl the live page (follow redirects) and grep for the new content
  3. Optionally move the build invocation into the publish script so the gap closes permanently

How to verify the fix

The live-site fetch shows the new content string; a deliberate no-build run is caught by the verify step.

testing E2E suite reads the real user config and fights for machine-global resources

End-to-end tests pass locally for one person, fail for everyone else: language-dependent selectors, hotkeys taken, timing off.

The scene

“half of the e2e tests fail with selectors not found, but they passed yesterday on my machine — is it flaky?”

What happened

A large share of e2e tests fail on machines other than the author's: role names in the wrong language, hotkey registration conflicts, clicks swallowed by visible overlays or focus changes.

Why blind retries keep failing

The model patches individual selectors and adds waits, one test at a time — an endless whack-a-mole. The cause is systemic: the app under test reads the real per-user configuration (language, key bindings), and machine-global resources (global hotkeys, desktop focus) are shared with everything else running. No per-test fix can make that deterministic.

Root cause (human side)

Human ran e2e against the real user environment instead of an isolated one: no temp config, no reserved test-only key range, no handling of physically changing desktop focus.

Rephrase the task like this

Make e2e isolated by default: launch the app with a temp config (fixed language, defaults), choose test-only key codes outside both the production default and OS-reserved ranges, and neutralize decorative overlays (pointer-events none). Explicit config paths passed by a test must be respected, not overwritten.

Countermeasures

  1. Default to an isolated temp config: fixed language and known settings
  2. Pick test hotkeys from a range verified free on target machines; never reuse production defaults
  3. Disable/neutralize decorative overlays that can swallow pointer events
  4. Re-run the full suite on a second machine to confirm machine-independence

How to verify the fix

Full e2e suite passes identically on two different machines/configs, including a non-default language.

automation Interactive CLI inside an agent's non-interactive pipe hangs forever, invisibly

Running an interactive program (REPL/chat CLI) from an automation session floods logs with prompts and never exits; piping through tail hides all feedback.

The scene

“I asked the agent to run this model CLI to test something and now the session is stuck and the process won't die — what happened?”

What happened

The tool loads, then spews an interactive prompt thousands of times (stdin is an empty pipe), never exits. Output piped to `tail` shows nothing until EOF, so the session looks frozen. Timeout auto-backgrounding orphans the process, which keeps holding GPU/memory.

Why blind retries keep failing

The model adds 'non-interactive' flags and retries — but in a headless pipe environment some flags are silently ignored by the build, and there is no error telling you so. The environment (no TTY, empty stdin) invalidates assumptions the command's help text makes; every retry re-enters the same trap, and pipe buffering hides it.

Root cause (human side)

Human ran an interactive program in a non-interactive harness and assumed flags would force non-interactive mode. Long output was piped through a buffering viewer, removing all live feedback; there was no log file and no plan for terminating a stuck heavyweight process.

Rephrase the task like this

For automated runs, prefer tools that terminate naturally (benchmark/batch modes). If the interactive CLI is unavoidable: close stdin explicitly (`</dev/null`), first probe whether the non-interactive flags are accepted with a tiny run, redirect all output to a log file (no pipes), and keep a kill plan (record the PID) for heavyweight processes.

Countermeasures

  1. Replace interactive verification with a naturally-terminating alternative if one exists
  2. If not: `interactive-cli ... </dev/null > run.log 2>&1` and a minimal first run to validate flags
  3. Record the PID and define the kill command before starting heavyweight runs
  4. Read progress by grepping the log file, never by piping live output

How to verify the fix

The run terminates by itself within the expected time, the log file shows the real result, and no orphan process remains.

debugging Wrapping a command with '&& echo OK || echo FAIL' makes every exit code zero

A failed build/task is reported as success (exit code 0) because the last command in the chain is an echo.

The scene

“the task notification says exit code 0 but the build actually failed, why is the status wrong?”

What happened

Automation/notification reports success (exit code 0) while the actual work failed; failure is discovered late, after downstream steps already trusted the code.

Why blind retries keep failing

The model re-runs the command or inspects the build tool, since 'the command failed'. But the command's failure was never visible: `cmd && echo OK || echo FAIL` returns the exit code of the final echo (always 0), so every consumer of the exit code sees success.

Root cause (human side)

Human wrapped the command for nicer logging and accidentally destroyed the exit code channel. Exit code of a chain is the last command's, not the interesting command's.

Rephrase the task like this

Preserve the real exit code: redirect output to a log file and echo the captured status separately, e.g. `cmd > log 2>&1; echo exit=$?`, and gate downstream steps on that captured value.

Countermeasures

  1. Replace `cmd && echo OK || echo FAIL` with `cmd > log 2>&1; echo exit=$?` (or use set -e without masking wrappers)
  2. For background jobs, judge success by produced artifacts and the tail of the log file — never by a wrapped notification's exit code alone

How to verify the fix

Intentionally break the command; the captured/propagated exit code must be non-zero and downstream must not run.

automation Paid automation fails halfway because balance was never checked and failure was judged by the wrong signal

Automation clicks 'generate/submit' on a credit-based service; the click appears dead, and the team concludes the whole channel is blocked.

The scene

“the submit button click does nothing on this credit-based site — is the site blocking my automation?”

What happened

Clicking submit produces no toast, no navigation, no visible request. Later, the team declares the channel fully blocked and falls back to a worse alternative — while candidates from an earlier 'failed' attempt were actually being generated successfully in the background.

Why blind retries keep failing

The model retries selectors and clicks, because 'the click does nothing'. There are typically two stacked silent gates: automation-unfriendly browser control can prevent the click handler from ever firing, and separately a credit gateway can silently redirect away when balance is insufficient. Neither writes an obvious error. Also, a stale memory of 'balance was low this morning' drives decisions hours later, though daily free credits have since refreshed.

Root cause (human side)

Human skipped a cheap pre-check (balance) and had no final arbiter for success (the service's history/result records), relying on volatile UI signals (URL changes, input clearing) that both false-negative. Conclusions about 'the whole channel' were drawn from a single unprechecked attempt plus outdated memory.

Rephrase the task like this

Before any credit-consuming submission: query the balance endpoint and refuse early if below threshold. Capture a baseline of the results/history before submitting; after submitting, judge success ONLY by new records appearing after the baseline — click/UI signals are hints, not verdicts. If the channel truly blocks, report the blocking point and required unlock action; never silently degrade the deliverable.

Countermeasures

  1. Add a balance pre-check against the service's API before every paid run
  2. Capture a results-baseline before submit; poll the baseline diff as the sole success arbiter
  3. On apparent failure, compare network captures: request sent + flow stops = business gate; no request at all = control/fingerprint layer
  4. Re-read the current balance/state before declaring a channel dead — never reuse hours-old memory

How to verify the fix

A paid sample run completes with a new record after baseline and the expected balance decrement; the false-'blocked' scenario is disproven by finding the earlier results.

debugging Packaged app exits instantly with code 0 — check for leftover dev processes first

A packaged desktop app quits silently on launch while the dev build works fine; no log, no error.

The scene

“the packaged app starts and immediately exits with no error, but running from source works — what's wrong with the packaging?”

What happened

The packaged binary launches and exits immediately (exit 0, no output). Dev mode works. Process lists appear clean at a glance.

Why blind retries keep failing

The model re-examines packaging config, signing, or the single-instance lock mechanism itself — because 'no process found' seemed to rule out interference. The actual cause: a leftover dev-mode process from an earlier session (not matching the obvious name filter) still holds the same user-data single-instance lock, so the packaged instance quits by design.

Root cause (human side)

Human trusted a narrow process-name search and jumped to mechanism-level suspects before ruling out environment leftovers. Debugging order was wrong: cheap environment checks come before mechanism theories.

Rephrase the task like this

Before touching packaging config: list all related processes broadly (any dev/runtime binary of this stack, not just the app name), kill leftovers, and re-test the packaged build. Only then investigate the lock mechanism.

Countermeasures

  1. Search processes broadly (e.g. the framework's runtime binary name) and kill orphaned dev instances
  2. Re-launch the packaged app and confirm it stays up
  3. If still failing, check error output in full — truncated views (e.g. piping through tail) have hidden real errors before

How to verify the fix

Packaged app stays running with no related orphan processes alive; reproducibility confirmed by launching twice.

deploy Piping build output through 'tail' eats the exit code and hides the failure

A build that never produced artifacts reports success because `| tail` was the last command in the pipeline.

The scene

“the background build said it finished fine but there is no output artifact, where did it go wrong?”

What happened

Build task 'completes' with a green status; the expected artifact is missing or stale. Sometimes the real error (e.g. missing npm script) is nowhere in the captured output.

Why blind retries keep failing

The model searches for the artifact or re-runs with different flags. The channel that carried the truth was cut: a pipeline's exit code comes from its last command (`tail`), and `tail` only flushes at EOF — so a failing, long-running build shows zero feedback the whole time and ends 'successfully'.

Root cause (human side)

Human piped long build output for readability, trading away both the exit code and live feedback. Documentation references were also stale (script names changed), so the wrong command was being 'fixed'.

Rephrase the task like this

Run builds with output redirected to a log file and report the captured exit code: `npm run build > build.log 2>&1; echo exit=$?`. Check the artifact and the log tail afterwards. Verify command names against the project's package.json, not memory or docs.

Countermeasures

  1. Replace `... | tail -N` with `... > build.log 2>&1; echo exit=$?`
  2. After the run, check both the artifact exists and grep the log tail for errors
  3. Confirm the build command/script name from the repo (package.json scripts) before running

How to verify the fix

Break the build on purpose once: the redirected run must report a non-zero exit and the log must contain the real error.

automation Hardcoded screen coordinates fail for self-repositioning, focus-hiding windows

UI automation clicks empty space round after round because the window moves on every show and hides when it loses focus.

The scene

“my click automation worked in testing but keeps clicking the wrong place on this app — the coordinates are right, why?”

What happened

Screenshot shows the button; the click lands on nothing or another app. Between the screenshot and the click, the window repositioned (it spawns near the cursor) or hid itself (it auto-hides on blur — e.g. another automation session stole focus). Mis-clicks have even opened unrelated dialogs.

Why blind retries keep failing

The model adjusts coordinates pixel by pixel. But coordinates taken from an earlier frame are invalid by design: this window class re-anchors and self-hides, so 'visible in the screenshot' never implies 'clickable at those screen points now'. Round-trip automation (screenshot here, click there) leaves a gap the window state changes through.

Root cause (human side)

Human assumed window position and visibility are constant between observation and action. For repositioning/self-hiding windows, position must be queried and visibility asserted inside the same script that clicks.

Rephrase the task like this

Make each interaction atomic: in one script, query the window rect at action time, compute the control position relative to that rect, assert visible+foreground immediately before clicking. When multiple automation sessions share the machine, compress the whole observe-decide-act cycle into the single script instead of round-tripping through an agent. For web-view UIs where accessibility enumeration finds nothing, locate the control by scanning pixels for its known color signature as a fallback.

Countermeasures

  1. GetWindowRect at action time; derive control offset relative to the rect
  2. Assert IsWindowVisible + GetForegroundWindow right before the click; bail out otherwise
  3. Single-script atomic cycle for focus-stealing environments
  4. Pixel-signature fallback when UIA/enumeration returns nothing
  5. Leave true mouse-drag/multi-click human-verification to manual acceptance, automation only asserts command/screenshot level

How to verify the fix

N consecutive automated rounds all land on the control while a second session steals focus in between; zero mis-clicks into other apps.

ops Validation tooling crashes on legitimate placeholder values instead of reporting them

A daily audit script throws an exception because a config contains a documented 'TODO' placeholder — the whole check pipeline goes down.

The scene

“our audit script crashes with a URL parse error ever since someone added a new entry with a placeholder, how do we fix it?”

What happened

The check command dies with an exception (e.g. invalid URL) on the first incomplete entry; everything after it is unaudited. The placeholder was produced by the project's own official template, so it is a legitimate state.

Why blind retries keep failing

The model fixes the one bad entry or patches the specific exception. Next incomplete entry kills it again. The design flaw: probe/validate functions assume all inputs are fully-formed, but 'registered but not yet filled in' is a normal lifecycle state the tool must tolerate and report, not crash on.

Root cause (human side)

Human wrote validators that treat 'incomplete' as 'invalid input' — an unhandled exception path outside the try scope — instead of a first-class 'pending' outcome.

Rephrase the task like this

Every probe/validator must map unusable input (placeholder, empty, malformed) to a pending/skipped result with a reason, never throw. After adding any new field to the templates, immediately run the full check once to prove the chain stays self-consistent — don't wait until fields are filled.

Countermeasures

  1. Wrap each probe so placeholder/empty/malformed values yield a pending result carrying the reason
  2. Keep exceptions inside the caught set (including argument-construction errors)
  3. Add a regression test: a record full of TODO placeholders must produce pending, not a crash
  4. Run the whole check right after template/schema changes

How to verify the fix

The check completes over a dataset containing placeholders, reporting them as pending with reasons, exit code 0.

Contribute a case

One case per PR. Root cause must be human-side (survives model upgrades), real and reproducible, desensitized. Run failbook validate before submitting. Admission rules in docs/CONVENTIONS.md.