Systematic Debugging Loop

# Systematic Debugging Loop No fix until you can explain the failure mechanism in one sentence and point at the line that causes it. Patches written before that point fix symptoms and plant regressions. ## Phase 1 — Reproduce 1. Run the failing case yourself. Do not debug from a description or a screenshot. 2. Read the *entire* error output: full stack trace, stderr, surrounding log lines. The actual cause is often three frames below the one that looks interesting. 3. Reduce to the smallest reproduction: one test, one command, one input file. If the bug is intermittent, find the variable that makes it deterministic (seed, timing, ordering, fixture state) before going further. 4. Write the reproduction down as a runnable command. You will run it after every experiment. ## Phase 2 — Isolate - Bisect the search space, don't read it linearly. Useful axes: - **Time:** `git bisect` between the last known-good commit and HEAD. - **Code path:** disable half the pipeline (feature flag, early return, mock) and see if the failure survives. - **Data:** halve the input until the smallest failing payload remains. - Confirm assumptions with evidence, not memory: log the actual value, inspect the actual response, query the actual row. "It should be X here" is a hypothesis, not a fact. ## Phase 3 — Hypothesize - State the suspected mechanism explicitly: "The cache returns the stale entry because the key omits the locale." - Derive a prediction the hypothesis makes: "Then two requests with different locales must return the same body." - If you have two or more hypotheses, rank by cheapest-to-test and test in that order. Do not test two hypotheses with one change — you won't know which one the result speaks to. ## Phase 4 — Verify and fix 1. Run the experiment. If the prediction fails, the hypothesis is dead; return to Phase 3. Do not keep half-believing it. 2. Once confirmed, write a test that fails because of the root cause — not just one that reproduces the symptom. 3. Apply the smallest fix that makes that test pass. Resist drive-by cleanups in the same change. 4. Re-run the original reproduction *and* the full suite. A fix that breaks something else is a new Phase 1. ## Keep a lab notebook Maintain a running scratch note during the hunt: each hypothesis, the experiment, the result. Two reasons: after the fifth experiment you will otherwise re-test hypothesis two, and the notebook becomes the postmortem/PR explanation for free. If a hunt exceeds an hour with no narrowing, the notebook is what lets a second person pick it up without restarting. ## Anti-patterns to refuse - Changing code "to see if it helps." Every edit needs a prediction attached. - Adding a retry, sleep, or broad try/except around a failure you don't understand. - Declaring victory because the bug "stopped happening" without knowing why. - Stacking three speculative changes, getting a pass, and shipping all three.