Paper Replication Protocol for Agents

# Paper Replication Protocol You are replicating the method and results of a published paper. The goal is a faithful reproduction with an honest account of where and why your results differ — not a reimplementation that happens to score well. ## Phase 1 — Read before writing any code - Read the **entire** methods section, the experimental setup, **all appendices/supplementary material, table footnotes, and figure captions** before writing code. Critical details (learning-rate schedules, preprocessing, exclusion criteria) hide in footnotes and appendices. - Produce a `SPEC.md` extracting every concrete parameter the paper states: dataset and version/split, preprocessing steps in order, model/algorithm hyperparameters, optimization settings, evaluation metric definitions (exact formula — "accuracy" and "F1" have variants), number of runs/seeds, and hardware if stated. - For every parameter the paper does **not** state, add a row to the Deviation Log (below) with your chosen value and rationale. Do not silently default. - Check whether the authors released code or data. If official code exists, treat it as the ground truth where it contradicts the paper's text — and record each contradiction in the log. ## Phase 2 — Pin everything - Pin the language version, every library version, and dataset version/checksum before the first experiment. Use a lockfile; record it in the repo. If the paper states versions, match them; if matching is impossible (yanked package, dead URL), log the substitution. - Pin and record random seeds for every run. Default protocol: the paper's seed count and seeds if given; otherwise 5 seeds (0–4), reporting mean ± standard deviation. - Record hardware and runtime per experiment. Numerical results can differ across GPU/CPU and across nondeterministic kernels — enable deterministic modes where available and note the cost. - Snapshot raw data read-only; all preprocessing is scripted and re-runnable from raw data with one command. No hand-edited intermediate files. ## Phase 3 — Implement and verify incrementally - Implement the pipeline in stages and verify each against any checkable number in the paper: dataset sizes after filtering, class distributions, parameter counts, baseline scores. Catching a preprocessing mismatch at stage 1 is cheap; at the final table it is a week of bisection. - Reproduce the paper's **baseline** results before the proposed method. If baselines do not match, the gap is in your setup, not the method — stop and fix it first. - Keep one experiment = one config file = one results directory, named by config hash or timestamp. Never overwrite results; never tweak a config in place after its run exists. ## The Deviation Log Maintain `DEVIATIONS.md` as a table; every entry has: | # | Where (section/eq./table) | Paper says / omits | What we did | Why | Expected impact | Log: unstated hyperparameters, version substitutions, paper-vs-code contradictions, ambiguous formulas and your reading of them, compute-forced reductions (fewer epochs, smaller batch — flag these loudly), and bugs found in official code. An empty deviation log on a real replication is a red flag, not an achievement. ## Reporting results - Report your numbers next to the paper's, per metric, with variance across seeds. State the comparison plainly: matched (within reported variance), partially matched, or not matched. - Never tune on test data, and never select the best seed post hoc to close a gap — that is fitting, not replicating. - If results do not match, enumerate candidate causes ranked by likelihood, referencing specific deviation-log entries, and state what experiment would discriminate between them. - Conclude with what you are confident the paper's method does, what remains uncertain, and exactly what someone needs (data, compute, missing detail from authors) to resolve the gaps.