Notebook Hygiene Rules for AI Agents

# Notebook Conventions These rules apply whenever you create or modify a notebook in this repository. A notebook that cannot be re-run top-to-bottom on a fresh kernel is a broken artifact, even if its saved outputs look right. ## The golden rule: restart and run all - Before committing any notebook change, restart the kernel and run all cells. If that fails or produces different results, the notebook is not done. - Never rely on execution order other than top-to-bottom. If cell 12 must run before cell 8, the notebook is wrong — reorder it. - Never leave a notebook depending on a variable whose defining cell was deleted or edited away (hidden state). After deleting a cell, restart-and-run-all to prove nothing depended on it. ## Cell ordering Every notebook follows this skeleton, in order: 1. **Title + purpose** (markdown): one paragraph — the question this notebook answers and what "done" looks like. 2. **Imports**: all imports in one cell. No imports anywhere else. 3. **Config + seeds**: paths, constants, parameters, and seed pinning (below). Paths are relative to the repo root and read from a single `DATA_DIR` constant — no absolute `/Users/...` paths. 4. **Data loading**: raw data is read here and never mutated in place afterwards; derived frames get new names (`df_clean`, `df_features`), not reassigned over `df`. 5. **Analysis sections**: each section starts with a markdown header stating what it does and ends with the result it produced. 6. **Findings** (markdown): conclusions, caveats, and follow-ups. An analysis notebook without written findings is incomplete. ## Seeds and determinism Pin every source of randomness in the config cell: ```python SEED = 42 import random, numpy as np random.seed(SEED) np.random.seed(SEED) # plus the framework you use: # torch.manual_seed(SEED); tf.random.set_seed(SEED) # and pass random_state=SEED to every sklearn estimator/split ``` - Pass `random_state=SEED` explicitly to `train_test_split`, model constructors, and samplers — global seeding alone does not cover them all. - If a result is intentionally stochastic, run it multiple times and report variance; do not present a single lucky run. ## Graduating code out of notebooks - Any function used by more than one notebook, or longer than ~20 lines, moves to a module under `src/` and is imported back. Notebooks orchestrate and narrate; modules implement. - Use an autoreload setup (`%load_ext autoreload` / `%autoreload 2`) in the imports cell while developing against `src/`. - Code in `src/` follows the project's normal standards: type hints, tests, lint. Notebook cells do not need tests; extracted modules do. - Pipelines that must run on a schedule are scripts or pipeline tasks, never a notebook executed by cron. ## Repository hygiene - Clear bulky outputs before committing unless the rendered output is the point (e.g., a report notebook) — follow whatever the repo's nbstripout/pre-commit setup dictates rather than deciding per-notebook. - Notebook filenames: `NN-<topic>.ipynb` (`03-feature-importance.ipynb`) so order of work is visible. - Never commit data files, credentials, or API keys inside a notebook — including in outputs. Check outputs for leaked tokens before committing. - Record the environment: dependencies live in the project's lockfile, not in `!pip install` cells. If a notebook needs a new package, add it to the environment properly and note it in the PR.