Windsurf Rules for Python Data Pipelines

# Python Data Pipeline Rules ## DataFrame conventions - Prefer polars for new code; only use pandas when an upstream library requires it. Never mix both in one module. - All transformations are method chains on a fresh variable — never `inplace=True`, never reassigning columns on a DataFrame received as an argument. - Every pipeline stage is a pure function with the signature `def stage_name(df: pl.DataFrame, config: StageConfig) -> pl.DataFrame`. No stage reads global state or environment variables directly. - Column names are declared once as constants in `columns.py` (e.g. `COL_USER_ID = "user_id"`). String literals for column names in transformation code are a bug. - Never use `apply` with a Python lambda when a vectorized expression exists. If you must use `apply`, add a comment explaining why and the expected row count. ## Schema validation - Every stage boundary validates its input with a pandera (pandas) or explicit `df.schema` assertion (polars) before transforming. Fail fast with the stage name in the error message. - Schemas live in `schemas/` as importable objects, one per dataset, versioned by suffix (`orders_v2`). Changing a schema means adding a new version, not editing the old one. - Nullable columns must be declared nullable in the schema. An unexpected null should raise, not propagate. - After every join, assert the row count matches expectations (`how="left"` joins must not grow the left side unless explicitly documented). Add `validate="m:1"`-style checks where the library supports them. ## Incremental processing - Pipelines process one partition at a time (date, tenant, or batch ID) — never "read the whole table" unless the function name starts with `backfill_`. - Every run writes a watermark (last processed partition) to the state store before exiting successfully; on start, resume from the watermark, never from zero. - Output writes are idempotent: delete-then-insert or overwrite the target partition. Appending without deduplication is forbidden. - Backfill entry points take explicit `--start` and `--end` arguments; no defaults that silently reprocess everything. ## I/O and operations - All file I/O uses Parquet with explicit schemas; CSV is allowed only at ingestion edges and must be parsed with `dtypes` specified, never inferred. - Read paths and write paths come from config objects, never hardcoded strings or f-string bucket paths in stage code. - Log row counts in and out of every stage at INFO level. A stage that drops more than 10% of rows must log the reason at WARNING. - Never `print()` in pipeline code — use the module logger. ## Testing - Every stage gets a unit test with a 3–5 row inline DataFrame covering the happy path, a null edge case, and an empty input. - Schema objects get a test that a known-bad record is rejected. - Run `pytest -q` and `ruff check .` before suggesting any commit.