Windsurf Rules for Python Data Pipelines
Windsurf.windsurfrulesCoding
Opinionated rules for pandas/polars pipelines: validate schemas at every boundary, process incrementally by partition, and never mutate a DataFrame in place.
Use when: Use in Python repos that move data through ETL/ELT stages with pandas or polars, especially when silent schema drift or full-table reprocessing has burned you before.
Get this skill
Add the skill file with the CLI, by hand, or over MCP.Save this to your project or home directory so Windsurf can load it.
npx goodprompts add windsurf-python-data-pipelineSaves to
./.windsurfrulesnpx goodprompts add windsurf-python-data-pipelineSaves to
./.windsurfrulesSkill file
.windsurfrules2,883 characters# Python Data Pipeline Rules
## DataFrame conventions
- Prefer polars for new code; only use pandas when an upstream library requires it. Never mix both in one module.
- All transformations are method chains on a fresh variable — never `inplace=True`, never reassigning columns on a DataFrame received as an argument.
- Every pipeline stage is a pure function with the signature `def stage_name(df: pl.DataFrame, config: StageConfig) -> pl.DataFrame`. No stage reads global state or environment variables directly.
- Column names are declared once as constants in `columns.py` (e.g. `COL_USER_ID = "user_id"`). String literals for column names in transformation code are a bug.
- Never use `apply` with a Python lambda when a vectorized expression exists. If you must use `apply`, add a comment explaining why and the expected row count.
## Schema validation
- Every stage boundary validates its input with a pandera (pandas) or explicit `df.schema` assertion (polars) before transforming. Fail fast with the stage name in the error message.
- Schemas live in `schemas/` as importable objects, one per dataset, versioned by suffix (`orders_v2`). Changing a schema means adding a new version, not editing the old one.
- Nullable columns must be declared nullable in the schema. An unexpected null should raise, not propagate.
- After every join, assert the row count matches expectations (`how="left"` joins must not grow the left side unless explicitly documented). Add `validate="m:1"`-style checks where the library supports them.
## Incremental processing
- Pipelines process one partition at a time (date, tenant, or batch ID) — never "read the whole table" unless the function name starts with `backfill_`.
- Every run writes a watermark (last processed partition) to the state store before exiting successfully; on start, resume from the watermark, never from zero.
- Output writes are idempotent: delete-then-insert or overwrite the target partition. Appending without deduplication is forbidden.
- Backfill entry points take explicit `--start` and `--end` arguments; no defaults that silently reprocess everything.
## I/O and operations
- All file I/O uses Parquet with explicit schemas; CSV is allowed only at ingestion edges and must be parsed with `dtypes` specified, never inferred.
- Read paths and write paths come from config objects, never hardcoded strings or f-string bucket paths in stage code.
- Log row counts in and out of every stage at INFO level. A stage that drops more than 10% of rows must log the reason at WARNING.
- Never `print()` in pipeline code — use the module logger.
## Testing
- Every stage gets a unit test with a 3–5 row inline DataFrame covering the happy path, a null edge case, and an empty input.
- Schema objects get a test that a known-bad record is rejected.
- Run `pytest -q` and `ruff check .` before suggesting any commit.