AGENTS.md

Guidance for AI coding agents working in this repository. Humans: see README.md (usage, mental model) and DESIGN.md (rationale, prior-art check) instead — this file exists so an agent doesn’t have to re-derive the below from scratch, not to duplicate either of those.

What this is

duckpipe: a serverless-first, DuckDB-native pipeline orchestrator (a library, not a platform). A uv workspace: duckpipe itself at repo root, duckpipe-tuning (a genuinely separate, optional package) under packages/.

Setup, test, lint

uv sync --group dev
uv run pytest                                        # duckpipe
uv run --directory packages/duckpipe-tuning pytest   # duckpipe-tuning
uv run ruff check .

Two tests in tests/test_examples.py skip themselves automatically when their prerequisite isn’t available — not a failure, no extra config needed to get a green run without them:

• test_serverless_executor_example — needs a working Docker daemon.
• test_browser_wasm_example — needs uv run playwright install chromium once.

examples/08_browser_wasm/’s own runnable bundle (duckpipe_src/, sample_data/) is generated by prepare_bundle.py and gitignored, not committed — run that script before trying to serve or test that example directly.

Repo layout

• src/duckpipe/ — the package. cli.py/dag.py/fingerprint.py/ remote.py/scheduler.py/state.py/task.py.
• examples/NN_name/ — each a self-contained, runnable pipeline over real bundled data (NYC TLC taxi trips), one facet of DuckPipe per example. examples/README.md indexes all of them.
• docs/ — short chapters, each linking back to the example code it describes. docs/README.md indexes them.
• packages/duckpipe-tuning/ — separate package; duckpipe never imports it or knows it exists.
• .github/workflows/ — ci.yml (tests+lint on push/PR), pages.yml (builds/deploys the GH Pages site, including generating the browser example’s bundle), publish.yml (PyPI release on a v* tag, gated on tests passing and the tag matching pyproject.toml’s version).

Conventions already in force here — match them, don’t relitigate them

• Comments explain WHY (a constraint, an invariant, a workaround), not WHAT — the code already says what. Don’t add comments that restate the next line.
• No speculative abstraction, no unused flexibility, no backwards-compatibility shims for code that can just be changed.
• Docstrings and READMEs state honest limitations plainly (see any example’s “honest limitations” section) rather than glossing over them — keep that tone in anything you write.
• Examples are real, runnable pipelines over real data, not pseudocode — if you add one, it should actually run via duckpipe run and be covered by tests/test_examples.py.

Before proposing a change

Read DESIGN.md first — most “why not just do X instead” questions are already answered there (it explicitly tracks open questions too, in case yours is one of them). Run ruff check . and the full test suite before considering a change done.