# DuckPipe — Design Rationale *Why DuckPipe is built the way it is. For what it does and how to use it, see [`README.md`](README.md) — this document is the reasoning underneath, kept around because several error messages and docstrings point back to specific sections here, and because the tradeoffs are worth preserving even once the features they justify are just... how the thing works.* --- ## 1. Vision Most teams reach for Airflow, Prefect, or Dagster the moment they need "more than one script that depends on another script." All three solve that by asking you to first stand up persistent infrastructure — a scheduler, a webserver, a metadata database, and usually a broker/worker pool — before you've run a single real pipeline. That infrastructure tax is fixed cost, paid whether your pipeline moves a few thousand rows once a day or a billion rows every minute. DuckDB's own rise made a similar point about query engines: a huge share of "big data" workloads people defaulted to Spark for were never actually big-data workloads — they fit comfortably on one modern machine, and a single well-tuned embedded process finishes in seconds what a cluster took minutes to schedule. DuckPipe applies that same correction to orchestration, but to the *infrastructure*, not to a data-size ceiling: **most pipelines don't need standing infrastructure to be well-organized, incremental, and observable, regardless of how much data they move.** The orchestrator for that case should be a library, not a platform — importable, runnable as a plain Python process, requiring zero standing infrastructure, and disappearing entirely when it's not running. Scaling out (§8) is an extension of that same core for when a single node genuinely isn't enough, not evidence a pipeline has outgrown DuckPipe. **The bet:** if the single-node, zero-server experience is right — genuinely as easy as `pip install duckpipe` and running a Python file — there's "almost no reason not to use it," the same way there's rarely a reason to reach for Spark before trying DuckDB first. Distributed/remote execution and browser execution are extensions of the same core (§8), not a redesign. --- ## 2. Design tenets (the non-negotiables) 1. **No persistent servers required to get started.** No scheduler daemon, no central metadata database, no broker. A DuckPipe run is a process that starts, does work, records what it did, and exits. State lives in a file (a DuckDB database) by default, not a running service. This is a default, not a restriction: an opt-in DuckLake catalog upgrade (§8) is fully compatible with a centralized-metadata-store architecture when a team wants one — including one backed by a dedicated, long-running database (Postgres, self-hosted or managed) shared across several DuckPipe deployments for genuine multi-user/multi-tenant use — it's just never the thing you have to stand up before running your first pipeline. The plain-file state can optionally be synced to durable remote storage (S3/GCS/Azure/etc.) via `fsspec` — download-before-run, upload-after-run — so DuckPipe stays correct even when the process itself runs in an ephemeral, container-per-invocation environment (its own laptop/CI runner, or nested inside another orchestrator's worker — see §9). This is a sync pattern, not a live remote database: DuckDB's own docs confirm its native database file format only supports read-only remote attach (`ATTACH 's3://...'` via `httpfs`), not read-write, so a local-scratch-copy-then-sync approach is the correct one, not a workaround. 2. **The orchestrator core is data-blind. DuckDB is the orchestrator's brain (its state store), never a requirement for what a task does.** Keep two claims separate. First, the *orchestrator's own state* (run history, task fingerprints, logs, lineage) always lives in a DuckDB file/catalog — that's the core architectural bet (§4) and isn't optional. Second, and this is the tighter, more important rule: **a task's signature is `Callable[..., Any]`, and the core never inspects, converts, or reasons about what a task returns.** Passing a task's output to the next task is a plain Python function call — whatever the object is (a DuckDB relation, a Polars `LazyFrame`, a pandas DataFrame, a string, `None`) crosses zero boundaries and costs the orchestrator nothing, because there's no serialization to do. This is what makes "efficiency is a secondary bonus, not a core feature" literally true: DuckPipe adds no computational overhead on top of your data processing, because it never touches your data. Where a real boundary *does* exist — caching a result to disk between runs, or shipping a result to another process — that's handled by an explicitly optional, separately documented layer (§6), never by the core. 3. **Never introduce a parallelism or tuning primitive that duplicates one an engine or Python already has.** Direct response to two related pain points researched early on: Prefect ships `ConcurrentTaskRunner`/`DaskTaskRunner`/`RayTaskRunner` and steers users toward wrapping pandas in Dask inside tasks, adding a second parallelism model on top of a data engine that already parallelizes internally; and no orchestrator found offers a *unified* resource-tuning abstraction across engines, because DuckDB's knobs (`threads`, `memory_limit`), Polars' (thread env vars, `collect(engine=...)`), and Daft's (runtime config) are genuinely different APIs — inventing one abstraction over all three would be exactly this kind of duplication, just relocated. DuckPipe's rule: **DAG-level concurrency** (running independent tasks concurrently) is the orchestrator's job, using a plain `asyncio`/thread pool invisibly. **Everything about a task's internal parallelism or resource tuning** — DuckDB's `threads`/`memory_limit`, Polars' thread count, a `concurrent.futures` pool, Ray/Dask for a genuinely distributed step — is the user's own code, using whichever engine's own native controls, optionally aided by small standalone utility functions (§6) the user calls themselves. The core has no opinion and no wrapper for any of it. The same principle extends to composing tasks across files: no DuckPipe-specific module/package concept, just ordinary Python imports (§8). 4. **Zero-to-running in one file, one command.** `duckpipe run pipeline.py` works with no config file, no `duckpipe init`, no database migration step. A `duckpipe.db` file appears next to the script holding run history; that's the entire "infrastructure." 5. **Local dev and production are the same code path.** No "local executor vs. Celery executor vs. Kubernetes executor" branching that changes behavior. Scaling out (§8) changes *where* a task runs, never *how you write* a task. 6. **Incremental by default, not by manual flag.** Borrowing from SQLMesh: every task gets a content fingerprint (hash of its code + config + resolved upstream fingerprints — never the output data itself, which is what lets fingerprinting stay data-blind per tenet #2). If nothing changed, DuckPipe skips it. No `--select state:modified+` incantation required — this is on by default, with an explicit `--force` escape hatch. 7. **Small, boring surface area.** The full list of concepts a new user needs to learn should fit on one page (README.md's own "whole mental model" section). A DAG is inferred from plain Python function signatures, not a parallel YAML/config DSL — see §4 for why this is a deliberate, defended choice rather than a shortcut. Any proposed new concept (a second decorator, a "profile" object, a plugin system) gets evaluated against "does this let us delete something instead," per the exact failure mode Prefect fell into over three major versions (§10). --- ## 3. What's explicitly *not* solved by the core A hosted UI/control plane, alerting/SLA infrastructure, and dynamic-mapping-over-huge-fan-out (thousands of parallel tasks) are all out of the *core* — but "out of scope" means solved by composition, not left unsolved. A hosted UI/alerting layer is a `SELECT` away rather than a missing feature, precisely because tenet #1 keeps state in a plain, queryable DuckDB file (or a DuckLake catalog, §8, for real snapshot history) instead of hiding it behind a service boundary. Genuine massive fan-out is exactly the thing Airflow/Dagster/Prefect are already good at, so DuckPipe leans on that instead of reimplementing it: embed a DuckPipe pipeline as the unit a heavier orchestrator's own dynamic-mapping feature drives, one per partition — §9 covers this directly, including why it's a genuinely good idea and not just a curiosity. DuckPipe's stance: be *excellent* at the case where standing infrastructure isn't required at all, and extend outward via the DuckLake/distributed-execution path (§8) or by embedding inside a heavier platform (§9), rather than trying to be a general distributed-systems platform itself. Also explicitly out of the *core* (though available as opt-in companion utilities, §6): any engine-specific resource tuning (DuckDB PRAGMAs, Polars thread config, Daft runtime settings), and any built-in "efficient data handoff" machinery beyond a plain function call. Both were drafted as core subsystems in earlier design iterations and were deliberately demoted — see §6 for why baking them into the core would have reintroduced the exact multi-engine complexity DuckPipe exists to avoid. --- ## 4. Landscape check — what already exists, what's a real gap Prior art looked at before committing to this scope: - **yato** (github.com/Bl3f/yato) — closest existing thing. A minimal SQL-only orchestrator: point it at a folder of `.sql` files, it infers dependencies and runs them via DuckDB. No Python task nodes, no scheduling, single-author/early-stage. Validates the appetite for this category; doesn't compete with the scope described here (fully generic Python tasks, DuckDB-backed state/fingerprinting, distributed execution). - **duckle** — a full no-code ETL/ELT platform on DuckDB (connectors, CDC, lineage). Different product category (Airbyte-alternative, not Airflow-alternative). - **pydiverse.pipedag** — a caching layer that runs *on top of* Prefect. Doesn't remove the infrastructure dependency DuckPipe removes. - No project found uses **a DuckDB file itself as the DAG's metadata/coordination store**, and no project found uses **DuckDB/DuckLake on object storage as the coordination point for stateless serverless task execution** (the common pattern instead is AWS Step Functions + Lambda, where a managed service — not DuckDB — holds DAG state). Both are genuine, currently-unclaimed territory. This means the core wedge — DuckDB file/catalog *as* the orchestrator's brain, coordinating tasks that are free to use whatever data-processing engine fits the job, or none at all — is real white space, not a reinvention. **Why signature-based DAG inference over explicit YAML/config.** This is a legitimate question (Dagster/Prefect both use Python-native inference successfully, but it's fair to ask how it scales) and worth answering explicitly. The usual scaling objection to implicit Python DAGs is actually an objection to one specific thing: a *persistent scheduler* that has to keep re-parsing/re-importing pipeline code to know the graph — this was Airflow's own documented bottleneck (§10, first row). DuckPipe has no persistent scheduler; `duckpipe run` imports the module exactly once per invocation, so that cost doesn't scale with DAG size or exist between runs at all. The other legitimate case for explicit config — non-Python-fluent stakeholders authoring pipelines, or needing to inspect DAG topology without executing code — doesn't match DuckPipe's target user (Python-fluent data/ML engineers writing their own tasks), and introducing a YAML DSL to serve it would recreate the exact "second language for control flow" complaint that pushes Airflow users toward Jinja-templated YAML in the first place. For large fan-out (hundreds of per-partition tasks), the answer is a plain Python loop generating uniquely-identified task instances — the same mechanism Dagster's dynamic outputs and Prefect's `.map()` already use — documented as a first-class pattern rather than left as a gap. --- ## 5. Concept surface — the design commitment README.md documents the actual "whole mental model" a user needs. The commitment behind it, worth stating separately: that list is deliberately short, and if it grows without a very good reason, that's a design smell (tenet #7). No `@flow` wrapper concept separate from the DAG — the DAG *is* the set of tasks plus their declared dependencies. No work pools, no workers, no blocks, no deployments-as-a-separate-entity, no task-runner selection menu, and — per tenet #2 — no resource-profile object either; that's the fully optional layer below (§6). Changing *where* code runs (§8) is a runtime setting, never a new concept. **`memory_limit_mb`, and why it clears tenet #7's bar.** `@task(memory_limit_mb=N)` is a new `Task` parameter, not a new concept: same shape as `retries`/`cache`/`depends_on` already are, opt-in (default `None`, meaning zero behavior change and zero import cost for anyone not using it — enforced by keeping `cloudpickle`/`psutil` lazy imports behind the `duckpipe[memcap]` extra, never touched by the core), and it doesn't introduce a resource-profile object or an engine-specific abstraction (tenet #2/#3 stay intact — it caps *physical RSS*, the one thing every engine's process shares regardless of which one a task happens to use). What earns it a place in the core rather than living beside the opt-in §6 utilities: it isn't engine-specific tuning advice (§6's whole reason for existing outside the core) but a scheduler-level guarantee — turning an OOM into a recorded `status="oom"` result requires the *scheduler* to own the subprocess boundary and the state-file write, the same way retries and fingerprinting already do. Proven out first in a real, external dogfooding project (a multi-engine benchmark harness enforcing the identical cap on every engine under test) before being generalized into the core, not designed speculatively. **`to_mermaid`'s `subgraphs` parameter, and why it stays a parameter, not a new mechanism.** DuckPipe's own DAG discovery has no way to know a task's body happens to run another pipeline (that would mean statically analyzing arbitrary Python — the exact kind of complexity tenet #7 exists to refuse), so it can't render that nesting automatically. What it *can* do safely (previous note) is actually run that nesting; `subgraphs` just lets the one party who does know the relationship — the pipeline's own author — say so explicitly when calling `to_mermaid`, the same "call it yourself, nothing automatic" shape `duckpipe-tuning` already uses. Mermaid's own `subgraph...end` block is the existing primitive for "a bordered box containing its own mini-flowchart, itself a node in the outer graph" — reused directly rather than inventing a DuckPipe-specific diagram notation. No new concept in the task-authoring surface at all: it's a rendering-time argument, not something a pipeline's own code ever needs to touch. Recurses to any depth for the same reason: a real pipeline that nests another pipeline whose own last task nests a third is exactly as safe as nesting once (next note), so the rendering shouldn't artificially stop at one level either. The fix wasn't a new mechanism, just removing one: each `subgraphs` entry's tuple takes an optional third element, that sub-pipeline's own `subgraphs` argument, and the recursive render passes it through instead of a hardcoded empty dict. `examples/09_nested_pipeline` demonstrates three genuine levels (not a synthetic stress test) — and building it caught a second, unrelated bug worth naming honestly: `report_card`/`report_cash` communicate their payment type to their own nested call via a process-wide environment variable, and running them concurrently (DuckPipe's default for independent tasks) races on it once there's enough work in each nested call for the interleaving to actually bite. `max_workers=1` isn't cosmetic there — see the example's own README for how that was confirmed, not assumed. --- ## 6. Optional companion utilities (opt-in, outside the core) Earlier design drafts put resource auto-tuning and a tiered lazy/eager/Arrow I/O model directly into the orchestrator core. On reflection that was a mistake worth naming explicitly: both required the core to understand engine-specific APIs (DuckDB PRAGMAs vs. Polars thread config vs. Daft runtime settings; DuckDB relations vs. Polars `LazyFrame`s vs. Daft `DataFrame`s), which is exactly the kind of cross-engine complexity multiplication tenet #3 exists to prevent — just relocated from "parallelism primitives" into "tuning and I/O primitives." The fix: keep this valuable, well-researched design work, but move all of it out of the core and into small, independent, opt-in pieces reached for only if wanted. None of what follows is imported or invoked by `duckpipe run` unless the user's own task code calls it. **6.1 — Tuning helpers, as plain functions, not a core object.** No `ResourceBudget` class, no auto-detection wired into the scheduler. Instead, `duckpipe-tuning` (a fully separate optional package, so the core dependency tree doesn't even import `psutil`) offers pure functions like `suggest_duckdb_settings()`/`suggest_thread_count()`/`suggest_temp_dir_limit()`, built from the verified DuckDB internals in §7. A user who wants tuned behavior calls one of these themselves, inside their own task, and applies it to their own connection. A documented pattern, not a framework feature. **6.2 — Cross-run/cross-process data handoff, as an opt-in caching backend.** A tiered lazy-vs-eager-vs-Arrow design only matters once data actually needs to cross a real boundary — writing a `cache=True` task's output to disk so a later run can reuse it without re-executing upstream, or shipping a result to another process/machine (§8). `cache=True` defaults to the simplest thing that works for literally any Python object — pickling — with zero engine awareness, satisfying tenet #2 completely. The Arrow-PyCapsule-based fast path (DuckDB/Polars/pandas/pyarrow all implement `__arrow_c_stream__`; Daft doesn't yet) is an **optional alternate cache backend** (`@task(cache=True, cache_backend="arrow")`) for tasks moving large tabular data that want cheaper disk round-trips than pickle — never the default, never something the core needs to understand to function correctly. **6.3 — Docs, not framework code, carry the "lazy is the obvious choice" narrative.** DuckPipe's examples use `duckdb.sql(...)`/`pl.scan_parquet(...)`/lazy Daft frames because they're genuinely the best default for most pipelines, not because the framework requires or specially recognizes them. Docs carry that narrative instead of a lint rule or adapter layer baked into `@task`. **6.4 — Where big-single-task distribution goes.** If a task genuinely needs cluster-scale, multi-machine data processing *within itself* (not DAG-level distribution), that's "use Daft's Flotilla/Ray runner inside that task" — the user's own code, the user's own dependency, nothing DuckPipe needs to wrap. DuckPipe's own distributed-execution story (§8) is about distributing *task orchestration* across ephemeral compute, not about distributing *a single task's* data processing. --- ## 7. Reference notes: DuckDB/Polars/Daft resource behavior (for the optional utilities in §6) Not a core subsystem — the technical grounding for the opt-in `duckpipe-tuning` helpers in §6.1, kept here so the research underpinning them isn't lost. **Threading.** DuckDB auto-detects host cores for its `threads` setting; over-launching on SMT/hyperthreaded machines can hurt performance (default to physical cores, not logical, except for I/O/remote-scan-heavy workloads, where DuckDB's own guidance suggests a 2–5× multiplier). **Memory.** `memory_limit` defaults to ~80% of detected RAM and bounds DuckDB's buffer manager; spillable operators are hash aggregations, sorts, and joins. Per-thread rules of thumb from DuckDB's own docs: ~1–2GB/thread for aggregation-heavy workloads, ~3–4GB/thread for join-heavy ones. **Disk.** `max_temp_directory_size` bounds spill-to-disk bytes (default ~90% of free space on the temp volume); some DuckDB versions have had temp-file cleanup edge cases worth a defensive cleanup helper if wanted ([duckdb/duckdb#14142](https://github.com/duckdb/duckdb/issues/14142)). **Concurrency safety.** A single DuckDB connection isn't thread-safe for concurrent use; per-thread `cursor()` off a shared connection is the documented-safe pattern. DuckDB releases the GIL during query execution, so it already parallelizes internally via `threads` — stacking many Python threads each issuing heavy DuckDB queries risks oversubscribing the same core pool rather than adding throughput. Advice for task authors, not a scheduler feature. **Streaming accessors.** DuckDB's current (non-deprecated) streaming Arrow accessor is `to_arrow_reader()`, returning a real `pyarrow.RecordBatchReader`. Polars' streaming engine (`collect(engine="streaming")`/`.sink_*()`) is recommended but still opt-in, not default. Daft is lazy by default with two runners — Swordfish (single-machine streaming) and Flotilla (Ray-distributed) — see §6.4. --- ## 8. Distributed, serverless, and browser execution The primitive everything below runs on: object-storage-native, put-if-absent conditional writes (`duckpipe.remote.locked()`), no server or catalog database involved (§12 has the full writeup of why this beat both a persistent Quack server and a full DuckLake catalog for the underlying concurrent-state-file problem). **Task-scoped execution — one new concept, and it's a narrowing of an existing one, not an addition beside it: `duckpipe.run(module, only=task_name)` (`--only` on the CLI) runs exactly one task instead of the whole DAG.** Locally this is trivial — build the DAG, execute one node instead of all of them, same file, same OS-level lock as always. Against a `state_uri`, it's what turns a single shared state file into something many nodes can contribute to at once, without ever generalizing `locked()` into something heavier: - A scoped run downloads state as usual, absorbs any pending contributions from other concurrent invocations, decides skip-or-run for its one task exactly like a whole run, and — if it ran — writes *only the rows it produced* to a small file under a unique key in the same bucket, instead of re-uploading the whole state file. A uniquely-keyed object can never collide with anyone else's write, so this needs no lock at all. - "A small file" is not a new format: it's a `duckpipe.db` that happens to hold only a few rows, because that's the shape the existing `task_runs`/`task_fingerprints`/`task_cache` schema already has for "what one task run adds." Absorbing one is `ATTACH` plus a few `INSERT ... SELECT`, using tables that already exist. - What this buys: many nodes can each run `duckpipe run pipeline.py --only ` against the same `state_uri` at the same time, on different tasks — or even the same task redundantly, since fingerprint-based skip already makes redoing finished work harmless rather than corrupting. That's a real multi-node cluster run, no DuckLake or Quack required. - Dispatch — deciding which node runs which task, in what order — stays outside DuckPipe, on purpose (tenet #1). `duckpipe show` already resolves topological order and previews skip-vs-run per task; that's the discovery primitive a coordinator (a Step Functions state machine, a Modal `.map()`, a shell loop over SSH) needs, and each dispatched unit is the same command any trigger already calls (§9), just narrower in scope. See it actually running one in `examples/04_distributed_cluster`. **DuckLake-backed state — an observability upgrade, not a coordination one.** `db_path="ducklake:sqlite:pipeline.ducklake.sqlite"` — the same argument a plain file goes in, pointed at a different kind of string; `duckpipe run`/`show`/`stats` don't change at all. What it actually buys: every task's outcome becomes its own DuckLake snapshot, tagged with a plain-English commit message, so `task_runs AT (VERSION => n)` turns "what happened" into a real, queryable history instead of a present-tense table — and schema evolution (`ALTER TABLE ... ADD COLUMN`) needs no migration step. See `examples/06_ducklake_observability`. The same catalog string can instead be backed by a dedicated, long-running metadata database — `db_path="ducklake:postgres:dbname=... host=..."` (or MySQL), with `data_path` passed explicitly since there's no local file to derive a sibling directory from. Nothing about `duckpipe.run(...)` changes; `state.py`'s catalog-extension install and DATA_PATH handling are already generic across sqlite/duckdb/postgres/mysql. What it buys over the SQLite catalog: several DuckPipe deployments — different teams, tenants, or machines — sharing one catalog with genuine concurrent-write support. Verified directly, not assumed: 8 concurrent commits against a real Postgres catalog (no retry logic) all succeeded, where the identical test against a SQLite catalog failed 3 of 8 outright (see `examples/05_distributed_with_ducklake`'s SQLite-vs-Postgres writeup). Entirely optional — the SQLite catalog needs no such infrastructure and stays the right default for a single team's own history. Deliberately *not* wired to `state_uri`/`only=` regardless of which catalog backs it — both raise clearly if combined with a `ducklake:` `db_path` — three independent, verified reasons why, not a hunch: (1) a SQLite catalog can't live on an object store (DuckLake's own maintainers confirm this), so it would still need the exact sync dance this backend exists to not need; (2) a Postgres/MySQL catalog removes that limitation but is provisioned infrastructure, which is why it stays opt-in rather than becoming the distributed mechanism's default; (3) even against Postgres, DuckLake's optimistic concurrency control is retry-based (documented real failures under load in [duckdb/ducklake#233](https://github.com/duckdb/ducklake/issues/233), since fixed) — strictly weaker than the delta-merge mechanism's conflict-free-by-construction guarantee for "many small workers each report one fact." `examples/05_distributed_with_ducklake` demonstrates that narrower coordination use directly, on its own terms, separate from DuckLake's actual wiring into `duckpipe.run()`'s own state store described above. **Reference serverless executor.** A task becomes a small function on Lambda/Modal/a container/etc.; each invocation is `duckpipe.run(module, only=task, state_uri=...)` — already a plain Python function with no platform-specific glue, so "pick a reference platform" would mostly produce code whose only job is proving something already structurally true. `examples/07_serverless_executor` dispatches one DAG's two tasks through two genuinely different invocation shapes into the *same* distributed run — a container (`docker run`, argv-invoked, the way ECS/Cloud Run/a Kubernetes Job calls one) and a `handler(event, context)` function (the calling convention Lambda/Modal/most FaaS platforms actually use) — making the "not locked to one platform" claim checkable instead of asserted. **Quack** doesn't fit this path at all — it requires a persistent DuckDB server, a direct contradiction of "stateless executor" — so it's not part of DuckPipe's own sync layer; it remains available for a *coordinator* to optionally run for one distributed run's lifetime, never as DuckPipe's default. **Remote "beefy node" mode.** One job too large for a laptop but not worth distributing: the same code runs unchanged on a bigger remote machine, optionally paired with the §6.1 tuning helpers. Needs no coordination model at all — no `state_uri` locking, no delta files, no dispatch coordinator — just `rsync` the pipeline over, `ssh` in, run the exact same command. See `docs/remote_execution.md`. **Browser execution (via Pyodide).** Verified directly with DuckPipe's actual source loaded into a live Pyodide runtime, first via a Node.js harness for the initial spike and then via an actual headless browser (Playwright + Chromium) for the shipped example — not assumed from DuckDB's own WASM marketing. The full `duckdb` Python package (real SQL engine, not a stub) loads via `pyodide.loadPackage("duckdb")` straight from Pyodide's own package repository. DuckPipe's core — `task.py`, `dag.py`, `fingerprint.py`, `remote.py`, `state.py`, `scheduler.py` — imports and runs a real multi-task pipeline through `scheduler.run()` inside that runtime with **zero code changes**, `asyncio.run()` included. The one real gotcha: `asyncio.run()` inside Pyodide needs the host to invoke Python via the promise-based entry point (`runPythonAsync`) and needs WASM JSPI (stack-switching) support in the JS engine — standardized (W3C Wasm CG Phase 4) as of this writing, shipping in Chrome 137+ stable with no flag, Firefox 139 behind a flag, Safari committed. `examples/08_browser_wasm` ("is this file safe to send anywhere?") is the shipped deliverable, built around a concrete niche rather than a generic demo — see its own README for the full writeup, including two things checked rather than claimed (zero network requests for a user-picked file; state genuinely surviving a real page reload via IndexedDB). Honest, scoped-not-glossed-over limitation: no `state_uri` remote sync and no DuckLake backend in-browser — Pyodide's DuckDB build has no runtime-loaded extensions (`httpfs`/`ducklake` unavailable), it's single-threaded (a Pyodide constraint on DuckDB's own internal parallelism, not on DuckPipe's `asyncio`-based task concurrency), and whether `fsspec` works against Pyodide's virtual filesystem/CORS-constrained fetch model is genuinely unverified — the next open spike, not a claimed result. Nothing here changes DuckPipe's task-authoring API (§5) — a task function written for local mode is the same function deployed to a Lambda executor or run inside a browser tab, just invoked with a narrower scope or a different host. --- ## 9. Interoperability: embedding DuckPipe inside other orchestrators A deliberately small addition, worth stating explicitly rather than leaving implicit: **DuckPipe should always be embeddable as a single task/step inside Airflow, Prefect, Dagster, or anything else, without any special integration code on DuckPipe's side.** This isn't a new subsystem — it's a free consequence of tenets #1 (no persistent daemon) and #5 (local dev and production are the same code path): a DuckPipe pipeline is just `duckpipe.run(module)`, an importable Python call that starts, does work, and exits, indistinguishable from any other function a `PythonOperator`, a Prefect `@task`, or a Dagster `@op` might wrap. The only discipline required is to never add something that assumes DuckPipe owns the top-level process (a global signal handler, an entrypoint with no clean importable form) — that's a constraint to preserve, not a feature to build. **Where this is a genuinely good idea, not just a curiosity:** - **Adoption wedge for teams that can't or won't replace their existing orchestrator.** A nightly transform expressed today as fifteen brittle, XCom-shuffling Airflow tasks can collapse into a single Airflow task that internally runs one DuckPipe pipeline — Airflow keeps doing what it's organizationally good at (cross-team scheduling visibility, existing alerting, audit trail), while the data-heavy inner loop gets DuckPipe's in-process data movement and fine-grained fingerprint-based incrementality. - **Local dev/test parity.** A data engineer iterates against the DuckPipe pipeline directly (seconds, no scheduler/DB test harness) and only wraps the finished, unmodified module in a one-line task when deploying — directly addressing "DAGs are hard to test locally." - **Fine-grained incrementality inside a fan-out unit.** If dynamic task mapping runs one sub-pipeline per partition, and that sub-pipeline is itself a small dependent-step DAG, each mapped instance can be "run a DuckPipe pipeline for partition X," giving step-level skip-if-unchanged behavior the outer platform's task-level caching can't express. **Where to deliberately *not* go, because it's the exact scope creep this whole design exists to avoid:** no DuckPipe-side awareness of the host orchestrator. No Airflow provider package with custom operators/sensors/hooks, no DuckPipe retries deferring to the host's retry semantics, no fingerprints flowing through XCom. From the host's point of view, `duckpipe.run(module)` stays an opaque, atomic unit — success, failure, duration, nothing more. The entire "integration" is a five-line documented recipe per platform (`docs/interop.md`), not a maintained package. **Also worth stating plainly:** nesting DuckPipe inside a heavier platform is a decision driven by real governance/observability needs the outer platform provides (existing alerting, RBAC, multi-team visibility) — not something to do reflexively just because Airflow happens to already be installed. A pipeline that doesn't need any of that is still better served by plain cron/CI directly invoking DuckPipe (tenet #1), which stays lighter for that case. --- ## 10. Competitive positioning (pain point → design response) | Researched pain point | Source | DuckPipe response | |---|---|---| | Airflow: scheduler re-parses every DAG file on a fixed loop; degrades at scale | [GitHub discussion #44727](https://github.com/apache/airflow/discussions/44727), [cutting DAG parse time from 60s to ms](https://medium.com/@adrianmroz.7/optimising-airflow-cutting-dag-parse-time-from-60-s-to-milliseconds-a-practical-guide-part-1-d15081f419ae) | No persistent scheduler/parser process at all; "trigger" is just invoking the run command | | Airflow (FLYR postmortem): CeleryExecutor requires pre-allocated workers; wastes spend; KubernetesExecutor caused unexplained task failures at high parallelism | [FLYR Labs blog: "Why We're Switching Off Airflow — Sort Of"](https://medium.com/flyr-labs-blog/why-were-switching-off-airflow-sort-of-780c4f58a660) | No broker/worker-pool concept; scaling is "bigger single node" or stateless serverless execution (§8), not a worker fleet to keep healthy | | Airflow: XCom size limits, awkward inter-task data passing | [Airflow XCom docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html) | In-process task-to-task handoff is a plain function call (§2, tenet #2) — no serialization, no size cap, no side-channel | | Prefect: Flow→Deployment→Flow Run hierarchy confuses users; not resolved in onboarding (issue closed "not planned") | [GitHub #10811](https://github.com/PrefectHQ/prefect/issues/10811) | No separate "deployment" entity — a pipeline module *is* the runnable unit | | Prefect: work pools/workers/blocks/task-runner menu — "which one do I need" is a recurring question | [Prefect docs: work pools](https://docs.prefect.io/v3/concepts/work-pools), linen.prefect.io threads | No work pools/workers/blocks; one concurrency model in the core, with any engine-specific tuning left to the user's own code or the optional §6 utilities | | Prefect: users reach for wrapping pandas in `DaskTaskRunner` instead of using the data engine's own parallelism | [GitHub Discussion #3022](https://github.com/PrefectHQ/prefect/discussions/3022) | Explicit tenet (#3): the core never ships a competing parallelism or tuning primitive; an engine's own controls (or the opt-in helpers in §6) are the answer | | Prefect: 1→2→3 breaking rewrites; migration pain even in typed codebases | [Prefect migration guide](https://docs.prefect.io/v3/how-to-guides/migrate/upgrade-to-prefect-3), [issue #15275](https://github.com/PrefectHQ/prefect/issues/15275) (2→3, mypy breakage) | Small, boring core API (§5) deliberately resists growth; task-authoring surface is the one thing committed to keeping stable | | Dagster: assets-vs-ops split-brain — two overlapping graph paradigms | [Discussion #10512](https://github.com/dagster-io/dagster/discussions/10512) | One task/dependency model, no parallel paradigm | | Dagster: "I/O managers aren't required" but a default one is silently initialized anyway — contradicts stated minimalism | [Issue #32065](https://github.com/dagster-io/dagster/issues/32065) | The core has no IO-manager concept to be silently mandatory about — data never passes through the framework at all (§2, §6) | | No tool auto-tunes DuckDB resource settings to host specs | Own DuckDB internals research — confirmed gap | Filled by the opt-in §6.1 utilities, deliberately kept outside the core so it never becomes a second engine-abstraction layer | | No tool re-runs only what changed, by default, at the task level (dbt/SQLMesh do this for SQL models only) | [SQLMesh docs](https://sqlmesh.readthedocs.io/en/stable/concepts/overview/), [dbt `--state` docs](https://docs.getdbt.com/reference/node-selection/state-comparison-caveats) | Fingerprint-based incrementality is default behavior for *any* task, not just SQL models | | No lightweight orchestrator lets a single task's physical memory be capped with the breach recorded as a first-class task outcome, rather than a crash, a silent OS OOM-kill, or standing up a whole resource-manager subsystem | Own dogfooding — a real multi-engine benchmark harness needed exactly this and built it standalone before DuckPipe had an equivalent (confirmed gap, not researched against a specific competitor issue) | `@task(memory_limit_mb=N)` (§5) — opt-in, scheduler-level, `status="oom"` | --- ## 11. Implementation notes A few build-time findings worth keeping, grouped by subsystem rather than by when they happened: **Fingerprinting.** An edge is a parameter whose *default value* is another `Task` object (`def b(x=a)`) — unambiguous, no type-hint parsing, structurally excludes `*args`/`**kwargs` (Python forbids defaults there). Confirmed with a toy 4-task diamond DAG (`tests/fixtures/toy_dag.py`): `cache=True` everywhere skips entirely on an unchanged re-run and fully re-executes under `--force`. Concurrent-cursor fan-out (several DuckDB queries sharing one connection via `cursor()`) measured a real 2.2x speedup over serial execution on one benchmark machine (`scripts/phase0_bench_fanout.py`) — the basis for using a plain `asyncio` + thread-pool scheduler rather than pulling in a third-party graph-execution library. **Nesting `duckpipe.run()` inside a task is safe, confirmed empirically, not just reasoned about.** A task's synchronous body always runs via `loop.run_in_executor(None, ...)` (a thread-pool worker thread with no event loop of its own) rather than directly on the scheduler's own event loop — so a task that itself calls `duckpipe.run(other_pipeline)` (which internally calls `asyncio.run()` again) never collides with the outer scheduler's loop: `asyncio.run()` only raises when called from a thread that already has one running, and the executor thread doesn't. Verified directly with a real two-level pipeline (an outer task nesting a full inner `run()` on a separate module, its own state file) before relying on it. This is what makes `to_mermaid`'s `subgraphs` parameter (§5) meaningful: rendering a task as containing another real pipeline's shape reflects something DuckPipe actually supports running, not just drawing. **Resume and observability.** Partial-DAG resume needed no new API: a failed task never gets a fingerprint/cache entry, so re-running the same command naturally re-executes only the failed task and its downstream, while unaffected `cache=True` upstream tasks skip via the existing incrementality mechanism. What genuinely needed building: a task cascade-skipped because its upstream failed gets its own `task_runs` row (status `upstream_failed`) instead of vanishing from the state file — and a skip-if-unchanged cache hit needs its own row too — otherwise resume and `duckpipe stats` would silently undercount. **Concurrent state-file access (§9's lock).** Two heavier alternatives were weighed and ruled out for the "many workers touching one remote-synced state file" problem: **Quack** (DuckDB's client-server protocol) needs a persistent DuckDB *server* process, reintroducing exactly the standing infrastructure tenet #1 refuses — at most a fit for a coordinator process in the serverless-executor path (§8). **DuckLake** gives genuine multi-writer ACID, but only once its catalog is itself a reachable, always-on database service; a SQLite-file catalog doesn't help here because that catalog file would need the identical download-mutate-upload round-trip this problem is about, one level down. What shipped instead: an advisory lock object next to the state file, using each object-store backend's own native conditional-write primitive (S3 `If-None-Match`, GCS `if_generation_match`, Azure ETag preconditions) through fsspec's exclusive-create (`"x"`) file mode — no server, no catalog database, held for the whole download-run-upload sequence, wired into `duckpipe.run(..., state_uri=...)` by default. A stale lock (crashed holder) is reclaimed after `max_lock_age` rather than deadlocking forever. This same put-if-absent primitive turned out to be the right foundation for task-scoped concurrency too (§8) — that mostly reuses it at a finer grain rather than needing something new. **DuckLake schema portability.** Dropping `PRIMARY KEY`/`ON CONFLICT` from the whole schema (DuckLake supports neither at all, verified directly) in favor of one portable `DELETE`-then-`INSERT`/`INSERT ... WHERE NOT EXISTS` idiom turned out to be a simplification, not a workaround — both backends now share one dialect. `CREATE OR REPLACE VIEW` was found to commit a new DuckLake snapshot on every single store open even when byte-identical, which would have flooded the exact run-history story that backend exists to make useful; fixed with `CREATE VIEW IF NOT EXISTS`. **Packaging.** `duckpipe-tuning` and the Arrow-aware `cache_backend="arrow"` option shipped as genuinely optional: `packages/duckpipe-tuning/` is a real second distribution in the uv workspace, zero psutil/DuckDB coupling in `duckpipe` itself (enforced by a real test, not just a docstring promise); `cache_backend="arrow"` lives behind a lazy `pyarrow` import and the `duckpipe[arrow]` extra. Daft's own Arrow-PyCapsule support ([Eventual-Inc/Daft#2504](https://github.com/Eventual-Inc/Daft/issues/2504)) is still pending upstream as of this writing — a documented known gap, not silently papered over. **`memory_limit_mb` (§5).** Two real bugs surfaced only by testing the actual watchdog rather than reasoning about it: (1) `only=`-scoped execution (`_execute_one`, Phase 3a) is a genuinely separate code path from the whole-DAG one (`_execute_dag`/`run_node`) — both needed their own `isinstance(exc, MemoryLimitExceeded)` check to record `status="oom"` instead of a generic `"failed"`, and it's easy to fix one and miss the other. (2) The SQL views (`v_task_stats`/`v_run_summary`) hardcode their failure-status list (`status IN ('failed', 'upstream_failed')`) for `failed_count` — a new status value silently undercounts unless that list is updated too, precisely the class of bug this project already went out of its way to avoid for `upstream_failed` itself (see the "Resume and observability" note above). Separately, confirmed directly: cloudpickle prefers pickling a function *by reference* (re-import by dotted module name) whenever the module looks normally importable, and gets this specifically wrong for a function defined in a Jupyter cell or a pytest test module — neither is reliably re-importable from a fresh subprocess's own `sys.path`, though cloudpickle can't tell that from outside. A task defined in an actual pipeline file (the normal case) doesn't hit this. **`max_workers`, two real bugs found by measuring actual scheduling behavior, not by reasoning about the semaphore.** (1) A task's concurrency slot was acquired *before* it checked whether its own upstream dependencies had finished, not after — so a task that's merely waiting on a dependency could win a slot and idle in it for that dependency's entire duration, leaving genuinely independent, ready-to-run tasks with no slot to run in even though nothing was actually contending for a resource. Confirmed directly, not assumed: a 4-task DAG (one slow root, one task depending on it, two independent tasks) ran *slower* under `max_workers=2` than fully unbounded — 3.40s vs. 2.07s, a 64% regression from adding a limit that should have been more than sufficient — because the dependent task's name happened to sort right after its own dependency in `topological_order`'s alphabetical tie-break, so it consistently won a slot ahead of the independent tasks and sat in it, idle, doing nothing. Fixed by moving the dependency-wait outside the semaphore-guarded region — no new concept, just an `await` moved from inside `run_node` to before `async with sem` in `run_guarded`. (2) `max_workers=None` ("unbounded") doesn't mean unbounded: task bodies dispatch via `loop.run_in_executor(None, ...)`, and `None` there means Python's own lazily-created default `ThreadPoolExecutor`, capped at `min(32, cpu_count + 4)` — 14 on a typical 10-core machine. A wide fan-out (§4's own documented pattern) silently queues behind that cap regardless of what the DAG's own shape could support. Confirmed directly: 50 independent 0.5s tasks under `max_workers=None` took 2.84s, not ~0.5s. Fixed with an explicit `ThreadPoolExecutor` sized to `max_workers` when bounded, or to the DAG's own task count when not — the real ceiling on how many could ever be in flight at once, so "unbounded" means what it says without introducing an arbitrary number of its own. **A correction to the above, from digging further.** The "~30ms/task" figure first measured wasn't actually a per-task cost at all — isolating it precisely (a 1-task run vs. a 50-task run in the same process) showed the 1-task run alone cost ~990ms, and the *other 49 tasks* added only ~82ms combined. The real marginal cost per task, measured directly against `StateStore` (lineage + task-run + fingerprint writes, one transaction) with the one-time cost warmed out of the way first, is ~2ms — genuinely fine, not a scaling problem for "organizing small things" inside one `run()` call. The ~800ms–1s isn't DuckPipe's own code at all: it's DuckDB's Python client's *first-ever parameterized write* on a connection, confirmed by isolating exactly that statement (subsequent inserts on the same connection, any table, any shape, any parameter count, all ≥1000x faster) and confirmed as a real, currently-unresolved, "wontfix"-tagged DuckDB characteristic ([duckdb/duckdb-python#238](https://github.com/duckdb/duckdb-python/issues/238), [duckdb/duckdb#20824](https://github.com/duckdb/duckdb/issues/20824)) — not something pragma-tunable (`threads=1` vs `8` made no reliable difference across isolated processes) or fixable from outside the engine. DuckPipe already, incidentally, pays this cost in the right place: `start_run()`'s own insert runs before any task's timer starts, confirmed directly (a task's own first write, run right after, costs ~0.6ms) — so it isn't silently inflating a task's recorded duration in `task_runs`. Where this genuinely matters: not a single `run()` call (the cost is fixed and one-time, paid once regardless of task count), but the `only=`-per-subprocess dispatch pattern (examples 04/07/10) — every dispatched subprocess is a fresh DuckDB connection, so every one repays this ~0.5–0.8s floor. That's a real, inherent cost of choosing process-per-task isolation, best addressed by knowing it's there (favor this pattern for tasks whose own work clearly exceeds the floor) rather than by trying to engineer it away. --- ## 12. Open questions and past decisions **Still open:** 1. **Versioning/compatibility promise.** A policy decision, not an implementation one. Given Prefect's breaking-rewrite pain was a named complaint (§10), worth deciding before any 1.0. 2. **`fsspec` in Pyodide.** Whether it works against Pyodide's virtual filesystem/CORS-constrained fetch model is genuinely unverified — needed before `state_uri` sync (or the DuckLake backend) means anything in-browser (§8). **Decided, and why it's not revisited:** - **Dependency inference mechanics.** An edge is a parameter whose default value is another `Task` object — no `Depends(...)` marker needed, no type-hint parsing, no ambiguity. - **Fingerprint scope.** The hash covers a task's own source, its declarative config, and upstream fingerprints — never closed-over globals, imported module versions, or output data. An upstream *external* state change (a file that changed on disk without the task's own code changing) is invisible to it by design; `@task(extra_fingerprint=[...])` is the opt-in escape hatch. A footgun worth naming explicitly: if what you pass changes *as a side effect of the task's own run* (a notebook whose outputs get rewritten by executing it, say), the cache invalidates itself on every single run — key it on something the run doesn't itself mutate (a source file's own content, not its full byte-for-byte state). - **Pickle-as-default-cache limitations.** A size warning (`LARGE_CACHE_WARN_BYTES`) fires above 50MB; an object that can't be pickled at all (a live `DuckDBPyRelation`, a raw DB handle) degrades to a logged warning rather than failing the run. `cache_backend="arrow"` (§6.2) is the opt-in alternative for large tabular results. - **Boundary for the `duckpipe-tuning` helpers.** Every function takes only host specs (CPU count, total RAM, free disk) and returns a suggestion — none accept a DuckDB connection, run a query, or inspect data. Enforced by a real test, not just a docstring promise. - **`duckpipe show --mermaid` stays flat; nesting isn't a CLI feature.** Considered three shapes for wiring `to_mermaid`'s `subgraphs` (§5) into the CLI: a new `@task(nested_pipeline=...)` parameter (rejected — contradicts §5's own "rendering-time argument, never a task-authoring concept"); a magic module-level name `show` looks for, e.g. a `SUBGRAPHS` variable or a `duckpipe_subgraphs()` function (rejected — every existing discovery mechanism finds things by *type* (`isinstance(value, Task)`, including inside lists/dicts), never by a specific variable/function *name*; this would be a new, precedent-setting kind of magic); a repeatable `--subgraph name=path` flag (rejected on cost/benefit — real argument parsing and per-sub-pipeline state-file resolution, reimplementing in `cli.py` what a ~15-line script already does directly against the public `to_mermaid`/`build_dag` API, for a need that's opt-in and rare). `examples/09_nested_pipeline/show_nested_mermaid.py` is the answer instead — the same "call it yourself" shape `duckpipe-tuning` is already built around, not a workaround standing in for a missing feature. - **Final project name.** Settled by publishing: "duckpipe" has been on PyPI since 0.1.0, with a real external dependent (a separate production project) pinning it as a genuine dependency. Renaming now would break real consumers for no offsetting benefit — the placeholder became the name the moment the first outside project took a dependency on it. - **Concurrency default.** Independent tasks (no edge between them) run with no concurrency limit unless `max_workers=` is set — deliberate, since most pipelines' tasks are cheap/IO-bound relative to the whole run and benefit from running together. "No limit" means an executor genuinely sized to the DAG's own task count, not Python's own much smaller default thread-pool size (§11) — the distinction used to be invisible, and isn't anymore. The footgun this creates: tasks that *do* compete for the same physical resource (several benchmark-style measurements on one box, say) need `max_workers=1` (or however many the resource can actually afford) stated explicitly — DuckPipe has no way to infer "these tasks shouldn't overlap" from the DAG shape alone, and shouldn't guess. --- ## 13. Appendix — key verified facts underpinning this design *(For implementer reference; sources are live links, verified August 2026. The DuckDB/Polars/Daft/Arrow-specific facts below ground the optional §6 utilities, not the core; the fsspec/DuckDB-remote-attach facts ground §2/§9's state-persistence design.)* - DuckDB auto-detects host cores for default `threads`; morsel-driven parallelism needs enough row-group-sized morsels to use all threads; SMT/hyperthreaded over-launch can hurt performance. [DuckDB: Tuning Workloads](https://duckdb.org/docs/current/guides/performance/how_to_tune_workloads) - `memory_limit` defaults to ~80% of detected RAM; spillable ops are hash aggregations, sorts, and joins; too-low a limit on a blocking operator can still throw `OutOfMemoryException`. [DuckDB: Memory Management](https://duckdb.org/2024/07/09/memory-management), [OOM Errors guide](https://duckdb.org/docs/lts/guides/troubleshooting/oom_errors) - `max_temp_directory_size` bounds spill-to-disk bytes; some versions have had temp-file cleanup edge cases. [DuckDB Limits](https://duckdb.org/docs/current/operations_manual/limits), [duckdb/duckdb#14142](https://github.com/duckdb/duckdb/issues/14142) - `DuckDBPyRelation` (from `.sql()`, `.table()`, `.read_parquet()`, etc.) is lazy and composes freely before execution. Its streaming Arrow accessor is now `to_arrow_reader()` — the older `fetch_arrow_reader()`/`fetch_record_batch()` names are deprecated as of 2026; `to_arrow_reader()` returns a real `pyarrow.RecordBatchReader` that streams batch-by-batch rather than materializing. DuckDB recommends partitioned Parquet read/write for larger-than-RAM workloads rather than full materialization. [DuckDB Quacks Arrow](https://duckdb.org/2021/12/03/duck-arrow), [DuckDB Arrow export docs](https://duckdb.org/docs/current/guides/python/export_arrow) - **Polars `LazyFrame`** (`.scan_parquet()`, `.scan_csv()`, etc.) is lazy the same way. Its rewritten streaming engine (3–7x faster than the in-memory engine on larger data) is the *recommended* engine as of late 2025 but is still **opt-in**, not default — invoke via `lf.collect(engine="streaming")` or `.sink_parquet()`/other `.sink_*()` methods for a fully streaming write. [Polars in Aggregate (Dec 2025)](https://pola.rs/posts/polars-in-aggregate-dec25/), [Polars streaming guide](https://docs.pola.rs/user-guide/concepts/streaming/) - **Daft `DataFrame`** is lazy by default (builds a logical plan; executes on `.collect()`/`.show()`/`.write_parquet()`). Two runners: **Swordfish** (single-machine, Rust, streaming/out-of-core via async channels) and **Flotilla** (distributed, each Ray worker running Swordfish over its partition) — the latter is the natural answer when a single task genuinely needs multi-machine data processing (§6.4), rather than DuckPipe building its own distributed dataframe engine. [Daft Architecture](https://docs.daft.ai/en/stable/architecture/) - **Arrow PyCapsule interface** (`__arrow_c_stream__`/`__arrow_c_array__`) is the standardized zero-copy interchange mechanism for eager objects; confirmed implemented by pyarrow, pandas (2.x+), Polars, and DuckDB — the basis for the optional §6.2 `cache_backend="arrow"`. Daft's eager-output support for this interface is still pending (tracked in Eventual-Inc/Daft#2504 as of this research), so a Daft-produced eager result would need `.to_arrow()` + `.to_batches()` chunking instead. [Narwhals + Arrow PyCapsule](https://labs.quansight.org/blog/narwhals-pycapsule), [Daft PyCapsule issue #2504](https://github.com/Eventual-Inc/Daft/issues/2504) - A single DuckDB connection isn't thread-safe for concurrent use; per-thread `cursor()` off a shared connection is the documented-safe pattern; DuckDB releases the GIL during execution. [DuckDB: Multiple Python Threads](https://duckdb.org/docs/current/guides/python/multiple_threads), [duckdb-python discussion #40](https://github.com/duckdb/duckdb-python/discussions/40) - **Quack protocol** (May 2026): DuckDB-to-DuckDB client-server communication over HTTP/TCP, multi-writer without lock contention, token auth, port 9494. [DuckDB blog](https://duckdb.org/2026/05/12/quack-remote-protocol) - **DuckLake** (v1.0, April 2026): catalog (ACID SQL DB) + storage (Parquet) separation; concurrent multi-writer with ACID, time travel, schema evolution; production users include PostHog. [DuckLake](https://ducklake.select/), [DuckDB docs](https://duckdb.org/docs/current/core_extensions/ducklake) - **DuckDB-WASM**: full engine compiled to WASM, runs in-browser or minimal edge builds (e.g., Cloudflare Workers); single-threaded by default, ~4GB memory ceiling. [DuckDB WASM docs](https://duckdb.org/docs/current/clients/wasm/overview) - **DuckDB-python under Pyodide**: the *Python* `duckdb` package, not just the JS `duckdb-wasm` library, is compiled to WebAssembly in its entirety and ships in Pyodide's own package repository — confirmed by DuckDB's own writeup and independently verified in-session by loading DuckPipe's real source and running a full pipeline through it. Its Pyodide build has no runtime-loaded extensions and can't reach remote files directly (pull them into Pyodide's virtual filesystem first). [DuckDB: DuckDB in Python in the Browser with Pyodide](https://duckdb.org/2024/10/02/pyodide) - **WebAssembly JSPI (JavaScript Promise Integration)**: lets synchronous WASM code (e.g. Python's `asyncio.run()` inside Pyodide) block on an async JS operation without deadlocking the host event loop — required for DuckPipe's `scheduler.run()` to work unmodified inside Pyodide. Standardized (W3C Wasm CG Phase 4) as of this writing; shipping in Chrome 137+ stable with no flag, behind a flag in Firefox 139, committed (engineer assigned) in Safari. [V8: Introducing the WebAssembly JSPI API](https://v8.dev/blog/jspi), [Chrome for Developers: WebAssembly JSPI origin trial](https://developer.chrome.com/blog/webassembly-jspi-origin-trial) - **DuckLake constraints verified directly, not assumed**: DuckLake supports no `PRIMARY KEY`/`UNIQUE` constraints at all (confirmed via `NotImplementedException` from the extension itself), so `ON CONFLICT` upserts are unusable against it. A SQLite (or DuckDB-file) catalog cannot live on object storage — a DuckLake maintainer states this is "a fondamental design decision, sqlite can not work on an object store" — so only a live network catalog (Postgres/MySQL, self-hosted or managed) gives a DuckLake-backed store genuine attach-directly-no-sync-dance behavior. [duckdb/ducklake discussion #519](https://github.com/duckdb/ducklake/discussions/519), [duckdb/ducklake issue #233 (concurrent writes)](https://github.com/duckdb/ducklake/issues/233) - **Postgres-backed DuckLake catalog, verified directly against a real Postgres 17 instance**: `duckpipe.state`'s catalog-extension install and DATA_PATH handling needed zero code changes to work against `db_path="ducklake:postgres:..."` — confirmed with `StateStore` itself, including its `read_only=True` path and the CLI's `stats --snapshots`. The concurrency payoff is real, not just DuckLake's own marketing: 8 concurrent, no-retry commits against a fresh Postgres catalog all succeeded, where the identical test against a fresh SQLite catalog failed 3 of 8 outright with `TransactionException: ... database is locked` (matching `examples/05_distributed_with_ducklake`'s own earlier, separately-verified finding). - SQLMesh's virtual-environment/fingerprint model (content-hash per model, environments as pointers over physical tables) is the direct inspiration for DuckPipe's task-level incrementality. [SQLMesh Overview](https://sqlmesh.readthedocs.io/en/stable/concepts/overview/) - Landscape check found no existing project using a DuckDB file/catalog as the orchestrator's own coordination/state store, locally or across serverless invocations — this is the core validated gap. (yato, duckle, pydiverse.pipedag reviewed as closest prior art.) - **`fsspec`** is the standard uniform Python filesystem abstraction (`AbstractFileSystem`, `fsspec.open()`), with `s3fs`/`gcsfs`/`adlfs` providing S3/GCS/Azure Blob support through the same interface local disk uses; it originated out of Dask specifically to avoid per-backend adapter code, and pandas/Dask/xarray all use it as their storage I/O layer. This is the basis for §2/§9's state-persistence design. [fsspec docs](https://filesystem-spec.readthedocs.io/en/latest/), [fsspec background](https://filesystem-spec.readthedocs.io/en/latest/intro.html) - **DuckDB's own database file format supports remote `ATTACH` (via `httpfs`, e.g. `ATTACH 's3://bucket/file.duckdb'`) for read-only access only** — its docs state explicitly that writing the database via HTTPS or the S3 API is not possible; read-write access is filesystem-local only. This confirms a download-before-run/upload-after-run sync pattern (not a live remote-attached database) is the correct, currently-only approach for a read-write state file on object storage. [DuckDB: Attach over HTTPS or S3](https://duckdb.org/docs/current/guides/network_cloud_storage/duckdb_over_https_or_s3), [duckdb/duckdb#10967](https://github.com/duckdb/duckdb/issues/10967)