A serverless-first, DuckDB-native pipeline orchestrator.
Why DuckPipe is built the way it is. For what it does and how to use it, see README.md — this document is the reasoning underneath, kept around because several error messages and docstrings point back to specific sections here, and because the tradeoffs are worth preserving even once the features they justify are just… how the thing works.
Most teams reach for Airflow, Prefect, or Dagster the moment they need “more than one script that depends on another script.” All three solve that by asking you to first stand up persistent infrastructure — a scheduler, a webserver, a metadata database, and usually a broker/worker pool — before you’ve run a single real pipeline. That infrastructure tax is fixed cost, paid whether your pipeline moves a few thousand rows once a day or a billion rows every minute.
DuckDB’s own rise made a similar point about query engines: a huge share of “big data” workloads people defaulted to Spark for were never actually big-data workloads — they fit comfortably on one modern machine, and a single well-tuned embedded process finishes in seconds what a cluster took minutes to schedule. DuckPipe applies that same correction to orchestration, but to the infrastructure, not to a data-size ceiling: most pipelines don’t need standing infrastructure to be well-organized, incremental, and observable, regardless of how much data they move. The orchestrator for that case should be a library, not a platform — importable, runnable as a plain Python process, requiring zero standing infrastructure, and disappearing entirely when it’s not running. Scaling out (§8) is an extension of that same core for when a single node genuinely isn’t enough, not evidence a pipeline has outgrown DuckPipe.
The bet: if the single-node, zero-server experience is right — genuinely as easy as pip install duckpipe and running a Python file — there’s “almost no reason not to use it,” the same way there’s rarely a reason to reach for Spark before trying DuckDB first. Distributed/remote execution and browser execution are extensions of the same core (§8), not a redesign.
No persistent servers required to get started. No scheduler daemon, no central metadata database, no broker. A DuckPipe run is a process that starts, does work, records what it did, and exits. State lives in a file (a DuckDB database) by default, not a running service. This is a default, not a restriction: an opt-in DuckLake catalog upgrade (§8) is fully compatible with a centralized-metadata-store architecture when a team wants one — including one backed by a dedicated, long-running database (Postgres, self-hosted or managed) shared across several DuckPipe deployments for genuine multi-user/multi-tenant use — it’s just never the thing you have to stand up before running your first pipeline. The plain-file state can optionally be synced to durable remote storage (S3/GCS/Azure/etc.) via fsspec — download-before-run, upload-after-run — so DuckPipe stays correct even when the process itself runs in an ephemeral, container-per-invocation environment (its own laptop/CI runner, or nested inside another orchestrator’s worker — see §9). This is a sync pattern, not a live remote database: DuckDB’s own docs confirm its native database file format only supports read-only remote attach (ATTACH 's3://...' via httpfs), not read-write, so a local-scratch-copy-then-sync approach is the correct one, not a workaround.
The orchestrator core is data-blind. DuckDB is the orchestrator’s brain (its state store), never a requirement for what a task does. Keep two claims separate. First, the orchestrator’s own state (run history, task fingerprints, logs, lineage) always lives in a DuckDB file/catalog — that’s the core architectural bet (§4) and isn’t optional. Second, and this is the tighter, more important rule: a task’s signature is Callable[..., Any], and the core never inspects, converts, or reasons about what a task returns. Passing a task’s output to the next task is a plain Python function call — whatever the object is (a DuckDB relation, a Polars LazyFrame, a pandas DataFrame, a string, None) crosses zero boundaries and costs the orchestrator nothing, because there’s no serialization to do. This is what makes “efficiency is a secondary bonus, not a core feature” literally true: DuckPipe adds no computational overhead on top of your data processing, because it never touches your data. Where a real boundary does exist — caching a result to disk between runs, or shipping a result to another process — that’s handled by an explicitly optional, separately documented layer (§6), never by the core.
Never introduce a parallelism or tuning primitive that duplicates one an engine or Python already has. Direct response to two related pain points researched early on: Prefect ships ConcurrentTaskRunner/DaskTaskRunner/RayTaskRunner and steers users toward wrapping pandas in Dask inside tasks, adding a second parallelism model on top of a data engine that already parallelizes internally; and no orchestrator found offers a unified resource-tuning abstraction across engines, because DuckDB’s knobs (threads, memory_limit), Polars’ (thread env vars, collect(engine=...)), and Daft’s (runtime config) are genuinely different APIs — inventing one abstraction over all three would be exactly this kind of duplication, just relocated. DuckPipe’s rule: DAG-level concurrency (running independent tasks concurrently) is the orchestrator’s job, using a plain asyncio/thread pool invisibly. Everything about a task’s internal parallelism or resource tuning — DuckDB’s threads/memory_limit, Polars’ thread count, a concurrent.futures pool, Ray/Dask for a genuinely distributed step — is the user’s own code, using whichever engine’s own native controls, optionally aided by small standalone utility functions (§6) the user calls themselves. The core has no opinion and no wrapper for any of it. The same principle extends to composing tasks across files: no DuckPipe-specific module/package concept, just ordinary Python imports (§8).
Zero-to-running in one file, one command. duckpipe run pipeline.py works with no config file, no duckpipe init, no database migration step. A duckpipe.db file appears next to the script holding run history; that’s the entire “infrastructure.”
Local dev and production are the same code path. No “local executor vs. Celery executor vs. Kubernetes executor” branching that changes behavior. Scaling out (§8) changes where a task runs, never how you write a task.
Incremental by default, not by manual flag. Borrowing from SQLMesh: every task gets a content fingerprint (hash of its code + config + resolved upstream fingerprints — never the output data itself, which is what lets fingerprinting stay data-blind per tenet #2). If nothing changed, DuckPipe skips it. No --select state:modified+ incantation required — this is on by default, with an explicit --force escape hatch.
Small, boring surface area. The full list of concepts a new user needs to learn should fit on one page (README.md’s own “whole mental model” section). A DAG is inferred from plain Python function signatures, not a parallel YAML/config DSL — see §4 for why this is a deliberate, defended choice rather than a shortcut. Any proposed new concept (a second decorator, a “profile” object, a plugin system) gets evaluated against “does this let us delete something instead,” per the exact failure mode Prefect fell into over three major versions (§10).
A hosted UI/control plane, alerting/SLA infrastructure, and dynamic-mapping-over-huge-fan-out (thousands of parallel tasks) are all out of the core — but “out of scope” means solved by composition, not left unsolved. A hosted UI/alerting layer is a SELECT away rather than a missing feature, precisely because tenet #1 keeps state in a plain, queryable DuckDB file (or a DuckLake catalog, §8, for real snapshot history) instead of hiding it behind a service boundary. Genuine massive fan-out is exactly the thing Airflow/Dagster/Prefect are already good at, so DuckPipe leans on that instead of reimplementing it: embed a DuckPipe pipeline as the unit a heavier orchestrator’s own dynamic-mapping feature drives, one per partition — §9 covers this directly, including why it’s a genuinely good idea and not just a curiosity. DuckPipe’s stance: be excellent at the case where standing infrastructure isn’t required at all, and extend outward via the DuckLake/distributed-execution path (§8) or by embedding inside a heavier platform (§9), rather than trying to be a general distributed-systems platform itself.
Also explicitly out of the core (though available as opt-in companion utilities, §6): any engine-specific resource tuning (DuckDB PRAGMAs, Polars thread config, Daft runtime settings), and any built-in “efficient data handoff” machinery beyond a plain function call. Both were drafted as core subsystems in earlier design iterations and were deliberately demoted — see §6 for why baking them into the core would have reintroduced the exact multi-engine complexity DuckPipe exists to avoid.
Prior art looked at before committing to this scope:
.sql files, it infers dependencies and runs them via DuckDB. No Python task nodes, no scheduling, single-author/early-stage. Validates the appetite for this category; doesn’t compete with the scope described here (fully generic Python tasks, DuckDB-backed state/fingerprinting, distributed execution).This means the core wedge — DuckDB file/catalog as the orchestrator’s brain, coordinating tasks that are free to use whatever data-processing engine fits the job, or none at all — is real white space, not a reinvention.
Why signature-based DAG inference over explicit YAML/config. This is a legitimate question (Dagster/Prefect both use Python-native inference successfully, but it’s fair to ask how it scales) and worth answering explicitly. The usual scaling objection to implicit Python DAGs is actually an objection to one specific thing: a persistent scheduler that has to keep re-parsing/re-importing pipeline code to know the graph — this was Airflow’s own documented bottleneck (§10, first row). DuckPipe has no persistent scheduler; duckpipe run imports the module exactly once per invocation, so that cost doesn’t scale with DAG size or exist between runs at all. The other legitimate case for explicit config — non-Python-fluent stakeholders authoring pipelines, or needing to inspect DAG topology without executing code — doesn’t match DuckPipe’s target user (Python-fluent data/ML engineers writing their own tasks), and introducing a YAML DSL to serve it would recreate the exact “second language for control flow” complaint that pushes Airflow users toward Jinja-templated YAML in the first place. For large fan-out (hundreds of per-partition tasks), the answer is a plain Python loop generating uniquely-identified task instances — the same mechanism Dagster’s dynamic outputs and Prefect’s .map() already use — documented as a first-class pattern rather than left as a gap.
README.md documents the actual “whole mental model” a user needs. The commitment behind it, worth stating separately: that list is deliberately short, and if it grows without a very good reason, that’s a design smell (tenet #7). No @flow wrapper concept separate from the DAG — the DAG is the set of tasks plus their declared dependencies. No work pools, no workers, no blocks, no deployments-as-a-separate-entity, no task-runner selection menu, and — per tenet #2 — no resource-profile object either; that’s the fully optional layer below (§6). Changing where code runs (§8) is a runtime setting, never a new concept.
memory_limit_mb, and why it clears tenet #7’s bar. @task(memory_limit_mb=N) is a new Task parameter, not a new concept: same shape as retries/cache/depends_on already are, opt-in (default None, meaning zero behavior change and zero import cost for anyone not using it — enforced by keeping cloudpickle/psutil lazy imports behind the duckpipe[memcap] extra, never touched by the core), and it doesn’t introduce a resource-profile object or an engine-specific abstraction (tenet #2/#3 stay intact — it caps physical RSS, the one thing every engine’s process shares regardless of which one a task happens to use). What earns it a place in the core rather than living beside the opt-in §6 utilities: it isn’t engine-specific tuning advice (§6’s whole reason for existing outside the core) but a scheduler-level guarantee — turning an OOM into a recorded status="oom" result requires the scheduler to own the subprocess boundary and the state-file write, the same way retries and fingerprinting already do. Proven out first in a real, external dogfooding project (a multi-engine benchmark harness enforcing the identical cap on every engine under test) before being generalized into the core, not designed speculatively.
to_mermaid’s subgraphs parameter, and why it stays a parameter, not a new mechanism. DuckPipe’s own DAG discovery has no way to know a task’s body happens to run another pipeline (that would mean statically analyzing arbitrary Python — the exact kind of complexity tenet #7 exists to refuse), so it can’t render that nesting automatically. What it can do safely (previous note) is actually run that nesting; subgraphs just lets the one party who does know the relationship — the pipeline’s own author — say so explicitly when calling to_mermaid, the same “call it yourself, nothing automatic” shape duckpipe-tuning already uses. Mermaid’s own subgraph...end block is the existing primitive for “a bordered box containing its own mini-flowchart, itself a node in the outer graph” — reused directly rather than inventing a DuckPipe-specific diagram notation. No new concept in the task-authoring surface at all: it’s a rendering-time argument, not something a pipeline’s own code ever needs to touch.
Recurses to any depth for the same reason: a real pipeline that nests another pipeline whose own last task nests a third is exactly as safe as nesting once (next note), so the rendering shouldn’t artificially stop at one level either. The fix wasn’t a new mechanism, just removing one: each subgraphs entry’s tuple takes an optional third element, that sub-pipeline’s own subgraphs argument, and the recursive render passes it through instead of a hardcoded empty dict. examples/09_nested_pipeline demonstrates three genuine levels (not a synthetic stress test) — and building it caught a second, unrelated bug worth naming honestly: report_card/report_cash communicate their payment type to their own nested call via a process-wide environment variable, and running them concurrently (DuckPipe’s default for independent tasks) races on it once there’s enough work in each nested call for the interleaving to actually bite. max_workers=1 isn’t cosmetic there — see the example’s own README for how that was confirmed, not assumed.
Earlier design drafts put resource auto-tuning and a tiered lazy/eager/Arrow I/O model directly into the orchestrator core. On reflection that was a mistake worth naming explicitly: both required the core to understand engine-specific APIs (DuckDB PRAGMAs vs. Polars thread config vs. Daft runtime settings; DuckDB relations vs. Polars LazyFrames vs. Daft DataFrames), which is exactly the kind of cross-engine complexity multiplication tenet #3 exists to prevent — just relocated from “parallelism primitives” into “tuning and I/O primitives.” The fix: keep this valuable, well-researched design work, but move all of it out of the core and into small, independent, opt-in pieces reached for only if wanted. None of what follows is imported or invoked by duckpipe run unless the user’s own task code calls it.
6.1 — Tuning helpers, as plain functions, not a core object. No ResourceBudget class, no auto-detection wired into the scheduler. Instead, duckpipe-tuning (a fully separate optional package, so the core dependency tree doesn’t even import psutil) offers pure functions like suggest_duckdb_settings()/suggest_thread_count()/suggest_temp_dir_limit(), built from the verified DuckDB internals in §7. A user who wants tuned behavior calls one of these themselves, inside their own task, and applies it to their own connection. A documented pattern, not a framework feature.
6.2 — Cross-run/cross-process data handoff, as an opt-in caching backend. A tiered lazy-vs-eager-vs-Arrow design only matters once data actually needs to cross a real boundary — writing a cache=True task’s output to disk so a later run can reuse it without re-executing upstream, or shipping a result to another process/machine (§8). cache=True defaults to the simplest thing that works for literally any Python object — pickling — with zero engine awareness, satisfying tenet #2 completely. The Arrow-PyCapsule-based fast path (DuckDB/Polars/pandas/pyarrow all implement __arrow_c_stream__; Daft doesn’t yet) is an optional alternate cache backend (@task(cache=True, cache_backend="arrow")) for tasks moving large tabular data that want cheaper disk round-trips than pickle — never the default, never something the core needs to understand to function correctly.
6.3 — Docs, not framework code, carry the “lazy is the obvious choice” narrative. DuckPipe’s examples use duckdb.sql(...)/pl.scan_parquet(...)/lazy Daft frames because they’re genuinely the best default for most pipelines, not because the framework requires or specially recognizes them. Docs carry that narrative instead of a lint rule or adapter layer baked into @task.
6.4 — Where big-single-task distribution goes. If a task genuinely needs cluster-scale, multi-machine data processing within itself (not DAG-level distribution), that’s “use Daft’s Flotilla/Ray runner inside that task” — the user’s own code, the user’s own dependency, nothing DuckPipe needs to wrap. DuckPipe’s own distributed-execution story (§8) is about distributing task orchestration across ephemeral compute, not about distributing a single task’s data processing.
Not a core subsystem — the technical grounding for the opt-in duckpipe-tuning helpers in §6.1, kept here so the research underpinning them isn’t lost.
Threading. DuckDB auto-detects host cores for its threads setting; over-launching on SMT/hyperthreaded machines can hurt performance (default to physical cores, not logical, except for I/O/remote-scan-heavy workloads, where DuckDB’s own guidance suggests a 2–5× multiplier).
Memory. memory_limit defaults to ~80% of detected RAM and bounds DuckDB’s buffer manager; spillable operators are hash aggregations, sorts, and joins. Per-thread rules of thumb from DuckDB’s own docs: ~1–2GB/thread for aggregation-heavy workloads, ~3–4GB/thread for join-heavy ones.
Disk. max_temp_directory_size bounds spill-to-disk bytes (default ~90% of free space on the temp volume); some DuckDB versions have had temp-file cleanup edge cases worth a defensive cleanup helper if wanted (duckdb/duckdb#14142).
Concurrency safety. A single DuckDB connection isn’t thread-safe for concurrent use; per-thread cursor() off a shared connection is the documented-safe pattern. DuckDB releases the GIL during query execution, so it already parallelizes internally via threads — stacking many Python threads each issuing heavy DuckDB queries risks oversubscribing the same core pool rather than adding throughput. Advice for task authors, not a scheduler feature.
Streaming accessors. DuckDB’s current (non-deprecated) streaming Arrow accessor is to_arrow_reader(), returning a real pyarrow.RecordBatchReader. Polars’ streaming engine (collect(engine="streaming")/.sink_*()) is recommended but still opt-in, not default. Daft is lazy by default with two runners — Swordfish (single-machine streaming) and Flotilla (Ray-distributed) — see §6.4.
The primitive everything below runs on: object-storage-native, put-if-absent conditional writes (duckpipe.remote.locked()), no server or catalog database involved (§12 has the full writeup of why this beat both a persistent Quack server and a full DuckLake catalog for the underlying concurrent-state-file problem).
Task-scoped execution — one new concept, and it’s a narrowing of an existing one, not an addition beside it: duckpipe.run(module, only=task_name) (--only on the CLI) runs exactly one task instead of the whole DAG. Locally this is trivial — build the DAG, execute one node instead of all of them, same file, same OS-level lock as always. Against a state_uri, it’s what turns a single shared state file into something many nodes can contribute to at once, without ever generalizing locked() into something heavier:
duckpipe.db that happens to hold only a few rows, because that’s the shape the existing task_runs/task_fingerprints/task_cache schema already has for “what one task run adds.” Absorbing one is ATTACH plus a few INSERT ... SELECT, using tables that already exist.duckpipe run pipeline.py --only <task> against the same state_uri at the same time, on different tasks — or even the same task redundantly, since fingerprint-based skip already makes redoing finished work harmless rather than corrupting. That’s a real multi-node cluster run, no DuckLake or Quack required.duckpipe show already resolves topological order and previews skip-vs-run per task; that’s the discovery primitive a coordinator (a Step Functions state machine, a Modal .map(), a shell loop over SSH) needs, and each dispatched unit is the same command any trigger already calls (§9), just narrower in scope. See it actually running one in examples/04_distributed_cluster.DuckLake-backed state — an observability upgrade, not a coordination one. db_path="ducklake:sqlite:pipeline.ducklake.sqlite" — the same argument a plain file goes in, pointed at a different kind of string; duckpipe run/show/stats don’t change at all. What it actually buys: every task’s outcome becomes its own DuckLake snapshot, tagged with a plain-English commit message, so task_runs AT (VERSION => n) turns “what happened” into a real, queryable history instead of a present-tense table — and schema evolution (ALTER TABLE ... ADD COLUMN) needs no migration step. See examples/06_ducklake_observability.
The same catalog string can instead be backed by a dedicated, long-running metadata database — db_path="ducklake:postgres:dbname=... host=..." (or MySQL), with data_path passed explicitly since there’s no local file to derive a sibling directory from. Nothing about duckpipe.run(...) changes; state.py’s catalog-extension install and DATA_PATH handling are already generic across sqlite/duckdb/postgres/mysql. What it buys over the SQLite catalog: several DuckPipe deployments — different teams, tenants, or machines — sharing one catalog with genuine concurrent-write support. Verified directly, not assumed: 8 concurrent commits against a real Postgres catalog (no retry logic) all succeeded, where the identical test against a SQLite catalog failed 3 of 8 outright (see examples/05_distributed_with_ducklake’s SQLite-vs-Postgres writeup). Entirely optional — the SQLite catalog needs no such infrastructure and stays the right default for a single team’s own history.
Deliberately not wired to state_uri/only= regardless of which catalog backs it — both raise clearly if combined with a ducklake: db_path — three independent, verified reasons why, not a hunch: (1) a SQLite catalog can’t live on an object store (DuckLake’s own maintainers confirm this), so it would still need the exact sync dance this backend exists to not need; (2) a Postgres/MySQL catalog removes that limitation but is provisioned infrastructure, which is why it stays opt-in rather than becoming the distributed mechanism’s default; (3) even against Postgres, DuckLake’s optimistic concurrency control is retry-based (documented real failures under load in duckdb/ducklake#233, since fixed) — strictly weaker than the delta-merge mechanism’s conflict-free-by-construction guarantee for “many small workers each report one fact.” examples/05_distributed_with_ducklake demonstrates that narrower coordination use directly, on its own terms, separate from DuckLake’s actual wiring into duckpipe.run()’s own state store described above.
Reference serverless executor. A task becomes a small function on Lambda/Modal/a container/etc.; each invocation is duckpipe.run(module, only=task, state_uri=...) — already a plain Python function with no platform-specific glue, so “pick a reference platform” would mostly produce code whose only job is proving something already structurally true. examples/07_serverless_executor dispatches one DAG’s two tasks through two genuinely different invocation shapes into the same distributed run — a container (docker run, argv-invoked, the way ECS/Cloud Run/a Kubernetes Job calls one) and a handler(event, context) function (the calling convention Lambda/Modal/most FaaS platforms actually use) — making the “not locked to one platform” claim checkable instead of asserted. Quack doesn’t fit this path at all — it requires a persistent DuckDB server, a direct contradiction of “stateless executor” — so it’s not part of DuckPipe’s own sync layer; it remains available for a coordinator to optionally run for one distributed run’s lifetime, never as DuckPipe’s default.
Remote “beefy node” mode. One job too large for a laptop but not worth distributing: the same code runs unchanged on a bigger remote machine, optionally paired with the §6.1 tuning helpers. Needs no coordination model at all — no state_uri locking, no delta files, no dispatch coordinator — just rsync the pipeline over, ssh in, run the exact same command. See docs/remote_execution.md.
Browser execution (via Pyodide). Verified directly with DuckPipe’s actual source loaded into a live Pyodide runtime, first via a Node.js harness for the initial spike and then via an actual headless browser (Playwright + Chromium) for the shipped example — not assumed from DuckDB’s own WASM marketing. The full duckdb Python package (real SQL engine, not a stub) loads via pyodide.loadPackage("duckdb") straight from Pyodide’s own package repository. DuckPipe’s core — task.py, dag.py, fingerprint.py, remote.py, state.py, scheduler.py — imports and runs a real multi-task pipeline through scheduler.run() inside that runtime with zero code changes, asyncio.run() included. The one real gotcha: asyncio.run() inside Pyodide needs the host to invoke Python via the promise-based entry point (runPythonAsync) and needs WASM JSPI (stack-switching) support in the JS engine — standardized (W3C Wasm CG Phase 4) as of this writing, shipping in Chrome 137+ stable with no flag, Firefox 139 behind a flag, Safari committed. examples/08_browser_wasm (“is this file safe to send anywhere?”) is the shipped deliverable, built around a concrete niche rather than a generic demo — see its own README for the full writeup, including two things checked rather than claimed (zero network requests for a user-picked file; state genuinely surviving a real page reload via IndexedDB).
Honest, scoped-not-glossed-over limitation: no state_uri remote sync and no DuckLake backend in-browser — Pyodide’s DuckDB build has no runtime-loaded extensions (httpfs/ducklake unavailable), it’s single-threaded (a Pyodide constraint on DuckDB’s own internal parallelism, not on DuckPipe’s asyncio-based task concurrency), and whether fsspec works against Pyodide’s virtual filesystem/CORS-constrained fetch model is genuinely unverified — the next open spike, not a claimed result.
Nothing here changes DuckPipe’s task-authoring API (§5) — a task function written for local mode is the same function deployed to a Lambda executor or run inside a browser tab, just invoked with a narrower scope or a different host.
A deliberately small addition, worth stating explicitly rather than leaving implicit: DuckPipe should always be embeddable as a single task/step inside Airflow, Prefect, Dagster, or anything else, without any special integration code on DuckPipe’s side. This isn’t a new subsystem — it’s a free consequence of tenets #1 (no persistent daemon) and #5 (local dev and production are the same code path): a DuckPipe pipeline is just duckpipe.run(module), an importable Python call that starts, does work, and exits, indistinguishable from any other function a PythonOperator, a Prefect @task, or a Dagster @op might wrap. The only discipline required is to never add something that assumes DuckPipe owns the top-level process (a global signal handler, an entrypoint with no clean importable form) — that’s a constraint to preserve, not a feature to build.
Where this is a genuinely good idea, not just a curiosity:
Where to deliberately not go, because it’s the exact scope creep this whole design exists to avoid: no DuckPipe-side awareness of the host orchestrator. No Airflow provider package with custom operators/sensors/hooks, no DuckPipe retries deferring to the host’s retry semantics, no fingerprints flowing through XCom. From the host’s point of view, duckpipe.run(module) stays an opaque, atomic unit — success, failure, duration, nothing more. The entire “integration” is a five-line documented recipe per platform (docs/interop.md), not a maintained package.
Also worth stating plainly: nesting DuckPipe inside a heavier platform is a decision driven by real governance/observability needs the outer platform provides (existing alerting, RBAC, multi-team visibility) — not something to do reflexively just because Airflow happens to already be installed. A pipeline that doesn’t need any of that is still better served by plain cron/CI directly invoking DuckPipe (tenet #1), which stays lighter for that case.
| Researched pain point | Source | DuckPipe response |
|---|---|---|
| Airflow: scheduler re-parses every DAG file on a fixed loop; degrades at scale | GitHub discussion #44727, cutting DAG parse time from 60s to ms | No persistent scheduler/parser process at all; “trigger” is just invoking the run command |
| Airflow (FLYR postmortem): CeleryExecutor requires pre-allocated workers; wastes spend; KubernetesExecutor caused unexplained task failures at high parallelism | FLYR Labs blog: “Why We’re Switching Off Airflow — Sort Of” | No broker/worker-pool concept; scaling is “bigger single node” or stateless serverless execution (§8), not a worker fleet to keep healthy |
| Airflow: XCom size limits, awkward inter-task data passing | Airflow XCom docs | In-process task-to-task handoff is a plain function call (§2, tenet #2) — no serialization, no size cap, no side-channel |
| Prefect: Flow→Deployment→Flow Run hierarchy confuses users; not resolved in onboarding (issue closed “not planned”) | GitHub #10811 | No separate “deployment” entity — a pipeline module is the runnable unit |
| Prefect: work pools/workers/blocks/task-runner menu — “which one do I need” is a recurring question | Prefect docs: work pools, linen.prefect.io threads | No work pools/workers/blocks; one concurrency model in the core, with any engine-specific tuning left to the user’s own code or the optional §6 utilities |
Prefect: users reach for wrapping pandas in DaskTaskRunner instead of using the data engine’s own parallelism |
GitHub Discussion #3022 | Explicit tenet (#3): the core never ships a competing parallelism or tuning primitive; an engine’s own controls (or the opt-in helpers in §6) are the answer |
| Prefect: 1→2→3 breaking rewrites; migration pain even in typed codebases | Prefect migration guide, issue #15275 (2→3, mypy breakage) | Small, boring core API (§5) deliberately resists growth; task-authoring surface is the one thing committed to keeping stable |
| Dagster: assets-vs-ops split-brain — two overlapping graph paradigms | Discussion #10512 | One task/dependency model, no parallel paradigm |
| Dagster: “I/O managers aren’t required” but a default one is silently initialized anyway — contradicts stated minimalism | Issue #32065 | The core has no IO-manager concept to be silently mandatory about — data never passes through the framework at all (§2, §6) |
| No tool auto-tunes DuckDB resource settings to host specs | Own DuckDB internals research — confirmed gap | Filled by the opt-in §6.1 utilities, deliberately kept outside the core so it never becomes a second engine-abstraction layer |
| No tool re-runs only what changed, by default, at the task level (dbt/SQLMesh do this for SQL models only) | SQLMesh docs, dbt --state docs |
Fingerprint-based incrementality is default behavior for any task, not just SQL models |
| No lightweight orchestrator lets a single task’s physical memory be capped with the breach recorded as a first-class task outcome, rather than a crash, a silent OS OOM-kill, or standing up a whole resource-manager subsystem | Own dogfooding — a real multi-engine benchmark harness needed exactly this and built it standalone before DuckPipe had an equivalent (confirmed gap, not researched against a specific competitor issue) | @task(memory_limit_mb=N) (§5) — opt-in, scheduler-level, status="oom" |
A few build-time findings worth keeping, grouped by subsystem rather than by when they happened:
Fingerprinting. An edge is a parameter whose default value is another Task object (def b(x=a)) — unambiguous, no type-hint parsing, structurally excludes *args/**kwargs (Python forbids defaults there). Confirmed with a toy 4-task diamond DAG (tests/fixtures/toy_dag.py): cache=True everywhere skips entirely on an unchanged re-run and fully re-executes under --force. Concurrent-cursor fan-out (several DuckDB queries sharing one connection via cursor()) measured a real 2.2x speedup over serial execution on one benchmark machine (scripts/phase0_bench_fanout.py) — the basis for using a plain asyncio + thread-pool scheduler rather than pulling in a third-party graph-execution library.
Nesting duckpipe.run() inside a task is safe, confirmed empirically, not just reasoned about. A task’s synchronous body always runs via loop.run_in_executor(None, ...) (a thread-pool worker thread with no event loop of its own) rather than directly on the scheduler’s own event loop — so a task that itself calls duckpipe.run(other_pipeline) (which internally calls asyncio.run() again) never collides with the outer scheduler’s loop: asyncio.run() only raises when called from a thread that already has one running, and the executor thread doesn’t. Verified directly with a real two-level pipeline (an outer task nesting a full inner run() on a separate module, its own state file) before relying on it. This is what makes to_mermaid’s subgraphs parameter (§5) meaningful: rendering a task as containing another real pipeline’s shape reflects something DuckPipe actually supports running, not just drawing.
Resume and observability. Partial-DAG resume needed no new API: a failed task never gets a fingerprint/cache entry, so re-running the same command naturally re-executes only the failed task and its downstream, while unaffected cache=True upstream tasks skip via the existing incrementality mechanism. What genuinely needed building: a task cascade-skipped because its upstream failed gets its own task_runs row (status upstream_failed) instead of vanishing from the state file — and a skip-if-unchanged cache hit needs its own row too — otherwise resume and duckpipe stats would silently undercount.
Concurrent state-file access (§9’s lock). Two heavier alternatives were weighed and ruled out for the “many workers touching one remote-synced state file” problem: Quack (DuckDB’s client-server protocol) needs a persistent DuckDB server process, reintroducing exactly the standing infrastructure tenet #1 refuses — at most a fit for a coordinator process in the serverless-executor path (§8). DuckLake gives genuine multi-writer ACID, but only once its catalog is itself a reachable, always-on database service; a SQLite-file catalog doesn’t help here because that catalog file would need the identical download-mutate-upload round-trip this problem is about, one level down. What shipped instead: an advisory lock object next to the state file, using each object-store backend’s own native conditional-write primitive (S3 If-None-Match, GCS if_generation_match, Azure ETag preconditions) through fsspec’s exclusive-create ("x") file mode — no server, no catalog database, held for the whole download-run-upload sequence, wired into duckpipe.run(..., state_uri=...) by default. A stale lock (crashed holder) is reclaimed after max_lock_age rather than deadlocking forever. This same put-if-absent primitive turned out to be the right foundation for task-scoped concurrency too (§8) — that mostly reuses it at a finer grain rather than needing something new.
DuckLake schema portability. Dropping PRIMARY KEY/ON CONFLICT from the whole schema (DuckLake supports neither at all, verified directly) in favor of one portable DELETE-then-INSERT/INSERT ... WHERE NOT EXISTS idiom turned out to be a simplification, not a workaround — both backends now share one dialect. CREATE OR REPLACE VIEW was found to commit a new DuckLake snapshot on every single store open even when byte-identical, which would have flooded the exact run-history story that backend exists to make useful; fixed with CREATE VIEW IF NOT EXISTS.
Packaging. duckpipe-tuning and the Arrow-aware cache_backend="arrow" option shipped as genuinely optional: packages/duckpipe-tuning/ is a real second distribution in the uv workspace, zero psutil/DuckDB coupling in duckpipe itself (enforced by a real test, not just a docstring promise); cache_backend="arrow" lives behind a lazy pyarrow import and the duckpipe[arrow] extra. Daft’s own Arrow-PyCapsule support (Eventual-Inc/Daft#2504) is still pending upstream as of this writing — a documented known gap, not silently papered over.
memory_limit_mb (§5). Two real bugs surfaced only by testing the actual watchdog rather than reasoning about it: (1) only=-scoped execution (_execute_one, Phase 3a) is a genuinely separate code path from the whole-DAG one (_execute_dag/run_node) — both needed their own isinstance(exc, MemoryLimitExceeded) check to record status="oom" instead of a generic "failed", and it’s easy to fix one and miss the other. (2) The SQL views (v_task_stats/v_run_summary) hardcode their failure-status list (status IN ('failed', 'upstream_failed')) for failed_count — a new status value silently undercounts unless that list is updated too, precisely the class of bug this project already went out of its way to avoid for upstream_failed itself (see the “Resume and observability” note above). Separately, confirmed directly: cloudpickle prefers pickling a function by reference (re-import by dotted module name) whenever the module looks normally importable, and gets this specifically wrong for a function defined in a Jupyter cell or a pytest test module — neither is reliably re-importable from a fresh subprocess’s own sys.path, though cloudpickle can’t tell that from outside. A task defined in an actual pipeline file (the normal case) doesn’t hit this.
max_workers, two real bugs found by measuring actual scheduling behavior, not by reasoning about the semaphore. (1) A task’s concurrency slot was acquired before it checked whether its own upstream dependencies had finished, not after — so a task that’s merely waiting on a dependency could win a slot and idle in it for that dependency’s entire duration, leaving genuinely independent, ready-to-run tasks with no slot to run in even though nothing was actually contending for a resource. Confirmed directly, not assumed: a 4-task DAG (one slow root, one task depending on it, two independent tasks) ran slower under max_workers=2 than fully unbounded — 3.40s vs. 2.07s, a 64% regression from adding a limit that should have been more than sufficient — because the dependent task’s name happened to sort right after its own dependency in topological_order’s alphabetical tie-break, so it consistently won a slot ahead of the independent tasks and sat in it, idle, doing nothing. Fixed by moving the dependency-wait outside the semaphore-guarded region — no new concept, just an await moved from inside run_node to before async with sem in run_guarded. (2) max_workers=None (“unbounded”) doesn’t mean unbounded: task bodies dispatch via loop.run_in_executor(None, ...), and None there means Python’s own lazily-created default ThreadPoolExecutor, capped at min(32, cpu_count + 4) — 14 on a typical 10-core machine. A wide fan-out (§4’s own documented pattern) silently queues behind that cap regardless of what the DAG’s own shape could support. Confirmed directly: 50 independent 0.5s tasks under max_workers=None took 2.84s, not ~0.5s. Fixed with an explicit ThreadPoolExecutor sized to max_workers when bounded, or to the DAG’s own task count when not — the real ceiling on how many could ever be in flight at once, so “unbounded” means what it says without introducing an arbitrary number of its own.
A correction to the above, from digging further. The “~30ms/task” figure first measured wasn’t actually a per-task cost at all — isolating it precisely (a 1-task run vs. a 50-task run in the same process) showed the 1-task run alone cost ~990ms, and the other 49 tasks added only ~82ms combined. The real marginal cost per task, measured directly against StateStore (lineage + task-run + fingerprint writes, one transaction) with the one-time cost warmed out of the way first, is ~2ms — genuinely fine, not a scaling problem for “organizing small things” inside one run() call.
The ~800ms–1s isn’t DuckPipe’s own code at all: it’s DuckDB’s Python client’s first-ever parameterized write on a connection, confirmed by isolating exactly that statement (subsequent inserts on the same connection, any table, any shape, any parameter count, all ≥1000x faster) and confirmed as a real, currently-unresolved, “wontfix”-tagged DuckDB characteristic (duckdb/duckdb-python#238, duckdb/duckdb#20824) — not something pragma-tunable (threads=1 vs 8 made no reliable difference across isolated processes) or fixable from outside the engine. DuckPipe already, incidentally, pays this cost in the right place: start_run()’s own insert runs before any task’s timer starts, confirmed directly (a task’s own first write, run right after, costs ~0.6ms) — so it isn’t silently inflating a task’s recorded duration in task_runs.
Where this genuinely matters: not a single run() call (the cost is fixed and one-time, paid once regardless of task count), but the only=-per-subprocess dispatch pattern (examples 04/07/10) — every dispatched subprocess is a fresh DuckDB connection, so every one repays this ~0.5–0.8s floor. That’s a real, inherent cost of choosing process-per-task isolation, best addressed by knowing it’s there (favor this pattern for tasks whose own work clearly exceeds the floor) rather than by trying to engineer it away.
Still open:
fsspec in Pyodide. Whether it works against Pyodide’s virtual filesystem/CORS-constrained fetch model is genuinely unverified — needed before state_uri sync (or the DuckLake backend) means anything in-browser (§8).Decided, and why it’s not revisited:
Task object — no Depends(...) marker needed, no type-hint parsing, no ambiguity.@task(extra_fingerprint=[...]) is the opt-in escape hatch. A footgun worth naming explicitly: if what you pass changes as a side effect of the task’s own run (a notebook whose outputs get rewritten by executing it, say), the cache invalidates itself on every single run — key it on something the run doesn’t itself mutate (a source file’s own content, not its full byte-for-byte state).LARGE_CACHE_WARN_BYTES) fires above 50MB; an object that can’t be pickled at all (a live DuckDBPyRelation, a raw DB handle) degrades to a logged warning rather than failing the run. cache_backend="arrow" (§6.2) is the opt-in alternative for large tabular results.duckpipe-tuning helpers. Every function takes only host specs (CPU count, total RAM, free disk) and returns a suggestion — none accept a DuckDB connection, run a query, or inspect data. Enforced by a real test, not just a docstring promise.duckpipe show --mermaid stays flat; nesting isn’t a CLI feature. Considered three shapes for wiring to_mermaid’s subgraphs (§5) into the CLI: a new @task(nested_pipeline=...) parameter (rejected — contradicts §5’s own “rendering-time argument, never a task-authoring concept”); a magic module-level name show looks for, e.g. a SUBGRAPHS variable or a duckpipe_subgraphs() function (rejected — every existing discovery mechanism finds things by type (isinstance(value, Task), including inside lists/dicts), never by a specific variable/function name; this would be a new, precedent-setting kind of magic); a repeatable --subgraph name=path flag (rejected on cost/benefit — real argument parsing and per-sub-pipeline state-file resolution, reimplementing in cli.py what a ~15-line script already does directly against the public to_mermaid/build_dag API, for a need that’s opt-in and rare). examples/09_nested_pipeline/show_nested_mermaid.py is the answer instead — the same “call it yourself” shape duckpipe-tuning is already built around, not a workaround standing in for a missing feature.max_workers= is set — deliberate, since most pipelines’ tasks are cheap/IO-bound relative to the whole run and benefit from running together. “No limit” means an executor genuinely sized to the DAG’s own task count, not Python’s own much smaller default thread-pool size (§11) — the distinction used to be invisible, and isn’t anymore. The footgun this creates: tasks that do compete for the same physical resource (several benchmark-style measurements on one box, say) need max_workers=1 (or however many the resource can actually afford) stated explicitly — DuckPipe has no way to infer “these tasks shouldn’t overlap” from the DAG shape alone, and shouldn’t guess.(For implementer reference; sources are live links, verified August 2026. The DuckDB/Polars/Daft/Arrow-specific facts below ground the optional §6 utilities, not the core; the fsspec/DuckDB-remote-attach facts ground §2/§9’s state-persistence design.)
threads; morsel-driven parallelism needs enough row-group-sized morsels to use all threads; SMT/hyperthreaded over-launch can hurt performance. DuckDB: Tuning Workloadsmemory_limit defaults to ~80% of detected RAM; spillable ops are hash aggregations, sorts, and joins; too-low a limit on a blocking operator can still throw OutOfMemoryException. DuckDB: Memory Management, OOM Errors guidemax_temp_directory_size bounds spill-to-disk bytes; some versions have had temp-file cleanup edge cases. DuckDB Limits, duckdb/duckdb#14142DuckDBPyRelation (from .sql(), .table(), .read_parquet(), etc.) is lazy and composes freely before execution. Its streaming Arrow accessor is now to_arrow_reader() — the older fetch_arrow_reader()/fetch_record_batch() names are deprecated as of 2026; to_arrow_reader() returns a real pyarrow.RecordBatchReader that streams batch-by-batch rather than materializing. DuckDB recommends partitioned Parquet read/write for larger-than-RAM workloads rather than full materialization. DuckDB Quacks Arrow, DuckDB Arrow export docsLazyFrame (.scan_parquet(), .scan_csv(), etc.) is lazy the same way. Its rewritten streaming engine (3–7x faster than the in-memory engine on larger data) is the recommended engine as of late 2025 but is still opt-in, not default — invoke via lf.collect(engine="streaming") or .sink_parquet()/other .sink_*() methods for a fully streaming write. Polars in Aggregate (Dec 2025), Polars streaming guideDataFrame is lazy by default (builds a logical plan; executes on .collect()/.show()/.write_parquet()). Two runners: Swordfish (single-machine, Rust, streaming/out-of-core via async channels) and Flotilla (distributed, each Ray worker running Swordfish over its partition) — the latter is the natural answer when a single task genuinely needs multi-machine data processing (§6.4), rather than DuckPipe building its own distributed dataframe engine. Daft Architecture__arrow_c_stream__/__arrow_c_array__) is the standardized zero-copy interchange mechanism for eager objects; confirmed implemented by pyarrow, pandas (2.x+), Polars, and DuckDB — the basis for the optional §6.2 cache_backend="arrow". Daft’s eager-output support for this interface is still pending (tracked in Eventual-Inc/Daft#2504 as of this research), so a Daft-produced eager result would need .to_arrow() + .to_batches() chunking instead. Narwhals + Arrow PyCapsule, Daft PyCapsule issue #2504cursor() off a shared connection is the documented-safe pattern; DuckDB releases the GIL during execution. DuckDB: Multiple Python Threads, duckdb-python discussion #40duckdb package, not just the JS duckdb-wasm library, is compiled to WebAssembly in its entirety and ships in Pyodide’s own package repository — confirmed by DuckDB’s own writeup and independently verified in-session by loading DuckPipe’s real source and running a full pipeline through it. Its Pyodide build has no runtime-loaded extensions and can’t reach remote files directly (pull them into Pyodide’s virtual filesystem first). DuckDB: DuckDB in Python in the Browser with Pyodideasyncio.run() inside Pyodide) block on an async JS operation without deadlocking the host event loop — required for DuckPipe’s scheduler.run() to work unmodified inside Pyodide. Standardized (W3C Wasm CG Phase 4) as of this writing; shipping in Chrome 137+ stable with no flag, behind a flag in Firefox 139, committed (engineer assigned) in Safari. V8: Introducing the WebAssembly JSPI API, Chrome for Developers: WebAssembly JSPI origin trialPRIMARY KEY/UNIQUE constraints at all (confirmed via NotImplementedException from the extension itself), so ON CONFLICT upserts are unusable against it. A SQLite (or DuckDB-file) catalog cannot live on object storage — a DuckLake maintainer states this is “a fondamental design decision, sqlite can not work on an object store” — so only a live network catalog (Postgres/MySQL, self-hosted or managed) gives a DuckLake-backed store genuine attach-directly-no-sync-dance behavior. duckdb/ducklake discussion #519, duckdb/ducklake issue #233 (concurrent writes)duckpipe.state’s catalog-extension install and DATA_PATH handling needed zero code changes to work against db_path="ducklake:postgres:..." — confirmed with StateStore itself, including its read_only=True path and the CLI’s stats --snapshots. The concurrency payoff is real, not just DuckLake’s own marketing: 8 concurrent, no-retry commits against a fresh Postgres catalog all succeeded, where the identical test against a fresh SQLite catalog failed 3 of 8 outright with TransactionException: ... database is locked (matching examples/05_distributed_with_ducklake’s own earlier, separately-verified finding).fsspec is the standard uniform Python filesystem abstraction (AbstractFileSystem, fsspec.open()), with s3fs/gcsfs/adlfs providing S3/GCS/Azure Blob support through the same interface local disk uses; it originated out of Dask specifically to avoid per-backend adapter code, and pandas/Dask/xarray all use it as their storage I/O layer. This is the basis for §2/§9’s state-persistence design. fsspec docs, fsspec backgroundATTACH (via httpfs, e.g. ATTACH 's3://bucket/file.duckdb') for read-only access only — its docs state explicitly that writing the database via HTTPS or the S3 API is not possible; read-write access is filesystem-local only. This confirms a download-before-run/upload-after-run sync pattern (not a live remote-attached database) is the correct, currently-only approach for a read-write state file on object storage. DuckDB: Attach over HTTPS or S3, duckdb/duckdb#10967