DuckLake in Python: The DuckDB Lakehouse Format (2026)

Updated: August 16, 2026

DuckLake is an open lakehouse format from DuckDB Labs that stores table metadata in a regular SQL database (Postgres, MySQL, SQLite, or DuckDB itself) and keeps the actual data as Parquet files in object storage. Version 1.0 shipped in April 2026 and is production-ready, with sorted tables, bucket partitioning, data inlining, geometry support, and Iceberg-compatible deletion vectors. From Python you interact with a DuckLake either through the DuckDB ducklake extension or the pyducklake client, and both hit the same catalog and Parquet files, so writes from one show up in the other.

• DuckLake 1.0 (April 2026) is a lakehouse format where the catalog lives in a normal SQL database and the data lives as Parquet on object storage.
• Metadata queries become indexed SQL lookups instead of parsing thousands of JSON manifest files like Iceberg or Delta Lake require.
• The Python entry points are the DuckDB ducklake extension for SQL work, pyducklake for a PyIceberg-style client API, and ducklake-dataframe for a pure-Python Polars/pandas/PySpark reader with no DuckDB dependency.
• Data inlining stores tiny appends (10 rows or fewer) directly in the catalog, solving the small-file problem for streaming and CDC workloads.
• DuckLake supports ACID transactions across multiple tables in a single commit, which is something Iceberg and Delta Lake do not offer natively.
• Choose DuckLake for operational simplicity on a team that already runs Postgres; stick with Iceberg or Delta if you need Spark, Flink, or Snowflake as first-class engines.

What is DuckLake and why does it exist?

DuckLake is a lakehouse table format built around a single observation: every serious deployment of Iceberg or Delta Lake ends up needing a database anyway. Iceberg pushes you toward an external catalog service (Glue, Nessie, Polaris, Lakekeeper), and Delta Lake pushes you toward Unity Catalog. Underneath, those catalogs are backed by relational databases that store table locations and pointers to the current metadata file. The manifest and snapshot data, meanwhile, still lives as thousands of small JSON and Avro files scattered across S3. Every commit rewrites JSON. Every query has to list, download, and parse manifest files before it can even plan the Parquet reads.

DuckLake collapses that split. All metadata (table definitions, snapshots, statistics, deletion vectors, partition boundaries) lives in a plain SQL database. The data files stay as Parquet on object storage, exactly like Iceberg or Delta. A commit is a normal SQL transaction against the catalog, so ACID semantics come from the database itself. Listing snapshots is SELECT ... FROM ducklake_snapshot, not a full manifest crawl. The DuckLake specification is deliberately small: a handful of tables in a well-defined schema, plus rules for how Parquet files are named and referenced.

The format ships as a DuckDB extension (INSTALL ducklake), but engine support has expanded fast. DataFusion, Spark, Trino, and Postgres (via pg_ducklake) can all read and write the same tables in 2026, and hosted DuckLake is available on MotherDuck. For Python users specifically, the two libraries you'll actually use day-to-day are pyducklake (a PyIceberg-style client) and ducklake-dataframe (a pure-Python Polars/pandas reader with no DuckDB runtime).

How DuckLake works: metadata in SQL, data in Parquet

A DuckLake deployment has three pieces: a catalog database (DuckDB, SQLite, PostgreSQL, or MySQL) that owns the metadata, a storage location (local filesystem, S3, Azure Blob, or GCS) that holds Parquet data files, and a compute engine (typically DuckDB) that reads and writes both. The DuckLake spec fixes the schema of the metadata tables (things like ducklake_table, ducklake_snapshot, ducklake_data_file, ducklake_delete_file), so any client that speaks the spec can interoperate with any other.

When you insert a row, the engine writes a Parquet file to the storage path, then INSERTs the file's path, size, row count, and column statistics into the catalog. The commit is a single database transaction that also bumps the snapshot number. If two writers race, standard database locking arbitrates the conflict; nobody has to invent optimistic concurrency on top of blob storage the way Iceberg and Delta must. That's the whole trick. The reason DuckLake advertises "multiplayer DuckDB" is that DuckDB by itself can't do concurrent writes to a single database file, but a DuckLake catalog can, because Postgres (or MySQL) already knows how.

Data files are ordinary Parquet, so anything that reads Parquet can bypass DuckLake entirely and read the files directly if it wants. That's how ducklake-dataframe works: it queries the catalog with plain SQL, gets back a file list plus statistics, prunes files with predicate pushdown, and hands the surviving paths to Polars or pandas. No DuckDB process required.

DuckLake vs Iceberg vs Delta Lake compared

All three formats give you ACID transactions, snapshots, time travel, schema evolution, and Parquet-based storage. What differs is where the metadata lives, what engines can read it, and how much operational machinery you need to run before you can query a table. The comparison below is aimed at Python teams choosing between the three in 2026, not at Spark shops with an existing Databricks investment, where Delta is already the answer.

Dimension	DuckLake 1.0	Apache Iceberg	Delta Lake
Metadata storage	SQL database (Postgres, MySQL, SQLite, DuckDB)	JSON + Avro manifest files on object storage, plus an external catalog	JSON transaction log on object storage, plus an external catalog
Commit mechanism	Database transaction	Atomic file rename with catalog pointer swap	Optimistic concurrency on the _delta_log directory
Multi-table transactions	Yes, native	No	No
Engine support	DuckDB, DataFusion, Spark, Trino, Postgres	Spark, Flink, Trino, Snowflake, DuckDB, ClickHouse, and more	Spark (first class), Trino, Flink, DuckDB, delta-rs
Python client	pyducklake, ducklake-dataframe	pyiceberg	deltalake (delta-rs)
Small-file mitigation	Data inlining stores <=10-row commits in the catalog	Compaction procedure	OPTIMIZE with bin-packing
Operational footprint	Reuse an existing SQL database	Catalog service (Nessie, Polaris, Lakekeeper, Glue)	Catalog service (Unity Catalog) recommended
Best fit	Small-to-mid teams that already run Postgres	Multi-engine analytics with Spark/Flink/Trino	Databricks or heavy Spark workloads

The trade is real in both directions. Iceberg's file-based metadata lets you spin up a Trino cluster in a fresh AWS account, point it at an S3 prefix, and query without provisioning a database. DuckLake requires you to have a database running. In return, DuckLake commits are cheap indexed SQL updates instead of manifest rewrites, and small-transaction throughput is dramatically higher. That's the kind of workload that historically pushed Iceberg users toward compaction cron jobs.

If you're already using PyIceberg on the same site, the two coexist fine. See the earlier walkthrough on PyIceberg for Apache Iceberg tables for how the Iceberg side looks, and Delta Lake with delta-rs for the Rust-backed Delta client. DuckLake occupies a distinct niche from both.

Getting started with DuckLake in Python

You have two Python doors into DuckLake, and they compose. The DuckDB extension gives you full SQL: DDL, DML, time travel syntax, EXPLAIN. The pyducklake package gives you an imperative object API that feels like PyIceberg. Most projects use the SQL extension for ad-hoc analysis and the client library for pipelines. Install both:

pip install duckdb pyducklake
# Optional integrations:
pip install "pyducklake[pandas]" "pyducklake[polars]"
# For local storage backends, the defaults are fine.
# For S3/Azure/GCS, install object-store extras as needed.

Honestly, the fastest way to try DuckLake is with a SQLite catalog and a local data directory. No cloud credentials required. Open a DuckDB session, install the extension, and attach:

import duckdb

con = duckdb.connect()
con.execute("INSTALL ducklake")
con.execute("LOAD ducklake")

# Metadata in catalog.sqlite, Parquet data in ./lake_data
con.execute("""
    ATTACH 'ducklake:sqlite:catalog.sqlite' AS lake
    (DATA_PATH './lake_data')
""")

con.execute("USE lake")
con.execute("""
    CREATE TABLE events (
        id BIGINT,
        user_id BIGINT,
        event_type VARCHAR,
        occurred_at TIMESTAMP
    )
""")

con.execute("""
    INSERT INTO events VALUES
        (1, 42, 'signup',   TIMESTAMP '2026-08-01 09:12:00'),
        (2, 42, 'purchase', TIMESTAMP '2026-08-03 14:05:00')
""")

print(con.sql("SELECT count(*) FROM events").fetchone())
# -> (2,)

That's a working lakehouse. catalog.sqlite now contains a handful of DuckLake metadata tables, and ./lake_data holds a Parquet file. Swap the SQLite URI for postgres:... or duckdb:... and the same code targets a shared multi-writer catalog. The official DuckLake documentation keeps the full ATTACH grammar current if you need MySQL or alternative auth flows.

Reading and writing tables with pyducklake

The DuckDB extension is fine for SQL, but pipeline code usually prefers a typed Python API. pyducklake gives you one that mirrors PyIceberg's shape closely, so anyone who has worked with Iceberg from Python will feel at home. You start by opening a catalog, then create or load a table.

from pyducklake import Catalog, Schema, required, optional
from pyducklake.types import IntegerType, StringType, TimestampType
import pyarrow as pa

catalog = Catalog(
    "analytics",
    "sqlite:catalog.sqlite",
    data_path="./lake_data",
)

schema = Schema.of(
    required("id", IntegerType()),
    required("user_id", IntegerType()),
    optional("event_type", StringType()),
    required("occurred_at", TimestampType()),
)

table = catalog.create_table("events_typed", schema)

batch = pa.table({
    "id":          [10, 11, 12],
    "user_id":     [7, 7, 8],
    "event_type":  ["view", "click", "view"],
    "occurred_at": pa.array(
        ["2026-08-15T10:00:00", "2026-08-15T10:01:00", "2026-08-15T10:02:00"],
        type=pa.timestamp("us"),
    ),
})
table.append(batch)

Reads use a lazy scan object. You compose filters and projections, then materialise into whichever dataframe library you want. Predicate strings are parsed against the table schema, so DuckLake can push them down to Parquet row-group pruning instead of scanning every file.

# All rows as an Arrow table
table.scan().to_arrow()

# Filter and project. File pruning happens before I/O.
recent_clicks = (
    table.scan("event_type = 'click' AND occurred_at >= '2026-08-15'")
         .select("id", "user_id", "occurred_at")
         .to_polars()
)

# Or straight to pandas for notebook work
df = table.scan("user_id = 7").to_pandas()

Polars integration goes through the Arrow PyCapsule interface, so there is no serialisation step between the scan and the LazyFrame. That matters at pipeline scale. A typical few-GB CDC batch stays zero-copy end to end. If you want to go one step further and skip DuckDB entirely, ducklake-dataframe reads the same catalog directly from Polars, pandas, or PySpark using each engine's native Parquet reader.

Time travel and snapshots

Every commit creates a snapshot with a monotonically increasing ID and a wall-clock timestamp. Because snapshots live in the catalog database, listing them is a single SELECT, so no manifest scan required. You can query historical state by snapshot ID, by timestamp, or roll the table back to a previous version.

from datetime import datetime

# List all snapshots (they're just rows in the catalog)
for snap in table.snapshots():
    print(snap.id, snap.committed_at, snap.summary)

# Query the table as it was at a specific snapshot
old = table.scan().with_snapshot(7).to_arrow()

# Or at a wall-clock timestamp. DuckLake picks the snapshot in effect then.
before_incident = table.scan().with_timestamp(
    datetime(2026, 8, 15, 9, 30, 0)
).to_arrow()

# Rollback discards later snapshots and creates a new snapshot pointing at
# the old data files. Nothing is deleted immediately; expiration handles that.
table.rollback_to_snapshot(7)

From the SQL side the equivalent is SELECT ... FROM events AT (VERSION => 7) or AT (TIMESTAMP => ...). Snapshot retention is configurable; the default keeps everything until you run an expiration procedure, which deletes the metadata rows and marks the orphaned Parquet files for cleanup. Time-travel debugging in production is a genuine reason to run a lakehouse format rather than raw Parquet, and DuckLake makes the query cost of it essentially free. It's an indexed lookup on the snapshot table.

Schema evolution and multi-table transactions

Schema evolution works the way you'd hope: add nullable columns, rename columns, drop columns, and widen types, all as metadata-only operations that don't rewrite Parquet files. The Python API uses a context-manager pattern that stages the changes and commits atomically.

with table.update_schema() as update:
    update.add_column("session_id", StringType())
    update.rename_column("event_type", "action")
    update.drop_column("legacy_field")
# One commit, one snapshot bump.

The feature that Iceberg and Delta genuinely do not have is multi-table transactions. In DuckLake, because the catalog is a real database, you can wrap changes across several tables in a single commit. If any step fails, the whole thing rolls back and no snapshot is created on either table. I hit this exact problem shipping an outbox last year on Iceberg (we had to build ugly two-phase logic around a Postgres side-table), so having native atomic multi-table writes here is a genuine relief. It's also useful for keeping fact-and-dimension writes consistent.

with catalog.begin_transaction() as txn:
    orders = txn.load_table("orders")
    items  = txn.load_table("order_items")

    orders.append(order_batch)
    items.append(item_batch)
    # If either append raises, both are rolled back and no snapshot is
    # committed on either table. Clean exit commits atomically.

You can do similar cross-table upserts inside SQL with BEGIN TRANSACTION / COMMIT on the DuckLake attachment. Under the hood these are Postgres or DuckDB transactions on the catalog schema, so the ACID guarantees are whatever your catalog database provides.

Partitioning, sorted tables, and data inlining

DuckLake supports both hash-bucket and range partitioning, declared at table-create time. The catalog stores partition boundaries per file, so predicates on the partition column prune files without opening Parquet footers.

con.execute("""
    CREATE TABLE page_views (
        id BIGINT,
        user_id BIGINT,
        occurred_at TIMESTAMP
    ) PARTITIONED BY (day(occurred_at), bucket(16, user_id))
""")

Sorted tables, added in 1.0, let you declare an ordering column so DuckLake can produce sorted Parquet files during writes and prune on ranges. The classic use case is a timestamp column on an append-only event table.

con.execute("""
    CREATE TABLE metrics (
        ts TIMESTAMP,
        metric VARCHAR,
        value DOUBLE
    ) SORTED BY (ts)
""")

Small-file management is where DuckLake diverges most from Iceberg and Delta. Both older formats need periodic compaction jobs to prevent every streaming append from producing a tiny Parquet file. DuckLake introduces data inlining: commits smaller than a configurable threshold (default 10 rows) are stored directly in the catalog as inline row data instead of Parquet. A CHECKPOINT command flushes accumulated inline data into a proper Parquet file when it makes sense. That single feature makes DuckLake genuinely usable for CDC ingest and IoT-style micro-batches without a compaction cron job hanging over your head.

Deploying DuckLake on S3 with a Postgres catalog

Local SQLite is fine for exploration. For a real deployment you want the catalog in Postgres (so multiple writers can coordinate) and the data in S3 (so the compute layer is stateless). The wiring is straightforward.

import duckdb

con = duckdb.connect()
con.execute("INSTALL ducklake")
con.execute("INSTALL postgres")
con.execute("INSTALL httpfs")
con.execute("LOAD ducklake")

# S3 credentials. Use SECRET for anything long-lived.
con.execute("""
    CREATE SECRET s3_lake (
        TYPE S3,
        KEY_ID 'AKIA...',
        SECRET '...',
        REGION 'us-east-1'
    )
""")

con.execute("""
    ATTACH 'ducklake:postgres:dbname=lake_catalog host=... user=... password=...' AS lake
    (DATA_PATH 's3://my-lake/data/')
""")

con.execute("USE lake")

From here everything else (CREATE TABLE, INSERT, SELECT, time travel) is identical to the local example. The catalog rows go to Postgres, the Parquet files go to S3, and any DuckDB or pyducklake process with credentials to both can join in. This is where DuckDB's positioning as a lightweight analytical engine really pays off: your query pods stay stateless and cheap, and the coordination is offloaded to a database you were already going to run.

When to use DuckLake (and when not to)

So, when does this actually fit? DuckLake makes the most sense for teams that already run a Postgres instance, mostly write from Python or DuckDB, and want lakehouse semantics without the operational tail of an Iceberg catalog service. The wins are concrete: cheap commits, multi-table transactions, no compaction cron for small writes, and a single SQL query to list snapshots. If your workload is a Python ELT pipeline that lands data in S3 and gets queried by a handful of analysts, DuckLake will feel like a straight upgrade over managing Parquet files by hand.

It's a bad fit if you need Spark or Flink as your primary compute engine, if your organisation has already standardised on Iceberg with a shared catalog, or if you want a fully serverless story where nothing but S3 has to exist between writers and readers. Iceberg's file-based metadata is genuinely more portable across engines and clouds, and Snowflake and BigQuery both have first-class Iceberg support that DuckLake does not currently match. The DuckLake GitHub repository is the place to track engine-support progress if that's your gating question.

If you're evaluating alongside adjacent tooling, the earlier comparison of Python vector databases covers a different piece of the modern data stack, but the same "reuse the SQL database you already have" instinct that made pgvector popular is exactly what DuckLake is doing for lakehouses. That framing tends to click with backend teams who have been sceptical of the JSON-manifest lakehouse story from the start.

Frequently Asked Questions

Is DuckLake production ready?

Yes. DuckLake 1.0 was released in April 2026 with a stable, backward-compatible spec, sorted tables, deletion vectors, and 68 of the 108 pre-release PRs focused on reliability. MotherDuck ships a hosted DuckLake in public preview, and multiple engines (DuckDB, DataFusion, Spark, Trino, Postgres) now implement the spec.

Can DuckLake replace Iceberg?

For Python-first teams that already run a SQL database and don't need Spark or Snowflake, yes. DuckLake is a strictly simpler operational story with faster small commits and native multi-table transactions. For organisations built around Snowflake, Trino, or Databricks, Iceberg or Delta Lake remain the safer choice because engine support is broader and more mature.

What databases can DuckLake use as a catalog?

DuckDB, SQLite, PostgreSQL, and MySQL are all supported catalog backends. Use DuckDB or SQLite for single-writer local work, and Postgres or MySQL for any shared deployment with concurrent writers.

Does DuckLake support S3, Azure Blob, and GCS?

Yes. The DATA_PATH in ATTACH accepts any URI DuckDB can read via its httpfs extension, which covers S3, S3-compatible stores like R2 and MinIO, Azure Blob, and Google Cloud Storage. Credentials use the standard DuckDB SECRET machinery.

How is DuckLake different from just writing Parquet to S3?

Raw Parquet gives you no ACID guarantees, no snapshots or time travel, no schema evolution, and no coordination between writers. DuckLake adds all four using a SQL catalog while keeping the data files as ordinary Parquet, so you get lakehouse semantics without the manifest-file overhead of Iceberg or Delta.