Delta Lake in Python with delta-rs: Merge, Time Travel, and Z-Order Without Spark (2026)

Updated: July 5, 2026

Delta Lake in Python is a Rust-native storage layer (installed as pip install deltalake) that adds ACID transactions, MERGE upserts, time travel, and Z-ordered file skipping to Parquet files, without any Spark or JVM dependency. In this guide I'll walk through the deltalake 1.6 Python package (released June 2026), show every operation you actually need (write, merge, time travel, optimize, schema evolution), and pair each snippet with the reasoning behind it. If you've been staring at a bucket full of Parquet and wondering how to add mutability without touching Spark, you're in the right notebook.

• delta-rs is a native Rust implementation of the Delta Lake protocol; the Python deltalake package (v1.6.1, June 2026) gives you ACID Parquet without Spark or a JVM.
• You can read and write Delta tables from pandas, Polars, DuckDB, and PyArrow, all through the same Arrow-backed transaction log.
• MERGE supports upsert, delete-when-matched, insert-when-not-matched, and (new in 2026) schema evolution during merges.
• Time travel exposes any prior snapshot by version integer or timestamp, plus RESTORE to make an older snapshot the new head.
• OPTIMIZE + Z-ORDER compacts small files and colocates values so downstream queries skip most of the table via min/max statistics.
• Compared to PyIceberg, delta-rs is simpler to operate for single-writer Python pipelines; Iceberg wins for multi-engine catalogs.

What is delta-rs and how does it differ from Delta Spark?

Delta Lake started life as a Databricks project tightly coupled to Apache Spark. Every read replayed the JSON transaction log inside a JVM, and every write went through Spark's DataFrame API. That coupling was fine when your data lived in one warehouse, but painful when you wanted to write from a lightweight Python service, a Lambda function, or a Jupyter notebook. delta-rs is the answer: a native Rust library, published on GitHub as delta-io/delta-rs, that implements the same Delta protocol without Java or Spark.

The Python bindings (the deltalake package) expose that Rust core through PyO3 with zero-copy Arrow data exchange. In practical terms, a pandas.DataFrame and a polars.DataFrame can round-trip to Delta without a serialization tax, and a DuckDB scan of a Delta table hands the query planner the same Parquet files that Delta Spark would read. The 2026 releases (1.4 in January, 1.5 in March, 1.6 in May, 1.6.1 in June) have caught up on the two features that historically pushed teams toward Spark: full MERGE with schema evolution, and OPTIMIZE with Z-ordering executed via DataFusion 51 inside the Rust engine.

My rule of thumb after shipping several production pipelines on delta-rs: if a single Python worker (or a small Ray/Dask fleet) can push through your write volume, delta-rs is the right choice. It removes an entire class of Spark session-tuning headaches. When you need thousands of concurrent writers against one table, or a shared cluster of analysts hitting the same lakehouse, Delta Spark or a managed lakehouse still earns its keep.

Install and set up deltalake 1.6

The package requires Python 3.10 through 3.14. The 1.6 line dropped Python 3.9 to unlock the PyO3 26 upgrade, so pin your CI accordingly.

# Install with your favourite package manager
pip install "deltalake==1.6.1" pyarrow==17 pandas polars duckdb
# uv add deltalake  # if you use uv
# poetry add deltalake

# Sanity-check the version
python -c "import deltalake; print(deltalake.__version__)"
# 1.6.1

Cloud storage is configured through environment variables or an explicit storage_options dict. delta-rs speaks S3, GCS, Azure Blob, ADLS Gen2, and any S3-compatible endpoint (MinIO, R2, B2). For S3 with DynamoDB-based locking (the standard way to make concurrent writers safe on object storage without atomic renames), you set:

storage_options = {
    "AWS_REGION": "us-east-1",
    "AWS_S3_LOCKING_PROVIDER": "dynamodb",
    "DELTA_DYNAMO_TABLE_NAME": "delta_log_lock",
    # AWS creds picked up from env / IAM role in the usual way
}

Writing and reading Delta tables from pandas and Polars

The two entry points are write_deltalake for writes and DeltaTable for everything else. Both accept anything that can be turned into an Arrow table, which means pandas DataFrames, Polars DataFrames, PyArrow Tables, and iterators of PyArrow RecordBatches are all first-class inputs.

import pandas as pd
import polars as pl
from deltalake import DeltaTable, write_deltalake

# --- Write from pandas ---
pdf = pd.DataFrame({
    "order_id": [1, 2, 3, 4],
    "customer": ["ana", "ben", "cara", "dan"],
    "amount":  [12.5, 33.0, 8.75, 41.2],
    "region":  ["eu", "us", "eu", "us"],
})
write_deltalake(
    "s3://analytics/orders",
    pdf,
    mode="overwrite",           # 'append' | 'overwrite' | 'error' | 'ignore'
    partition_by=["region"],
    storage_options=storage_options,
)

# --- Append from Polars ---
new_rows = pl.DataFrame({
    "order_id": [5, 6],
    "customer": ["eli", "fay"],
    "amount":  [19.0, 27.4],
    "region":  ["eu", "us"],
})
new_rows.write_delta(
    "s3://analytics/orders",
    mode="append",
    storage_options=storage_options,
)

# --- Read back into pandas or Polars ---
dt = DeltaTable("s3://analytics/orders", storage_options=storage_options)
pdf_out = dt.to_pandas()                       # pandas
plf_out = pl.read_delta("s3://analytics/orders", storage_options=storage_options)  # Polars

# --- Partition pushdown: only scan region=eu files ---
eu_only = dt.to_pandas(partitions=[("region", "=", "eu")])

Under the hood, both writes commit a new JSON log entry into _delta_log/. The commit contains the list of Parquet files added or removed and the schema at that snapshot. Because the write is atomic (one JSON file), a crashed writer can never leave the table half-written. The transaction log either sees the commit or it doesn't. That guarantee is what makes Delta "ACID on top of Parquet," and it's the reason I default to Delta over raw Parquet for any dataset that will ever change.

If you're comparing dataframe engines while designing the pipeline, my Polars vs pandas benchmark shows Polars typically finishes the Delta write in half the wall time on tables above a few gigabytes, because it can spill row-groups in parallel through the Rust engine.

How do I merge and upsert with Delta Lake in Python?

MERGE is the operation that makes Delta feel like a database instead of a file dump. The DeltaTable.merge method takes a source (any Arrow-compatible input), a SQL predicate joining source and target, and a fluent chain of when_matched_* / when_not_matched_* clauses. Everything below runs without Spark:

import pyarrow as pa
from deltalake import DeltaTable

dt = DeltaTable("s3://analytics/orders", storage_options=storage_options)

updates = pa.table({
    "order_id": [3, 4, 7, 8],
    "customer": ["cara", "dan-v2", "gus", "hana"],
    "amount":  [10.0, 41.2, 55.5, 12.0],
    "region":  ["eu", "us", "eu", "us"],
})

(
    dt.merge(
        source=updates,
        predicate="target.order_id = source.order_id",
        source_alias="source",
        target_alias="target",
    )
    # Update matching rows only when the incoming amount actually changed
    .when_matched_update(
        predicate="target.amount != source.amount",
        updates={
            "customer": "source.customer",
            "amount":   "source.amount",
        },
    )
    # Insert brand-new order ids
    .when_not_matched_insert(
        updates={
            "order_id": "source.order_id",
            "customer": "source.customer",
            "amount":   "source.amount",
            "region":   "source.region",
        },
    )
    .execute()
)

Three things worth calling out. First, the predicate passed to when_matched_update is a filter, not a join key. Use it to skip no-op updates so you don't churn the transaction log. Second, delta-rs 1.5 added schema evolution during merge: pass merge_schema=True to merge() and any new columns in the source get added to the target. Third, delta-rs pushes the merge predicate down to file pruning, so a merge on a partition column reads only the touched partitions. That's critical for slowly-changing dimensions.

How does Delta Lake time travel work?

Every commit to a Delta table produces a numbered JSON file in _delta_log/: 00000000000000000000.json, 00000000000000000001.json, and so on. Reading version N means replaying commits 0…N to reconstruct the file list. Because commits are immutable, any prior version is trivially recoverable, which is the "time travel" primitive.

from deltalake import DeltaTable
from datetime import datetime, timezone

dt = DeltaTable("s3://analytics/orders", storage_options=storage_options)

# 1. Read the state of the table at commit version 3
dt_v3 = DeltaTable(
    "s3://analytics/orders",
    version=3,
    storage_options=storage_options,
)
snapshot = dt_v3.to_pandas()

# 2. Read by wall-clock timestamp (nearest commit <= timestamp)
dt.load_as_version(datetime(2026, 7, 1, tzinfo=timezone.utc))
before_july = dt.to_pandas()

# 3. Inspect commit history: who wrote what, when
for commit in dt.history(limit=10):
    print(commit["version"], commit["timestamp"], commit["operation"])

# 4. RESTORE the table to a prior version as a new HEAD commit
dt.restore(target=3)  # or a datetime

The two-line safety net that has saved me most often is restore(). When a bad backfill lands in production, you don't need to reload from source. You commit a restore to the last-known-good version and every downstream reader sees the fixed table on its next scan. I hit this exact scenario a couple of quarters back, and a one-line restore beat a two-hour re-ingest by, well, about two hours. It gets you closer to the recovery guarantees I described in the production-ready data cleaning guide without any custom rollback logic.

OPTIMIZE, Z-ORDER, and VACUUM without Spark

Streaming pipelines produce lots of small Parquet files (one per commit, often only a few MB). Small files kill scan performance because the query engine pays fixed per-file overhead. OPTIMIZE compacts them into fewer, larger files (target 100–500 MB), and Z-ORDER reorders rows inside each file so that similar values on the specified columns end up in the same row groups. Combined with Parquet's min/max statistics, this lets a query on customer_id = 'x' skip 95%+ of files without reading them.

from deltalake import DeltaTable

dt = DeltaTable("s3://analytics/orders", storage_options=storage_options)

# 1. Compact small files into target-sized ones
dt.optimize.compact(
    target_size=256 * 1024 * 1024,   # 256 MB
    max_temp_directory_size="8GB",   # spill limit for DataFusion
)

# 2. Z-ORDER on the columns you filter by most
dt.optimize.z_order(
    columns=["customer", "order_id"],
    target_size=256 * 1024 * 1024,
)

# 3. Clean up files no longer referenced by any live snapshot.
#    dry_run first, always.
to_delete = dt.vacuum(retention_hours=168, dry_run=True)  # 7 days
print(f"Would delete {len(to_delete)} files")
dt.vacuum(retention_hours=168, dry_run=False)

The max_temp_directory_size parameter (added in delta-rs 1.4) is worth knowing about: DataFusion needs scratch space for the sort, and on a small machine it'll OOM without a spill cap. Set it to about a third of available RAM.

Z-ORDER is not free (it rewrites every file it touches), so treat it as a periodic maintenance job, not a per-commit step. Nightly for high-churn tables, weekly for stable ones. In practice I schedule it inside the same orchestrator DAG that runs my nightly ETL, using patterns like the ones I covered in Airflow vs Prefect vs Dagster.

Schema evolution and partition management

Real datasets grow columns over time. delta-rs handles three modes of schema change: additive (new columns), permissive rewrites (type widening), and full overwrites. The 1.6 release finally unified the API so all three go through the same schema_mode parameter:

import pyarrow as pa
from deltalake import write_deltalake

# 1. Append a superset of the current schema; new columns get filled with NULL
enriched = pa.table({
    "order_id":  [9, 10],
    "customer":  ["ivy", "jax"],
    "amount":   [22.0, 44.0],
    "region":   ["eu", "us"],
    "channel":  ["web", "mobile"],   # NEW column
})
write_deltalake(
    "s3://analytics/orders",
    enriched,
    mode="append",
    schema_mode="merge",             # additive evolution
    storage_options=storage_options,
)

# 2. Overwrite the table with a different schema entirely (rare, for backfills)
write_deltalake(
    "s3://analytics/orders",
    corrected,
    mode="overwrite",
    schema_mode="overwrite",
    storage_options=storage_options,
)

Unlike Iceberg, Delta does not support partition evolution. Changing a table's partitioning scheme means rewriting every file. That's the single biggest reason I still reach for PyIceberg on tables whose access pattern I can't fully predict up front. For tables where I know the query shape (which is most of what a Python data team produces), Delta's simpler mental model wins.

Delta Lake vs Apache Iceberg for Python users

These two formats now cover roughly the same feature surface, and thanks to Delta UniForm and Apache XTable you can even write one and read the other. So the real decision is about ecosystem fit.

Dimension	Delta Lake (delta-rs)	Apache Iceberg (PyIceberg)
Python package	deltalake 1.6.1 (Rust core)	pyiceberg 0.11.1 (pure Python)
Concurrent writers	Requires DynamoDB lock on S3	Handled by catalog (Glue, Polaris, Nessie)
Engine reach	Delta Spark, Databricks, DuckDB, Polars, pandas	Spark, Flink, Trino, Snowflake, BigQuery, DuckDB, Polars
Partition evolution	No, full rewrite required	Yes, native
MERGE support	Full, with schema evolution (1.5+)	Full (1.10)
Time travel	Version or timestamp	Version or timestamp
Best fit	Databricks shops, Python-first pipelines, single-writer ETL	Multi-engine lakehouses, catalog-driven governance

My working heuristic in 2026: if the same Python team owns writes and reads, use Delta. If separate teams read from Trino or BigQuery while another writes from Spark, use Iceberg. Interop bridges exist for both directions, so this is a "start with the format your dominant workload prefers" decision, not a permanent one.

A production checklist for delta-rs pipelines

Everything above works in a notebook. Getting it to stay working in production requires a few extra habits. Here's the checklist I hand to teams before their first delta-rs pipeline goes live.

• Enable DynamoDB locking the moment you have more than one writer, even if today "more than one" means "the pipeline plus one ad-hoc notebook."
• Pin deltalake in your lockfile. The protocol writer version bumps in minor releases and older readers can refuse to open newer tables.
• Set configuration={"delta.checkpointInterval": "10"} when creating the table. Checkpoints collapse the transaction log so cold reads don't replay thousands of JSON files.
• Schedule OPTIMIZE and VACUUM as their own DAG task, not inside the write path. Never vacuum with less than 168 hours of retention unless you fully understand the consequences for time travel.
• Validate the schema on ingest with a Pandera or PyArrow schema check before calling write_deltalake. Delta will accept whatever Arrow schema you throw at it; you want to reject bad data at the boundary.
• Emit OpenTelemetry traces (delta-rs 1.6 wires them through natively) so you can see commit latency and file counts on your dashboards.
• Test time travel and restore in staging quarterly. Recovery is a muscle; if the first time you use restore() is during an incident, the mean time to repair will be higher than it needs to be.

Frequently Asked Questions

Do I need Spark to use Delta Lake in Python?

No. The deltalake package (delta-rs) implements the full Delta protocol in Rust with a Python binding. You can read, write, merge, time travel, optimize, and Z-order from a plain Python process on your laptop or a small VM. No JVM, no Spark cluster.

Can Polars read and write Delta Lake tables?

Yes. Polars ships pl.read_delta(path) and df.write_delta(path, mode=...), which delegate to delta-rs under the hood. Write modes include append, overwrite, error, ignore, and merge (returning a TableMerger for upsert workflows).

What is the difference between Delta Lake and Apache Iceberg?

Both are open ACID table formats built on Parquet. Delta uses a flat transaction log and integrates most deeply with the Spark/Databricks ecosystem; Iceberg uses a hierarchical metadata tree with an external catalog and has broader engine support (Trino, Snowflake, BigQuery, Flink). For single-team Python pipelines, Delta is simpler; for multi-engine lakehouses, Iceberg is safer.

How much retention do I need for Delta time travel?

Time travel works only for versions whose Parquet files still exist. The default retention is 7 days (168 hours), controlled by delta.deletedFileRetentionDuration. If you need a longer rollback window (say, 30 days for audit), raise that value and delay your VACUUM schedule to match.

Is delta-rs production-ready in 2026?

Yes. The 1.x line stabilized the API in 2025, and the 1.6 releases (Q2 2026) closed the last major gaps: schema evolution during MERGE, and DataFusion-backed OPTIMIZE with spill control. It's used in production by dozens of companies for single-writer or DynamoDB-locked multi-writer pipelines. For thousand-writer workloads inside a Databricks lakehouse, Delta Spark still has the edge.