PyIceberg in Python: A Practical Guide to Apache Iceberg Tables (2026)

Updated: June 16, 2026

PyIceberg is the official Python library for reading, writing, and managing Apache Iceberg tables without a JVM or Spark cluster. In 2026 it has become the default way to work with lakehouse data from notebooks, FastAPI services, and lightweight ETL jobs, talking to REST, Glue, or SQL catalogs and returning PyArrow tables, pandas DataFrames, or Polars LazyFrames. This guide walks through PyIceberg 0.9 and covers catalog setup, reads, appends, upserts, schema evolution, time travel, and how it stacks up against Spark, Delta Lake, and Hudi.

• PyIceberg 0.9 (released April 2026) adds full upsert support, partition statistics, and a stable InMemoryCatalog for local testing.
• You can read an Iceberg table into a pandas DataFrame in three lines once a catalog is configured. No Spark, Hadoop, or JVM needed.
• Catalog choice matters more than engine choice. A REST catalog is the 2026 default for portability, while Glue and Hive remain common on AWS.
• Iceberg's hidden partitioning, snapshot isolation, and time travel survive across writers, so a Python writer and a Trino reader stay consistent.
• For tables under a few TB, PyIceberg paired with DuckDB or Polars often beats a Spark cluster on both cost and developer iteration speed.
• Common production pitfalls: forgetting to expire snapshots, writing tiny files without compaction, and mixing engine versions that disagree on the spec.

What is PyIceberg, and why use it over Spark?

PyIceberg is the native Python implementation of the Apache Iceberg table spec. It implements the same catalog, snapshot, manifest, and data-file layout that Spark, Trino, Flink, and Snowflake use, but without dragging in a JVM. You install it with pip, point it at a catalog, and you get table objects whose scan() method returns PyArrow or pandas data structures directly.

So, why does that matter in 2026? Because most data work doesn't actually need Spark. Iceberg's value is the spec (atomic writes, schema evolution, partition pruning via metadata, and snapshot isolation), and that value is now accessible from a 200 MB Python container instead of a 4 GB Spark image. I've been moving FastAPI ingest services and Airflow tasks off PySpark and onto PyIceberg plus DuckDB for the past year, and the cost-per-job has fallen roughly 8x. Spark still wins on shuffles over hundreds of TB, but for the long tail of "merge a few million rows and re-publish" jobs, PyIceberg is plainly the right tool.

The 0.9 release added the missing piece many teams had been waiting for: a real upsert() API backed by merge-on-read, plus partition statistics that let engines like DuckDB skip files using min/max values from manifests. The library now covers the v2 spec almost completely, with experimental v3 (deletion vectors, row lineage) behind feature flags.

Installing PyIceberg and setting up a catalog

Install PyIceberg with the extras you need. The library is modular, so you only pull in the catalog and filesystem clients you'll actually use.

# Local testing: SQLite catalog + local filesystem
pip install "pyiceberg[sql-sqlite,pyarrow]==0.9.1"

# Production: REST catalog + S3 + Glue + DuckDB integration
pip install "pyiceberg[rest,glue,s3fs,duckdb,pyarrow]==0.9.1"

An Iceberg table is identified by a catalog plus a namespace plus a table name. The catalog is the source of truth for "which metadata file is the current one." Picking the right catalog is the single most consequential decision when adopting Iceberg, and it's where most teams stumble. Honestly, I hit this exact issue on my first production rollout, with two services pointing at the same warehouse via different catalogs, and the resulting "lost commit" took a weekend to untangle.

The three catalogs that matter in 2026

• REST catalog. The portable default. Any spec-compliant REST server (Polaris, Lakekeeper, Nessie, Tabular's hosted catalog) works with every engine. This is what I reach for on greenfield projects.
• Glue catalog. Pragmatic on AWS if you already use Athena. PyIceberg talks to it natively; just make sure your IAM role can both read Glue and write to S3.
• SQL catalog. SQLite or Postgres-backed. Perfect for local development, CI tests, and small single-node deployments.

A minimal local catalog you can use right now, useful for examples and unit tests:

from pyiceberg.catalog.sql import SqlCatalog

catalog = SqlCatalog(
    "local",
    **{
        "uri": "sqlite:///./iceberg_catalog.db",
        "warehouse": "file://./warehouse",
    },
)

catalog.create_namespace_if_not_exists("analytics")

And the equivalent REST catalog wiring for production, with credentials read from environment variables. Never check secrets into a notebook.

import os
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "prod",
    **{
        "type": "rest",
        "uri": os.environ["ICEBERG_REST_URI"],
        "credential": os.environ["ICEBERG_REST_CREDENTIAL"],
        "warehouse": "s3://my-lakehouse/warehouse",
        "s3.region": "us-east-1",
    },
)

How do you read an Iceberg table in Python?

Reading is the fastest path to seeing PyIceberg's value, because it costs nothing to add it next to your existing pandas or Polars code. The pattern is always the same: load the catalog, load the table, scan with optional filters, then materialise.

from pyiceberg.expressions import GreaterThan, And, EqualTo

table = catalog.load_table("analytics.orders")

scan = table.scan(
    row_filter=And(
        GreaterThan("order_total", 100),
        EqualTo("country", "PT"),
    ),
    selected_fields=("order_id", "customer_id", "order_total", "created_at"),
)

# Materialise into whichever runtime you prefer
arrow_table = scan.to_arrow()      # PyArrow Table, zero copy
df = scan.to_pandas()              # pandas DataFrame
pl_df = scan.to_polars()           # Polars DataFrame (0.9+)
duck_rel = scan.to_duckdb("orders")  # DuckDB relation, queryable via SQL

Two things are happening here that matter for performance. First, the row_filter is pushed all the way down to file pruning: PyIceberg reads the table's manifests, uses the min/max statistics on partition columns and data columns to discard entire Parquet files, and only opens the survivors. Second, selected_fields restricts column projection at the Parquet level, so you pay for the bytes you actually read.

For interactive analysis, the DuckDB path is the one I reach for most often. You get full SQL (joins, windows, CTEs) over an Iceberg table without copying data, and DuckDB's query planner combines with Iceberg's file pruning to deliver surprisingly fast queries on tables in the hundreds of GB. If you're new to that combination, our practical guide to Polars and DuckDB covers the broader ergonomics.

Writing to Iceberg: appends, overwrites, and upserts

The 0.9 release moved PyIceberg from "good reader, awkward writer" to a credible production writer. Three write modes cover almost every use case: append for incremental loads, overwrite for full refreshes (often partition-scoped), and upsert for merge-on-read CDC.

The full lifecycle of an append-only ingestion, from table creation to write, looks like this:

import pyarrow as pa
from pyiceberg.schema import Schema
from pyiceberg.types import (
    NestedField, LongType, StringType,
    DoubleType, TimestampType,
)
from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform

schema = Schema(
    NestedField(1, "order_id", LongType(), required=True),
    NestedField(2, "customer_id", LongType(), required=True),
    NestedField(3, "country", StringType(), required=True),
    NestedField(4, "order_total", DoubleType(), required=True),
    NestedField(5, "created_at", TimestampType(), required=True),
)

# Hidden partitioning: physical layout follows day(created_at),
# but queries still filter on created_at directly.
partition_spec = PartitionSpec(
    PartitionField(
        source_id=5, field_id=1000,
        transform=DayTransform(), name="created_at_day",
    ),
)

table = catalog.create_table_if_not_exists(
    "analytics.orders",
    schema=schema,
    partition_spec=partition_spec,
)

batch = pa.table({
    "order_id": [1001, 1002, 1003],
    "customer_id": [42, 42, 99],
    "country": ["PT", "PT", "ES"],
    "order_total": [120.0, 45.5, 880.0],
    "created_at": pa.array(
        ["2026-06-15T10:00:00", "2026-06-15T10:05:00", "2026-06-16T09:00:00"],
        type=pa.timestamp("us"),
    ),
})

table.append(batch)

For idempotent reloads (the pattern most ETL jobs actually need), use a partition-scoped overwrite. This is the safest way to make a job rerunnable without leaving zombie rows behind:

from pyiceberg.expressions import EqualTo

# Replace exactly the 2026-06-15 partition, leaving every other day untouched.
table.overwrite(
    df=batch,
    overwrite_filter=EqualTo("created_at_day", "2026-06-15"),
)

Upserts (new in 0.9) take a key list and either insert new rows or update existing ones, using merge-on-read delete files under the hood:

updates = pa.table({
    "order_id": [1001, 1004],
    "customer_id": [42, 77],
    "country": ["PT", "FR"],
    "order_total": [125.0, 30.0],  # 1001 is corrected; 1004 is new
    "created_at": pa.array(
        ["2026-06-15T10:00:00", "2026-06-16T11:30:00"],
        type=pa.timestamp("us"),
    ),
})

result = table.upsert(updates, join_cols=["order_id"])
print(result.rows_updated, result.rows_inserted)  # 1 1

If your ingestion is fed by HTTP APIs, our walkthrough of async ETL with httpx and asyncio pairs well with this writer pattern. You fan out async fetches, batch results into Arrow, and land them in Iceberg with a single append() call per micro-batch.

Schema evolution and hidden partitioning

Schema evolution is the killer feature that Parquet alone doesn't give you. Iceberg tracks columns by stable numeric IDs, not by name, so renaming a column or dropping one mid-flight doesn't corrupt downstream readers. The 2026 PyIceberg API exposes this through an update_schema() transaction.

from pyiceberg.types import StringType, DoubleType

with table.update_schema() as update:
    update.add_column("currency", StringType(), required=False)
    update.rename_column("order_total", "amount_eur")
    update.update_column("amount_eur", field_type=DoubleType())
    # Drop is allowed but irreversible. Old data files still exist
    # but the column becomes invisible to all readers.
    # update.delete_column("legacy_flag")

Hidden partitioning is the other Iceberg-specific concept worth internalising. With Hive-style partitioning you'd write WHERE order_date = '2026-06-15' and pray the partition column exists. Iceberg lets you partition by day(created_at), but consumers still query the original created_at column. The engine translates the predicate into a partition prune using the transform metadata. The transforms PyIceberg supports today are:

• IdentityTransform: same value (the Hive equivalent).
• YearTransform, MonthTransform, DayTransform, HourTransform: time bucketing.
• BucketTransform(n): hash bucketing, ideal for high-cardinality join keys.
• TruncateTransform(n): string or numeric truncation.

Partition specs can themselves evolve. If a table starts daily and a year later you want hourly, you can add a new spec and old data keeps its old partitioning while new writes use the new one, no rewrite required. This kind of "I changed my mind about the physical layout" recovery is the reason Iceberg has displaced raw Parquet in most lakehouse stacks.

Time travel and snapshot management

Every write to an Iceberg table produces a new snapshot, and every snapshot is queryable by id or timestamp. This is invaluable for debugging "what did the data look like yesterday at 14:00?" and for cheap rollbacks after a bad ETL run.

import datetime as dt

# List snapshots and their summaries
for snap in table.snapshots():
    print(snap.snapshot_id, snap.timestamp_ms, snap.summary["operation"])

# Read the table as it was at a specific snapshot
old = table.scan(snapshot_id=table.snapshots()[0].snapshot_id).to_pandas()

# Or roll back when an upstream system wrote bad data
bad_snapshot_id = table.current_snapshot().snapshot_id
table.manage_snapshots().rollback_to_snapshot(
    target_snapshot_id=table.snapshots()[-2].snapshot_id
).commit()

Two practical notes. First, snapshots are cheap (they're just metadata files) but the data files they reference are not. Keeping a year of snapshots can quietly multiply storage cost if a busy table rewrites large partitions. Expire old snapshots on a schedule:

cutoff_ms = int((dt.datetime.utcnow() - dt.timedelta(days=14)).timestamp() * 1000)
table.expire_snapshots().expire_older_than(cutoff_ms).commit()

Second, the snapshot log is the right place to look when two engines disagree about table state. If a Trino query returns different rows than a PyIceberg scan, run table.refresh() and compare current snapshot ids. Nine times out of ten one of the engines is caching a stale metadata pointer.

PyIceberg vs Spark, Delta Lake, and Hudi

Iceberg isn't the only open table format, and PyIceberg isn't the only way to talk to Iceberg. Here is how the four options actually compare when you need to choose for a Python-first 2026 project.

Dimension	PyIceberg 0.9	Spark + Iceberg	Delta Lake (delta-rs)	PyHudi
Language runtime	Pure Python	JVM + PySpark	Rust core, Python binding	JVM + Python wrapper
Container image size	~200 MB	~4 GB	~300 MB	~3.5 GB
Catalog options	REST, Glue, Hive, SQL, in-memory	Same + Spark-native	Unity, AWS Glue, file-based	Hive, Glue, file-based
Upserts / merge	Yes (merge-on-read)	Yes (both modes)	Yes (copy-on-write)	Yes (both modes)
Schema evolution	Add, drop, rename, retype	Same	Add, drop only	Add, drop, rename
Best at	Notebooks, FastAPI, small/medium ETL	Petabyte shuffles	Databricks ecosystem	CDC-heavy ingest
Cross-engine reach	Trino, DuckDB, Flink, Snowflake, BigQuery	Same	Mostly Spark/Databricks	Spark, Flink, Presto

My rough rule: choose PyIceberg as the writer for jobs that fit on one machine and produce up to a few hundred GB of new data per run. Choose Spark when you need broadcast joins across multi-terabyte tables. Choose Delta Lake only if your stack is centred on Databricks. Hudi remains the strongest answer for streaming CDC ingestion, but its Python story still trails Iceberg's.

If you're orchestrating these jobs, the same trade-offs we covered in Airflow vs Prefect vs Dagster for 2026 apply. Dagster's asset model maps especially cleanly onto Iceberg tables, with each materialisation producing a new snapshot.

Production patterns, compaction, and pitfalls

Most teams hit the same handful of operational issues during their first six months on Iceberg. Front-loading the fixes saves a lot of pain.

Compact small files on a schedule

Streaming or micro-batch writes produce many small files. Iceberg's manifest layer hides this from readers, but per-file open cost still dominates query latency once you're past a few thousand files per partition. Run periodic compaction:

table.rewrite_files(
    target_file_size_bytes=512 * 1024 * 1024,  # 512 MB
    min_input_files=5,
).commit()

Expire snapshots and clean up orphans

Snapshot expiration removes metadata but does not delete data files. Run remove_orphan_files() weekly or monthly to actually reclaim S3 storage. Set a generous safety margin, longer than the longest possible in-flight query.

Watch out for catalog drift

If you have both PyIceberg and another engine writing to the same table, you'll occasionally see a CommitFailedException. This is Iceberg's optimistic concurrency control firing correctly. Retry the transaction. What you must not do is bypass the catalog and point at a metadata file directly, which is the single most common cause of silent data loss in lakehouse deployments.

Test against a real catalog in CI

Mocked Iceberg writes are useless, because most bugs live in catalog interaction. The InMemoryCatalog introduced in 0.9 is fast enough to spin up per-test and behaves identically to a REST catalog for everything except network failure simulation. Use it.

from pyiceberg.catalog.memory import InMemoryCatalog

def test_upsert_dedups_on_key():
    catalog = InMemoryCatalog("test", warehouse="file:///tmp/test_warehouse")
    catalog.create_namespace("t")
    table = catalog.create_table("t.orders", schema=schema)
    table.append(initial_batch)
    table.upsert(updates_batch, join_cols=["order_id"])
    df = table.scan().to_pandas()
    assert len(df) == expected_row_count

Frequently Asked Questions

Can pandas read Iceberg tables directly?

Not natively, but PyIceberg provides a one-liner: table.scan().to_pandas() returns a fully materialised DataFrame, with filters and column projection pushed down to the Parquet layer for free.

Do I need Spark or a JVM to use PyIceberg?

No. PyIceberg is a pure Python implementation of the Iceberg spec. It reads and writes the same metadata files that Spark, Trino, and Flink use, but runs in a normal Python process with no JVM dependency.

Which catalog should I use with PyIceberg?

Use a REST catalog (Polaris, Lakekeeper, or Nessie) for new projects, since it is the most portable across engines. Use AWS Glue if you are already invested in Athena. Use the SQLite-backed SQL catalog for local development and CI.

What is the difference between PyIceberg and Apache Iceberg?

Apache Iceberg is the open table-format specification. PyIceberg is the Python client library that implements that specification. The same table can be written from PyIceberg and read by Spark, Trino, or BigQuery, because they all agree on the spec.

How does PyIceberg handle concurrent writes?

Iceberg uses optimistic concurrency at the catalog layer. If two writers commit at the same time, one wins and the other receives a CommitFailedException that the application should retry. There is no row-level locking; conflict resolution is per-snapshot.

Is PyIceberg production-ready in 2026?

Yes for read-heavy and append-heavy workloads, and for merge workloads at moderate scale. The 0.9 release closed the major gaps around upserts and partition statistics. For petabyte-scale shuffles or complex MERGE statements, Spark on Iceberg is still the stronger choice.