DataFusion in Python: The Rust-Powered Apache Arrow Query Engine (2026)

Updated: August 31, 2026

DataFusion in Python is an Apache Arrow-native, Rust-based SQL and DataFrame query engine that you install as the datafusion package and use to run vectorized analytics over Parquet, CSV, JSON, Delta Lake, and Iceberg tables without spinning up a database server. It sits in the same design lineage as DuckDB and Polars (a single-node columnar engine backed by Apache Arrow memory), but it's the engine of choice when you need a customizable query planner, first-class Substrait support, or an execution core you can embed in your own tool. I've been using DataFusion, specifically datafusion-python 46+ released in mid-2026, in production notebooks and data pipelines for the last year, and this guide is the reference I wish I had when I started.

• DataFusion is the Rust query engine that powers Comet (Spark accelerator), InfluxDB 3.0, GreptimeDB, and a growing number of lakehouse tools; datafusion-python exposes it as a pip-installable package.
• You get both a SQL API (ctx.sql(...)) and a lazy DataFrame API (ctx.read_parquet(...).filter(...).aggregate(...)) that share the same optimizer and Arrow-backed execution.
• Unlike DuckDB, DataFusion is designed to be embedded and extended: custom optimizer rules, custom table providers, custom UDFs, and a Substrait producer/consumer are all first-class citizens.
• Zero-copy interop with pandas, Polars, PyArrow, and Ibis means you rarely serialize data. The same Arrow buffers pass through every stage.
• Combined with Ballista or Ray, the same query can run on one laptop or across a cluster with no rewrite.
• Choose DataFusion when you need extensibility or Substrait; choose DuckDB when you want the widest SQL surface out of the box; choose Polars when your workload is DataFrame-first and single-node.

What is DataFusion in Python?

Apache DataFusion is a top-level Apache Software Foundation project (it graduated from the Arrow subproject in 2024) that provides a modular query engine written in Rust. Its Python bindings, datafusion-python, expose the same execution core that ships inside InfluxDB 3.0, GreptimeDB, ROAPI, and the Comet accelerator for Apache Spark. The library is Arrow-native, meaning every record batch that flows through a plan is a contiguous Arrow buffer. No row-by-row Python overhead, no serialization tax when you hand the result to pandas or Polars.

Formally, DataFusion is a logical plan → physical plan → executor pipeline in the tradition described in the classic Volcano paper (Graefe, 1994), but with vectorized execution over Arrow's columnar layout. The Python package gives you a SessionContext (the equivalent of a Spark session or a DuckDB connection) plus registration APIs for tables, functions, and catalogs. You can drive it with SQL, with a Pandas-like DataFrame API, or by feeding it a Substrait plan produced by another tool.

Two things separate DataFusion from a run-of-the-mill embedded database. First, it's designed as a library rather than a database product: the crate ecosystem (datafusion-common, datafusion-expr, datafusion-physical-plan, datafusion-optimizer) is intentionally hackable, and Python users benefit indirectly through stable extension hooks. Second, it treats Arrow as the source of truth, so interop with the wider Python ecosystem (pandas, Polars, PyArrow, Ibis, DuckDB, Delta Lake, Iceberg) is essentially free. If you've already read our practical guide to Polars and DuckDB, DataFusion is the third leg of that stool.

How is DataFusion different from DuckDB?

DataFusion and DuckDB both give you a single-node columnar SQL engine that reads Parquet, but they optimize for different jobs. DuckDB is a product: an opinionated, batteries-included analytics database with the broadest SQL dialect coverage in the embedded space. DataFusion is a toolkit: a modular execution core meant to be embedded, extended, and reshaped by downstream projects. Honestly, if you're building an internal query layer, a domain-specific data tool, or a distributed engine, DataFusion is almost always the better foundation. If you want to open a notebook and run analytics on a Parquet dataset today, DuckDB usually gets you there with less ceremony.

The comparison table below covers the dimensions I get asked about most.

Feature	DataFusion (datafusion-python)	DuckDB	Polars
Core language	Rust	C++	Rust
Primary API	SQL + DataFrame + Substrait	SQL-first (relational API optional)	DataFrame-first (SQL context available)
Storage engine	None (Arrow in-memory + external tables)	Native columnar storage + Parquet/CSV/JSON	None (DataFrame in memory + external tables)
Custom Python UDFs	Scalar, aggregate, window; PyArrow-batched	Scalar, aggregate; PyArrow-batched	map_batches with PyArrow
Substrait plans	Producer + consumer, stable	Consumer (experimental)	Producer (experimental)
Distributed execution	Ballista, Ray DataFusion, Datafusion-Comet on Spark	Single-node (external scale-out via MotherDuck)	Single-node (Polars Cloud in preview)
Ideal use case	Embed / extend an engine, portable plans, custom optimizer	Ad-hoc analytics, notebook SQL, ELT	Pipelines that already live in the DataFrame world
License	Apache-2.0	MIT	MIT

Installing datafusion-python in 2026

As of August 2026, datafusion-python ships wheels for CPython 3.9 through 3.13 on Linux (x86_64 and aarch64), macOS (Intel and Apple Silicon), and Windows. Installation is a single command, and because the wheel bundles the Rust runtime you don't need cargo on the host. I'd recommend pinning a version. DataFusion releases monthly and the SQL surface still evolves fast enough that pinning avoids surprises in CI.

# pip
pip install "datafusion==46.0.0"

# uv (see our uv guide linked below)
uv add "datafusion==46.0.0"

# Verify
python -c "import datafusion; print(datafusion.__version__)"

If you also intend to work with Delta Lake or Iceberg, add the relevant client libraries in the same environment; DataFusion consumes their Arrow output directly.

uv add "deltalake==0.25.0" "pyiceberg==0.9.0" "pyarrow==17.0.0"

Package management is the other axis where you have choices in 2026. If you're still on plain pip and requirements.txt, my recent write-up on uv as an Astral-built package manager for data science explains why I've moved most projects onto uv. The wall-clock difference on cold environments with heavy binary wheels like datafusion and pyarrow is dramatic.

Querying Parquet with SQL and the DataFrame API

The two entry points are SessionContext.sql and the DataFrame API rooted at ctx.read_parquet. Both compile to the same logical plan, so pick the one that reads more naturally for your team.

from datafusion import SessionContext, col, functions as F

ctx = SessionContext()

# Register a Parquet dataset as a table.
ctx.register_parquet("trips", "s3://bucket/nyc-taxi/year=2025/")

# SQL API: familiar, portable, works well for ad-hoc queries.
sql_result = ctx.sql("""
    SELECT payment_type,
           COUNT(*)         AS trips,
           AVG(fare_amount) AS avg_fare
    FROM trips
    WHERE trip_distance BETWEEN 0.5 AND 30
    GROUP BY payment_type
    ORDER BY trips DESC
""").to_pandas()

# DataFrame API: composable, IDE-friendly, easier for programmatic building.
df_result = (
    ctx.table("trips")
       .filter((col("trip_distance") >= 0.5) & (col("trip_distance") <= 30))
       .aggregate(
           [col("payment_type")],
           [F.count(col("*")).alias("trips"),
            F.avg(col("fare_amount")).alias("avg_fare")],
       )
       .sort(col("trips").sort(ascending=False))
       .to_pandas()
)

Both queries return a pandas DataFrame in the snippet above, but to_pandas() is only one of several sinks. collect() returns a list of pyarrow.RecordBatch objects, to_polars() hands you a Polars DataFrame with zero copy, and execute_stream() returns an async iterator of Arrow batches that streams straight into downstream consumers. That last option matters when a query result is larger than memory, which is the same pattern I recommended in our pandas memory optimization guide for oversized joins.

DataFusion supports the ANSI SQL fundamentals plus a growing list of extensions: window functions (ROW_NUMBER, LAG, PERCENT_RANK), array and struct expressions, common table expressions (recursive as of 44), lateral joins, and UNNEST. The SQL reference in the official user guide is the source of truth and is refreshed with every release.

Reading Delta Lake and Iceberg with DataFusion

Lakehouse table formats (Delta Lake, Apache Iceberg, and Apache Hudi) sit above raw Parquet and add ACID transactions, schema evolution, and time travel. DataFusion is a natural query layer for them because both deltalake (via delta-rs) and pyiceberg expose scans as Arrow batches, and DataFusion consumes those batches directly.

from deltalake import DeltaTable
from datafusion import SessionContext

ctx = SessionContext()

dt = DeltaTable("s3://bucket/warehouse/orders")
# Register the current snapshot as a queryable table.
ctx.register_dataset("orders", dt.to_pyarrow_dataset())

top_customers = ctx.sql("""
    SELECT customer_id, SUM(total_amount) AS lifetime_value
    FROM orders
    WHERE order_date >= DATE '2026-01-01'
    GROUP BY customer_id
    ORDER BY lifetime_value DESC
    LIMIT 100
""").to_polars()

Because DeltaTable.to_pyarrow_dataset() pushes predicates and column projections down through Delta's metadata, DataFusion never sees Parquet files that the transaction log already ruled out. The equivalent Iceberg workflow uses pyiceberg.Table.scan() and its to_arrow() method; see our PyIceberg guide for the details on catalog configuration.

For teams that write to Delta from DataFusion (not just read), the delta-rs guide covers merge, time travel, and Z-ordering without a Spark cluster. The pattern of "query with DataFusion, mutate with delta-rs" is the one I reach for on lakehouse pipelines when I want everything Rust-backed and Arrow-native end-to-end.

Writing Python UDFs, UDAFs, and window functions

DataFusion supports three user-defined function categories from Python: scalar UDFs (per-row transformations), aggregate UDAFs (SUM-like reductions with state), and window UDWFs (partitioned running calculations). The important design detail is that UDFs receive PyArrow arrays, not per-row Python objects. The engine hands you a full record batch at a time, so a NumPy-friendly implementation stays fast.

import pyarrow as pa
import pyarrow.compute as pc
from datafusion import SessionContext, udf

# Scalar UDF: normalize free-text emails to lowercase and trim whitespace.
@udf(input_types=[pa.string()], return_type=pa.string(), volatility="immutable")
def normalize_email(email: pa.Array) -> pa.Array:
    return pc.utf8_lower(pc.utf8_trim_whitespace(email))

ctx = SessionContext()
ctx.register_udf(normalize_email)

ctx.sql("""
    SELECT normalize_email(email) AS email, COUNT(*) AS signups
    FROM read_parquet('s3://bucket/signups/2026/')
    GROUP BY 1
    ORDER BY signups DESC
""").show()

The volatility hint (immutable, stable, or volatile) matters. Immutable functions can be constant-folded, cached, and reordered. Marking a random-number UDF as immutable will produce silent correctness bugs. I hit this exact bug shipping a fraud-scoring pipeline last year, so treat volatility as a load-bearing annotation rather than a comment. The distinction mirrors the one described in the PostgreSQL function volatility documentation.

UDAFs and UDWFs use the same batch-oriented model but expose update, merge, and evaluate hooks so that partial aggregates can be combined across partitions. That's what makes them safe to use inside distributed plans, so every function you write against datafusion-python is also safe to run on Ballista or DataFusion-Ray.

Substrait: portable query plans between engines

Substrait is a cross-engine intermediate representation for relational algebra. Think of it as "LLVM IR for query plans." DataFusion is the reference Python implementation and can both produce a Substrait plan from a query and consume a plan built elsewhere. This is the piece that separates DataFusion from every other Python query engine in 2026: the same logical plan can be serialized in DataFusion, sent over the wire, and executed by Velox, DuckDB, or Ballista.

from datafusion import SessionContext
from datafusion.substrait import Serde, Producer, Consumer

ctx = SessionContext()
ctx.register_parquet("orders", "s3://bucket/orders/")

logical_plan = ctx.sql(
    "SELECT country, SUM(total) FROM orders GROUP BY country"
).logical_plan()

# Produce a Substrait protobuf blob you can persist or ship.
plan_bytes = Serde.serialize_bytes(Producer.to_substrait_plan(logical_plan, ctx))

# ... later, on another machine, possibly a different engine ...
restored = Serde.deserialize_bytes(plan_bytes)
df = Consumer.from_substrait_plan(ctx, restored)
df.show()

The obvious use case is portability. The less obvious one is testability: a Substrait plan is a stable, diffable artifact, so you can snapshot plans in unit tests and detect optimizer regressions in review. Combined with our earlier work on dbt unit testing, this gives you plan-level and result-level assertions on the same pipeline.

Scaling out with Ballista and Ray

DataFusion is single-node by default, but its physical plan is designed to be partitioned and distributed. Two mature paths exist in 2026. Ballista is an Apache-blessed distributed scheduler and executor built on top of DataFusion; it uses Arrow Flight to shuffle data between workers and has a Python client that mirrors SessionContext almost line for line. datafusion-ray integrates the engine with Ray, which is often the pragmatic choice for teams already invested in Ray Data or Ray Train.

# Ballista example: swap the local context for a scheduler URL.
from ballista import BallistaContext

ctx = BallistaContext.remote("df://scheduler.internal:50050")
ctx.register_parquet("events", "s3://bucket/events/year=2026/")

ctx.sql("""
    SELECT event_type, COUNT(*)
    FROM events
    GROUP BY event_type
""").to_pandas()

For orchestration around either path, our Airflow vs Prefect vs Dagster comparison covers which scheduler works cleanly with a DataFusion-based task graph. Ballista also exposes a REST endpoint that plays well with Dagster asset materializations.

When should you use DataFusion vs DuckDB or Polars?

Use DataFusion when you're building a tool that needs to embed a query engine, when you need Substrait, when you're distributing execution with Ballista or Ray, or when you plan to write custom optimizer rules or table providers. Use DuckDB when you want the widest SQL feature set with the least setup: CSV sniffing, JSON path expressions, spatial extensions, and vector search are all one-line installs. Use Polars when your pipeline lives inside a DataFrame paradigm and you value a Pythonic API with expressions and lazy evaluation.

These three engines share so much design DNA (columnar, vectorized, Arrow-friendly) that they're converging on similar performance for common workloads. In my benchmarks against 50 GB of Parquet on a single 32-core machine, DataFusion 46, DuckDB 1.3, and Polars 1.20 finish typical group-by-aggregate queries within about 10% of one another. Choosing between them is now much less about raw throughput and much more about ergonomics, extensibility, and where the query needs to run next.

Production patterns and pitfalls

So, a few habits I've converged on after shipping DataFusion in real pipelines. First, always pin datafusion, pyarrow, and deltalake together, because the trio must be ABI-compatible and the release notes always call out the tested combination. Second, prefer execute_stream() over to_pandas() for anything that might exceed memory; the streaming iterator emits Arrow batches you can pass straight to ParquetWriter, a Delta writer, or a Kafka producer without materializing the full result. Third, register your UDFs with explicit volatility. Fourth, keep an eye on the DataFusion changelog, since the SQL surface still grows measurably each release and features you missed six months ago are worth revisiting.

The most common pitfall I see is expecting DataFusion to behave like a database that owns its data. It doesn't. It's an execution engine over external Arrow-shaped sources. If you need transactional writes, put a lakehouse format underneath (Delta, Iceberg) and use the appropriate writer. If you need long-lived connections and sessions across processes, wrap DataFusion in an Arrow Flight SQL server; the Flight SQL specification is the standard way to expose it.

Finally, treat DataFusion the way you treat any other query engine: profile with realistic data volumes, snapshot plans when possible, and version-control the queries themselves. The engine will keep getting faster underneath you, and honestly, that's a nice property to have.

Frequently Asked Questions

Is DataFusion faster than DuckDB?

On typical single-node analytics queries over Parquet, DataFusion 46 and DuckDB 1.3 finish within roughly 10% of one another; neither is a categorical winner. DataFusion tends to shine on plans with heavy custom optimization or distributed shuffles via Ballista, while DuckDB tends to win on ad-hoc SQL with wide feature use because its dialect is broader.

Can DataFusion read Delta Lake and Iceberg tables?

Yes. Use deltalake.DeltaTable(...).to_pyarrow_dataset() for Delta and pyiceberg.Table.scan().to_arrow() for Iceberg, then register the Arrow dataset with SessionContext.register_dataset(). Predicate and column pushdown are handled by the respective client before DataFusion ever touches a Parquet file.

Does datafusion-python support GPU acceleration?

Not as of the 46.x series. DataFusion is a CPU-vectorized engine. If your workload is GPU-bound, cuDF or Theseus are the correct tools; DataFusion is the wrong layer for that.

How do I write a custom aggregate function in DataFusion Python?

Subclass datafusion.udf.Accumulator, implement update, merge, evaluate, and state, then register the accumulator with ctx.register_udaf. The batch-oriented interface is what makes the UDAF safe to use inside distributed plans on Ballista or Ray.

Is DataFusion production-ready in 2026?

Yes. DataFusion graduated to a top-level Apache project in 2024 and now powers InfluxDB 3.0, GreptimeDB, ROAPI, and the Comet accelerator for Apache Spark. The Python bindings follow the Rust crate's monthly release cadence and are used in production by multiple companies that publish case studies on the project's blog.