dlt in Python: A Practical Guide to the Data Load Tool (2026)

Updated: June 27, 2026

dlt (the data load tool) is an open-source Python library that turns any iterable of Python dicts (REST APIs, SQL queries, files) into typed, schema-evolving tables inside destinations like DuckDB, BigQuery, Snowflake, Postgres, and Iceberg, in roughly twenty lines of code. In 2026 the project hit version 1.x with a stable API, native Iceberg support, and first-class incremental loading. This guide is the practical, runnable tour I wish I had when I migrated three Airflow DAGs of brittle requests + pandas code over to dlt.

• dlt 1.x ships a stable Python API, native Apache Iceberg destination, and async-aware sources, making it a credible OSS alternative to Fivetran or Airbyte for many workloads.
• You write a Python generator that yields dicts; dlt infers the schema, normalizes nested JSON into relational tables, and writes Parquet or JSONL via DuckDB, Snowflake, BigQuery, Redshift, Postgres, or Iceberg.
• Incremental loading is a one-decorator change (@dlt.resource with dlt.sources.incremental), and dlt stores cursor state in the destination so pipelines stay resumable.
• Schema evolution is automatic and audited: new columns appear, type conflicts get logged in the _dlt_loads table, and you can pin contracts with schema_contract="freeze".
• For orchestration, dlt composes cleanly with Dagster, Airflow, and Prefect. The runtime is just a Python function call, no daemon required.
• Local-first development with DuckDB makes dlt one of the fastest libraries to iterate on; the same code targets a cloud warehouse with a one-line destination swap.

What is dlt in Python?

dlt is a declarative ELT framework that lives entirely inside your Python process. You define a source (a function that yields Python iterables of records), choose a destination (a warehouse, lake, or local database), then run pipeline.run(source). dlt handles type inference, JSON normalization, file staging, retries, and a transactional commit to the target. The project is maintained by dltHub and licensed under Apache 2.0; the official documentation is the canonical reference for new releases.

Conceptually, dlt sits between a hand-rolled requests + SQLAlchemy script and a heavyweight integration platform like Airbyte. It's not a UI tool, not a SaaS, and not a workflow engine. It's a library, and that narrower scope is exactly what makes it composable: you import it next to pandas or Polars, run it from a notebook, schedule it from Dagster, or invoke it inside an AWS Lambda. If you've read my earlier piece on choosing a Python data orchestrator in 2026, dlt is the loader those orchestrators were missing.

Architecturally, every dlt pipeline produces three artifacts in the destination: the user tables, a _dlt_pipeline_state table that records cursor positions for incremental sources, and a _dlt_loads + _dlt_version pair that audits every batch. That's what gives dlt its idempotency. Re-running the same job either picks up where it left off or replaces the failed load atomically.

Installation and project setup

dlt is distributed on PyPI with optional extras for each destination. Install the core library plus a destination of your choice. DuckDB is the friendliest for local development because it doesn't require any credentials.

# Python 3.10+ recommended (3.12 fully supported in 1.x)
python -m venv .venv && source .venv/bin/activate
pip install "dlt[duckdb]==1.4.*"

# Initialize a new project scaffold
dlt init github duckdb

The dlt init command writes a .dlt/ folder with two TOML files: config.toml for non-secret settings and secrets.toml for API keys and DSNs. Both follow a clear scoping rule. A key under [sources.github] is visible to any source named github; keys under [destination.bigquery] are visible to the BigQuery destination. In production I keep secrets.toml out of git and inject values via environment variables. dlt automatically reads SOURCES__GITHUB__ACCESS_TOKEN (double underscores as section separators).

One ergonomic detail worth knowing: dlt's CLI also exposes dlt deploy <pipeline.py> github-action, which scaffolds a CI workflow that runs your pipeline on a schedule. It's not a replacement for an orchestrator, but it's enough to ship a side-project pipeline in an afternoon.

Your first dlt pipeline in 15 lines

Here's the smallest useful dlt program. It pulls three records from a public API, normalizes a nested JSON structure into two relational tables, and writes them to a local DuckDB file.

import dlt

# A source is a Python function that yields dicts (or lists of dicts).
@dlt.resource(name="users", write_disposition="replace")
def users():
    yield [
        {"id": 1, "name": "Ada",   "addr": {"city": "London",  "zip": "EC1"}},
        {"id": 2, "name": "Linus", "addr": {"city": "Helsinki", "zip": "00100"}},
        {"id": 3, "name": "Grace", "addr": {"city": "NYC",     "zip": "10001"}},
    ]

pipeline = dlt.pipeline(
    pipeline_name="quickstart",
    destination="duckdb",
    dataset_name="demo",
)

load_info = pipeline.run(users)
print(load_info)

Run it with python quickstart.py and inspect the result with DuckDB:

import duckdb
con = duckdb.connect("quickstart.duckdb")
print(con.execute("SHOW TABLES").fetchall())
# [('_dlt_loads',), ('_dlt_pipeline_state',), ('_dlt_version',), ('users',), ('users__addr',)]

Notice users__addr: dlt flattened the nested addr object into a child table linked by _dlt_parent_id. This child-table normalization is configurable. You can pin it as a single column with max_table_nesting=0, or expand JSON arrays into separate rows. The default is sensible and matches how dbt models typically expect the data.

Loading from a REST API with pagination

Most real pipelines pull from a REST endpoint, and dlt's rest_api helper handles the boilerplate (pagination, retries, OAuth refresh) declaratively. Here's a complete loader for the GitHub issues endpoint that paginates by Link header and merges by issue ID:

import dlt
from dlt.sources.rest_api import rest_api_source

github = rest_api_source({
    "client": {
        "base_url": "https://api.github.com/repos/dlt-hub/dlt/",
        "auth": {
            "type": "bearer",
            "token": dlt.secrets["sources.github.access_token"],
        },
        "paginator": {"type": "header_link"},  # follows the Link: rel="next" header
    },
    "resource_defaults": {"primary_key": "id", "write_disposition": "merge"},
    "resources": [
        {
            "name": "issues",
            "endpoint": {
                "path": "issues",
                "params": {"state": "all", "per_page": 100},
            },
        },
        {
            "name": "comments",
            "endpoint": {"path": "issues/comments"},
        },
    ],
})

pipeline = dlt.pipeline("github_repo", destination="duckdb", dataset_name="gh")
print(pipeline.run(github))

What dlt does behind that config: it opens a connection pool with httpx, sends paginated requests, parses JSON, infers the schema on the first batch, writes Parquet files to a local staging folder, and finally executes a COPY into DuckDB inside a transaction. If the run fails halfway through, the _dlt_loads row is never committed, so re-running picks up clean. This transactional discipline is what makes dlt safe to schedule.

For non-REST APIs you write a generator yourself. The verified sources gallery ships ready-made loaders for Salesforce, HubSpot, Stripe, Notion, Google Sheets, and around forty others. Copy them into your project with dlt init <source> <destination>.

Incremental loading and cursor state

Full refreshes are fine until the source crosses a million rows. dlt's incremental primitive is one extra parameter:

import dlt
from dlt.sources.helpers.rest_client import RESTClient

@dlt.resource(primary_key="id", write_disposition="merge")
def orders(
    updated_at=dlt.sources.incremental("updated_at", initial_value="2026-01-01T00:00:00Z"),
):
    client = RESTClient(base_url="https://api.example.com")
    # Pass the watermark to the source if it supports it
    params = {"since": updated_at.last_value}
    for page in client.paginate("/orders", params=params):
        yield page

pipeline = dlt.pipeline("billing", destination="duckdb", dataset_name="ops")
pipeline.run(orders)

On the first run, dlt loads every order with updated_at >= initial_value and records the maximum updated_at it saw in _dlt_pipeline_state. On every subsequent run, that max becomes the new last_value. The cursor lives in the destination, not on the worker, so a fresh container picks up exactly where the previous one left off. No Redis, no Airflow XCom required.

Two patterns worth knowing. First, when the source doesn't let you filter server-side, set end_value and row_order="desc" so dlt stops iterating once it sees a record older than the cursor. Second, for late-arriving data, set allow_external_schedulers=True and pass a lag in seconds so the cursor walks back slightly on each run. I hit this exact bug shipping a Stripe webhook backfill that silently dropped four hours of refunds, and adding the lag was the one-line fix.

Schema evolution and contracts

dlt infers a destination schema from the first batch it sees and updates it as new fields appear. By default, adding a column is allowed, narrowing a type (e.g., int to float) is allowed, and dropping a column is ignored. That's the right policy for upstream APIs you don't own. Partners add fields without warning, and you'd rather the pipeline keep flowing than page you at 3 a.m.

When the destination is a production warehouse feeding dashboards, you usually want the opposite: fail fast. Pin a contract:

@dlt.resource(
    name="orders",
    schema_contract={"tables": "freeze", "columns": "evolve", "data_type": "freeze"},
)
def orders():
    ...

The four contract modes are evolve (default), freeze (raise on change), discard_row (skip non-conforming records), and discard_value (null out the offending value). You can mix levels (for example, let new columns appear but block type changes), which is the configuration I run for any table that downstream dbt models depend on. The audit trail lives in _dlt_version; running pipeline.schemas["orders"].to_pretty_yaml() prints the current contract.

This automatic-but-auditable approach is also why dlt pairs well with our earlier guide on data validation with Pandera. dlt handles the structural schema while Pandera enforces business-rule checks downstream.

Destinations: DuckDB, BigQuery, Snowflake, Iceberg

Swapping destinations is a one-line change because dlt abstracts the load semantics. Install the matching extra and update the destination kwarg.

Destination	Best for	Install	Load mechanism	Notes
DuckDB	Local dev, embedded analytics	dlt[duckdb]	Direct COPY from Parquet	Zero credentials. Fastest iteration loop.
BigQuery	Cloud-native lakehouse	dlt[bigquery]	GCS stage + load job	Supports clustering and partitioning hints.
Snowflake	Enterprise warehouse	dlt[snowflake]	Internal stage + COPY INTO	Use warehouse + role in secrets.toml.
Postgres	Operational store, smaller volumes	dlt[postgres]	Batched INSERT or COPY	Good for <100M rows; use psycopg3 driver.
Iceberg (PyIceberg)	Open lakehouse, multi-engine	dlt[pyiceberg]	Parquet + Iceberg metadata commit	New in 1.x. Works with Glue, REST, or Nessie catalogs.
Filesystem (S3/GCS/Azure)	Raw landing zone	dlt[filesystem]	Parquet/JSONL files	Pairs well with DuckDB or Polars downstream.

Iceberg is the most interesting addition in 1.x. If you already maintain Iceberg tables with tools like the ones I covered in the PyIceberg practical guide, dlt now writes directly into them, including schema-evolved commits. For teams committed to open table formats, this removes a major piece of glue code.

Is dlt better than Airbyte or Fivetran?

It depends on what you mean by "better." Airbyte and Fivetran are platforms. They ship a UI, a connector catalog with hundreds of sources, a managed scheduler, and (in Fivetran's case) a hosted runtime. dlt is a Python library. The right comparison isn't feature-for-feature; it's "where does the connector logic live and who owns it?"

Honestly, I reach for dlt when the source is a custom REST API I already query in Python, when the destination team prefers code-reviewed pipelines over UI-clicked ones, and when the volume doesn't justify a SaaS bill. I reach for Fivetran when someone wants Salesforce to Snowflake yesterday and is happy to pay for the ops. Airbyte sits in between: open-source, lots of connectors, but a heavier deployment footprint than a single Python script.

One concrete advantage of dlt that's easy to undervalue: the pipeline code lives in your repo, runs in your CI, and is debuggable with pdb. When a third-party API changes its pagination, you fix it in a pull request, not in a vendor's portal. For data teams that already maintain Python infrastructure, that ergonomic match is usually what tips the decision.

Deploying dlt with Dagster, Airflow, and Prefect

Because pipeline.run() is a synchronous function, scheduling dlt is whatever your orchestrator's "run a Python callable" primitive is. The packaging detail to get right is where dlt writes its working files. By default it uses ~/.dlt/pipelines/<name>, which on ephemeral containers vanishes between runs. Override it with pipelines_dir or by setting DLT_PIPELINES_DIR to a persistent volume.

A minimal Dagster asset wrapping the GitHub source above:

from dagster import asset
import dlt
from my_sources import github_source

@asset
def github_issues():
    pipeline = dlt.pipeline(
        pipeline_name="github_repo",
        destination="bigquery",
        dataset_name="gh",
        pipelines_dir="/var/dlt",  # persistent volume mount
    )
    load_info = pipeline.run(github_source())
    return load_info.asdict()

Dagster's dagster-dlt integration package goes further: it generates one Dagster asset per dlt resource so lineage shows up automatically in the UI. For Airflow, the equivalent is PythonOperator wrapping the same function. Prefect users can decorate the run function with @flow and call it a day. The dlt CLI's dlt deploy ... airflow-composer command will scaffold the DAG file if you prefer codegen.

Common errors and how to debug them

Three failure modes account for ~80% of the dlt issues I've triaged.

1. Mysterious "schema in destination is out of sync." This happens when you change a column type or rename a resource and the destination still holds the old schema. Fix: drop the dataset (pipeline.drop()) for dev, or run pipeline.sync_destination() to reconcile in production.

2. Cursor stuck at None after first run. If your records don't actually contain the incremental field, dlt can't advance the cursor. Inspect with pipeline.state["sources"] and verify the field name matches the JSON key (it's case-sensitive). For nested fields use the dotted-path syntax: dlt.sources.incremental("meta.updated_at").

3. load_id visible in dashboards. The _dlt_load_id column dlt adds to every row is useful for auditing but ugly in BI tools. Hide it with dlt.config["normalize.add_dlt_load_id"] = False, or filter it out in your dbt models. I prefer to keep it on in raw and project it away in staging models.

Frequently Asked Questions

What's the difference between dlt and dbt?

dlt is an EL tool, meaning Extract and Load. It moves data from a source into a destination. dbt is a T tool, meaning Transform. It runs SQL inside the destination to model that raw data. The two are designed to compose: dlt lands raw tables, dbt models them into marts. They overlap on zero features.

Does dlt support async sources?

Yes. As of dlt 0.5+, resource functions can be async def generators or use async for internally. dlt awaits them inside its own event loop. Pair this with httpx.AsyncClient for high-throughput REST scraping. I covered the underlying pattern in async ETL with httpx and asyncio.

Can dlt handle CDC (change data capture) from Postgres?

Partially. dlt 1.x ships a verified SQL source that supports incremental loads using a watermark column, which covers append-mostly tables. True logical-replication CDC with replays of UPDATE and DELETE is on the roadmap, but it's typically delegated to Debezium or pg_replicate today, then loaded into the destination via dlt's filesystem source.

How does dlt handle secrets in production?

dlt reads secrets from .dlt/secrets.toml, environment variables (double-underscore-scoped), or any Python provider you register, including AWS Secrets Manager, GCP Secret Manager, and HashiCorp Vault. In CI, the environment-variable form is simplest: set DESTINATION__BIGQUERY__CREDENTIALS to the JSON service account and dlt picks it up automatically.

Is dlt production-ready in 2026?

Yes. The 1.x release in early 2026 marks an API-stability commitment, and teams at PostHog, Hugging Face, and several Y Combinator-backed startups publicly run dlt at million-row-per-day scale. It's not a silver bullet for petabyte CDC, but for the 80% of pipelines that move JSON or SQL from A to B, it's a reasonable default choice.