Last Tuesday a teammate pinged me in Slack: "Our fct_events model has been running 47 minutes every night since we upgraded dbt-core to 1.9. It used to take 90 seconds. Nothing changed." Nothing ever "changed," of course. I pulled up the run, opened the compiled SQL, and within five minutes had the answer: dbt was silently doing a full refresh on every run, scanning 2.3 billion rows of BigQuery data, because somebody had added a column upstream and the on_schema_change default had quietly kicked in.
If you searched for "dbt incremental model not working," you are almost certainly looking at the same class of bug. The model still runs. It still materializes. The numbers in the target table are even correct. But dbt is rebuilding the entire table every time, and your warehouse bill is climbing while you sleep. I have hit this on Snowflake, BigQuery, and Redshift in the last 18 months, and the failure modes rhyme. Below are the five I keep seeing in 2026, in roughly decreasing order of frequency, plus the exact debug recipe I now run before I touch any model config.
How I confirm it is actually doing a full refresh
Before blaming a config, I want hard evidence. dbt's logs are friendly but lie by omission — a model labeled "incremental" in dbt run output can still be rebuilding from scratch. Three signals together convince me:
- The compiled SQL at
target/compiled/<project>/<model>.sql contains a CREATE OR REPLACE TABLE statement instead of a MERGE or INSERT.
- The
run_results.json at target/run_results.json has the model's adapter_response showing rows affected equal to the whole table size, not the delta.
- The warehouse query history (Snowflake
QUERY_HISTORY, BigQuery INFORMATION_SCHEMA.JOBS, Redshift STL_QUERY) shows a CREATE TABLE AS rather than a MERGE INTO.
Here is the one-liner I use to grep compiled output on a CI runner before I escalate anything:
dbt compile --select fct_events
grep -E "create or replace table|merge into" \
target/compiled/analytics/models/marts/fct_events.sql
If the first match wins, you are doing a full refresh, full stop. The rest of this article is about why.
Cause 1: unique_key mismatch (the most common by far)
The unique_key config is how dbt decides whether a row in the staging delta already exists in the target. On Snowflake and BigQuery this turns into the ON clause of a MERGE. If the column you name does not actually uniquely identify a row — or, worse, does not exist with that exact name in the target table — dbt-core 1.8+ will not raise an error during compile. It will compile a MERGE that matches zero rows, then either insert duplicates (with the insert_overwrite strategy on partitioned tables, this can silently rebuild the partition) or fall back depending on the adapter.
I have seen three flavors of this in production:
- Composite key passed as a string:
unique_key='order_id, line_item_id' is interpreted by some adapters as a single column literally named order_id, line_item_id. The correct form in modern dbt is a list: unique_key=['order_id', 'line_item_id']. See the official incremental models docs for the syntax matrix per adapter.
- Case sensitivity on Snowflake:
unique_key='OrderId' against a target column quoted as "OrderId" works; against an unquoted column folded to ORDERID it silently matches zero rows.
- Surrogate key recomputed each run: if you build the key with
dbt_utils.generate_surrogate_key() and one of the inputs is a timestamp that changes (like _loaded_at), every row looks new every run.
The fix is to actually test the merge predicate by hand. I copy the compiled SQL into a scratch query and replace the MERGE with a SELECT COUNT(*) on the join condition:
SELECT COUNT(*) AS matched_rows
FROM analytics.fct_events AS target
INNER JOIN analytics_dbt_tmp.fct_events__dbt_tmp AS source
ON target.event_id = source.event_id;
If matched_rows is zero on a non-first run, your unique_key is wrong. Period.
Cause 2: on_schema_change triggering a full rebuild
This is the one that bit my teammate. Since dbt-core 1.0 the default for on_schema_change is ignore, but a lot of teams have set it to sync_all_columns at the project level in dbt_project.yml because somebody read a blog post saying it was the "safe" choice. It is not.
With sync_all_columns, the moment an upstream model adds, removes, or changes a column type, dbt issues an ALTER TABLE on Snowflake and BigQuery — or, on adapters that cannot ALTER easily (Redshift's column drops are notoriously expensive), it falls back to a full refresh. The on_schema_change documentation spells out the per-adapter behavior, but in my experience the failure is almost always silent: the run succeeds, the model is "incremental," and the warehouse bill triples.
To debug, check both your project-level and model-level configs:
# dbt_project.yml — check for this
models:
analytics:
+on_schema_change: sync_all_columns # the culprit
# models/marts/fct_events.sql — and this
{{ config(
materialized='incremental',
unique_key='event_id',
on_schema_change='append_new_columns' # safer default
) }}
My rule of thumb in 2026: use append_new_columns for analytical models and fail for anything feeding a downstream reverse-ETL pipeline. sync_all_columns is fine only when you have alerts on warehouse spend that will catch the rebuild within a few hours.
Cause 3: partition pruning is silently disabled
On BigQuery and Databricks, the insert_overwrite strategy needs a partition_by config and a partitions list (or a dynamic predicate via partitions_to_replace) to actually prune. Skip either and dbt will overwrite every partition every run — the SQL succeeds, the row counts look right, but every partition gets scanned.
A real example from a project I audited in March 2026: the team had this config in a BigQuery model storing 14 months of clickstream data partitioned by day. The model claimed it was incremental, but the BigQuery job history showed 4.2 TB scanned per run.
# models/marts/fct_clicks.sql
{{
config(
materialized='incremental',
incremental_strategy='insert_overwrite',
partition_by={'field': 'event_date', 'data_type': 'date'},
# MISSING: partitions = [...] or partitions_to_replace
)
}}
SELECT * FROM {{ ref('stg_clicks') }}
{% if is_incremental() %}
WHERE event_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 2 DAY)
{% endif %}
The WHERE clause looks correct — and it does limit the source scan — but without telling dbt which partitions to replace, the insert_overwrite macro generates SQL that nukes and rebuilds every partition. The fix is to set partitions explicitly:
{% set partitions_to_replace = [
'date_sub(current_date(), interval 1 day)',
'current_date()'
] %}
{{
config(
materialized='incremental',
incremental_strategy='insert_overwrite',
partition_by={'field': 'event_date', 'data_type': 'date'},
partitions=partitions_to_replace
)
}}
After this change, scan dropped from 4.2 TB to 8 GB. Same data, same model, 500x cost reduction. The BigQuery storage best practices page is worth re-reading if you have not in a year — partition elimination rules have been clarified since 2024.
Cause 4: is_incremental() returns false because of state
The is_incremental() macro only returns true when all of the following are met: the target table already exists, dbt is not running with --full-refresh, and the materialization is set to incremental. The third condition is obvious. The first two are where I see teams get tripped up:
- CI environments rebuild the schema every run, so
is_incremental() is always false in CI. If you only test against CI, you will never catch incremental bugs.
- Somebody pinned
--full-refresh in a cron wrapper as a "just in case" safety net during a migration two years ago and nobody removed it. Grep your scheduler configs.
- Blue/green deployment swapped the schema, so the target table technically does not exist under the name dbt is checking. dbt-core 1.7 introduced clone-based blue/green that handles this, but custom implementations often do not.
To verify, run with verbose logging and look for the literal string "is_incremental(): True":
dbt --debug run --select fct_events 2>&1 | grep -i incremental
If you see is_incremental(): False on a run where you expect True, work backward from the three preconditions above.
Cause 5: the incremental predicate is wider than you think
This one is subtle. Your WHERE filter inside {% if is_incremental() %} determines how much source data dbt reads to compute the delta — but it does not affect the merge predicate. So if you write:
{% if is_incremental() %}
WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})
{% endif %}
and your MAX(updated_at) is NULL (empty table) or in the distant past (one bad row from 1970), the predicate becomes effectively WHERE updated_at > NULL (which returns no rows) or WHERE updated_at > '1970-01-01' (which returns everything). On Snowflake, the optimizer cannot prune anything from a subquery against the target, so the source table is fully scanned even when the predicate looks tight.
The defensive version I now write everywhere uses COALESCE and a hardcoded floor:
{% if is_incremental() %}
WHERE updated_at > COALESCE(
(SELECT MAX(updated_at) FROM {{ this }}),
'2024-01-01'::timestamp
)
AND updated_at >= DATEADD(day, -3, CURRENT_TIMESTAMP) -- safety window
{% endif %}
The second predicate is the safety belt — it caps the worst case at 3 days of source scan even if the subquery returns garbage. If you have late-arriving data beyond that window, widen it, but always cap it. I learned this the hard way during a Snowflake credit incident in late 2024 that cost roughly $4,200 in a single overnight run.
The reproducible debug recipe I run every time
When a model is suspected of silent full-refresh behavior, I now follow the same checklist before I touch any config:
- Run
dbt compile --select <model> and grep the output for create or replace versus merge into.
- Open
target/run_results.json and check rows_affected in adapter_response. Compare to the source delta you expect.
- Query the warehouse query history for the actual SQL statement type executed.
- Run
dbt --debug run --select <model> and confirm is_incremental(): True appears in logs.
- Manually run the merge predicate as a
SELECT COUNT(*) against the staging and target tables to confirm rows actually match.
- Check
dbt_project.yml for project-level +on_schema_change overrides.
- If on BigQuery or Databricks with
insert_overwrite, verify partitions is set.
This takes about ten minutes and catches all five causes above. If you want a deeper dive on how dbt's adapter macros actually generate the MERGE statements, the dbt-snowflake adapter source is surprisingly readable — I keep a tab on the get_merge_sql macro open whenever I am debugging this stuff. We covered related warehouse cost patterns in our Snowflake credit spike debug checklist, and the broader testing strategy for incremental models lives in dbt test coverage for incremental models.
FAQ
Why does my dbt incremental model work locally but not in production?
Almost always because your local run is against an empty schema, so is_incremental() returns false and the model does a full build that happens to be fast on small dev data. In production with billions of rows, the same logic path runs but is now slow. Test by manually seeding your dev target table with one row before running.
Does dbt warn me when my incremental model silently full-refreshes?
No, and this is the single biggest UX gap in dbt-core as of 1.9. There is an open feature request to surface a warning when the materialization falls back, but until it ships you have to check yourself. I have a CI job that fails the build if create or replace table appears in any compiled incremental model.
Should I use merge or delete+insert as my incremental strategy?
On Snowflake and BigQuery, merge is almost always faster because the warehouse can use its built-in MERGE primitives. On Redshift, delete+insert can win because Redshift's MERGE landed late and is still less optimized. Always benchmark on your actual data — the answer can flip when your delta exceeds about 10% of the target table.
What is the safest default config for a new incremental model in 2026?
I use this template now: materialized='incremental', unique_key=['id'] (always a list), on_schema_change='append_new_columns', incremental_strategy='merge', plus a hardcoded safety window in the WHERE clause. Pin nothing else until you have a reason.
Closing
The pattern across all five causes is the same: dbt is forgiving by design, and that forgiveness translates into silent fallbacks that look like success in your logs but bleed money in your warehouse. The fix is not a single config change — it is making the failure mode visible. Add the compile-output grep to your CI. Add the is_incremental() log check to your model template. Audit on_schema_change across your project once a quarter. The 47-minute model that used to take 90 seconds is almost never a mystery once you know where to look.
