Cron Is Not a Dependency Graph

What breaks when a general-purpose scheduler runs your dbt project

Most data platforms don't arrive at a scheduling problem. They grow into one.

It starts reasonably. You have a dbt project and something needs to run it, so you write a cron job. Then ingestion arrives — a managed connector here, a self-hosted open-source one there — and some of those tools have their own scheduler, so you let them use it. Then someone points out that transformations shouldn't run before the data lands, so the cron job becomes a workflow: sync, wait, then dbt. Then a second cadence appears, because a few tables need to be fresher than the rest. Then a third, for the heavy models overnight.

Two years later you have a handful of workflow definitions on different cron expressions, a dbt project of several hundred models whose refresh cadence is encoded in tags, ingestion connections split between "scheduled by the workflow" and "scheduled by the tool", and a Slack channel that tells you a workflow failed but not what broke.

Nothing here is a mistake, exactly. Each step was the obvious next one. But the result is a platform where the orchestrator does not understand the thing it is orchestrating — and that gap is where the operational pain lives.

This is a description of that pain, generalised from platforms we've assessed: a few dozen source systems feeding a few hundred streams through more than one ingestion tool, several hundred dbt models spread across two warehouses, and a small set of workflows running on tiered refresh cadences. Every name below is invented. The shape will be familiar.

What the workflow actually says

Here is the hourly workflow, sanitised. It is Argo Workflows, but the shape is the same in Airflow, in a Jenkins job, or in a bash script under cron.

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: warehouse-hourly
spec:
  schedule: "0 * * * *"                      # (1)
  workflowSpec:
    entrypoint: sync-and-transform
    onExit: notify-slack                     # (2)
    templates:
      - name: sync-and-transform
        dag:
          tasks:
            - name: trigger-source-sync
              template: ingestion-sync
              arguments:
                parameters:
                  - name: connection-id
                    value: "<connection-id>"  # (3)

            - name: wait-for-sync
              template: poll-ingestion-job    # (4)
              dependencies: [trigger-source-sync]

            - name: dbt-transform
              template: dbt-build
              dependencies: [wait-for-sync]
              arguments:
                parameters:
                  - name: selector
                    value: "tag:hourly"       # (5)

      - name: dbt-build
        inputs:
          parameters:
            - name: selector
        container:
          image: <registry>/dbt-runner:latest
          command: ["dbt"]
          args: ["build", "--select", "{{inputs.parameters.selector}}"]  # (6)

Six lines, six problems.

(1) The schedule is a guess about the world. 0 * * * * says the pipeline starts at the top of the hour. It says nothing about whether the source system has anything new, whether last hour's run has finished, or whether this hour's run has any reason to exist. When ingestion is slow and the hour rolls over, you get two runs of the same pipeline racing each other into the same tables.

(2) The failure signal is one bit wide. The exit handler posts to Slack when the workflow fails. That message says warehouse-hourly failed. It cannot say which of the couple of hundred models in the hourly tag failed, because the thing that failed — from the workflow's point of view — is a container that returned a non-zero exit code.

(3) Connection IDs are the dependency graph. The relationship between "this ingestion connection" and "the dbt models that read the tables it populates" exists only in an engineer's head, and in the tag on line (5). Nothing validates it. Nothing updates it when someone adds a model.

(4) wait-for-sync is where the honesty lives. This step polls the ingestion API until the sync reports done — which is the good version. The common version is sleep 900. Either way, the seam between ingestion and transformation is a piece of glue code whose job is to guess when the data is ready, because the two systems have no shared vocabulary for "this table has new rows in it".

(5) tag:hourly is a scheduling language pretending to be metadata. More on this below. It is the single largest source of quiet breakage in setups like this.

(6) dbt build is an opaque blob. This is the heart of it, and it deserves its own section.

The orchestrator cannot see inside dbt

When that container runs, dbt does an enormous amount of work that the orchestrator never learns about.

dbt parses the project and produces manifest.json — a complete, machine-readable dependency graph of every model, source, seed, snapshot and test, derived from the ref() and source() calls in the SQL. dbt already knows that mart_revenue depends on stg_orders, which depends on source('sales', 'orders'). It knows this precisely and automatically.

It then runs, and produces run_results.json — per-node outcomes. Model built, in this many seconds, this many rows affected. Test passed. Test failed. Model skipped because its parent failed.

Then the pod exits, and both files are deleted.

What the orchestrator retains is: 0 or 1.

The consequences compound:

• Retries are all-or-nothing. One model fails at minute 40 of a 45-minute run. The retry re-runs everything, including the 380 models that succeeded, because nothing recorded which ones those were.

• Partial failure is invisible downstream. dbt correctly skips the children of a failed model. But the workflow either fails (and you don't know what still ran fine) or, if the step is configured to tolerate errors, succeeds — and now stale tables sit in the warehouse with no marker on them.

• Tests are log output. dbt tests are the best data quality tooling most teams already own. Run this way, a failing test is a line in a container log that scrolled past at 03:14. There is no per-table history of "this uniqueness test has failed four times this month".

• There is no state to defer against. dbt supports comparing against a previous run's artifacts to build only what changed. That requires somewhere durable to put the artifacts. A pod's filesystem is not that place.

An orchestrator that read those two JSON files would know everything it needs. Most general-purpose schedulers never look, because they were built to run tasks, and a task is the wrong unit.

Tags are a hand-maintained copy of a graph you already have

This is the failure mode that costs the most and is noticed the least.

Because the workflow can only select models in bulk, cadence has to be expressed as a dbt tag: hourly, four_hourly, daily. Those tags become the real orchestration configuration. And they are maintained by hand, in dbt_project.yml and in model configs, in parallel with a dependency graph dbt derives automatically.

Three things go wrong.

Models fall through the gaps. A new model is added and nobody tags it. It is not in any workflow's selector, so it never runs. There is no error — it simply isn't there. Somebody notices weeks later when a dashboard is wrong.

Cadence gets promoted, and drags everything with it. One stakeholder needs one mart fresher, so the model is retagged hourly. Its four upstream models must be retagged too, or the hourly run builds it from stale parents. Now those four run 24 times a day instead of once, and the warehouse bill moves.

Mixed-cadence models have no correct answer. A model reading one source that syncs hourly and another that syncs daily has to be tagged one or the other. Tag it hourly and it rebuilds 23 times a day against a daily parent that hasn't moved. Tag it daily and the hourly freshness on the first source is thrown away. There is no tag that means "run when either parent actually changes", because the workflow has no concept of a parent changing.

Meanwhile dbt has known the true graph the whole time. It's in the manifest. Nobody is reading it.

Two schedulers, no clock

The other half of the problem sits upstream.

Some ingestion connections are triggered by the workflow. Others were configured with the ingestion tool's own native schedule, usually because it was easier at the time. Both are legitimate. Together they mean there is no single answer to "when does this table get updated" — you check the workflow, then you check the tool, and sometimes both are true and the connection syncs twice.

And with two ingestion tools in play, the seam multiplies. Alerts arrive in three places: connector failures from the ingestion tools, test failures buried in dbt logs, and workflow failures from the orchestrator. Three notification channels describing one pipeline, none of which can tell you the thing you actually want to know, which is: is the table I'm about to trust correct right now?

Then somebody asks for a backfill of one month, and there is nothing to do but run dbt by hand with a date variable and watch it.

What the debugging session actually looks like

Slack says warehouse-daily failed at 03:12.

You open the workflow UI, find the failed step, open the pod logs, scroll several thousand lines to find the first Database Error, identify the model, and then — the expensive part — work out by hand what depends on that model, which of those things are now stale, which dashboards read them, and who needs telling. That last part is not written down anywhere. It's a person, remembering.

Everything needed to answer it exists in the dbt manifest. It was in the pod, and the pod is gone.

What would actually fix this

Not a better cron expression. The gap is conceptual: the orchestrator's unit of work is a task, and the thing your business depends on is a table.

An orchestrator that understood this platform would need to:

1. Treat every table as a first-class object — ingestion streams and dbt models alike — with an identity that persists across runs, so "when was this last updated, by what, and did its tests pass" is a question with an answer.

2. Read dbt's own artifacts. Build the graph from manifest.json rather than from tags, and unpack run_results.json so one dbt invocation reports several hundred outcomes instead of one exit code.

3. Put ingestion and transformation in the same graph, so the seam between them is a real dependency rather than a polling loop.

4. Schedule declaratively — express when data should be fresh and what depends on what, and let the system work out execution order, instead of encoding that order in wall-clock offsets a human has to keep aligned.

5. Make tests belong to tables, with history, so data quality is a property of an object rather than a line in a log.

6. Keep state outside the pod, so retries are surgical and lineage survives the run.

That set of requirements has a name — asset-oriented orchestration — and it is what tools like Dagster are built around. The next post in this series works through what that model looks like in practice: what replaces the tags, what replaces the wait-for-sync step, and what you give up in exchange (there is a real trade, and it involves losing the comfort of knowing exactly when things run).

But the requirements come first. If your platform has a step whose job is to guess how long ingestion takes, and a set of dbt tags that duplicate a dependency graph dbt already computes, you have this problem — whichever scheduler you're running.

The patterns above are drawn from orchestration work across a number of client engagements — the specifics differ every time, the failure modes rarely do. MetaOps designs, builds and stabilises open-source data platforms: Airbyte, dbt, Dagster and Kubernetes, deployed as code with Terraform, Flux and GitOps, and shipped through CI/CD pipelines that make a platform change a reviewable pull request rather than an evening's work. If any of the above sounded like a description of your Tuesday, get in touch.