Apache Airflow Fundamentals

Last Updated: July 2026 | 11 min read

📚 Lesson 1 of the Orchestration & Transformation track in our free Learn Data Engineering course. This is how the pipelines you've built get scheduled and connected.

Quick Answer: Apache Airflow is an open-source platform to orchestrate data pipelines — author, schedule, and monitor workflows as code. You define a workflow as a DAG (Directed Acyclic Graph) in Python: a set of tasks with dependencies and no loops. Airflow's scheduler works out what's ready to run, an executor runs it, and the platform retries failures, tracks state, and shows everything in a UI. It's the difference between a pile of cron jobs and a coordinated, observable pipeline.

Once you have more than a couple of pipelines — especially with dependencies like "load only after transform succeeds" — plain cron falls apart. Airflow is the industry-standard answer. This lesson gives you the core concepts (DAGs, tasks, operators, dependencies, scheduling) and a real DAG you can read, so orchestration stops being a black box.

What is Apache Airflow?

Airflow is a workflow orchestrator: it decides what runs, in what order, when, and what to do when something fails. You describe pipelines as Python code, and Airflow turns that into scheduled, monitored, retryable runs.

Airflow exists because ETL pipelines rarely run in isolation. Real data platforms have dozens of tasks with dependencies, schedules, and failure-recovery needs. Airflow coordinates all of it, and gives you a UI to see the state of every run.

What is a DAG?

A DAG — Directed Acyclic Graph — is Airflow's model of a workflow: tasks connected by one-way dependencies, with no cycles. The name encodes two guarantees:

• Directed — each dependency points one way (extract runs before transform).
• Acyclic — no loops, so there's always a clear start and finish.

In the diagram, extract → transform → [load_warehouse, load_ml] → notify is a DAG. Airflow reads it and knows exactly what can run now and what must wait.

Tasks, operators, and dependencies

A task is one unit of work; an operator is the template that defines what that work does; dependencies define the order. These three concepts are the whole vocabulary.

• Operators — the what. PythonOperator runs a function, BashOperator runs a shell command, provider operators run SQL, Spark, or cloud jobs.
• Tasks — an operator instance in a DAG becomes a task node with its own retries and state.
• Dependencies — the order, set with the >> operator: a >> b means "b runs after a."

Here's the DAG from the diagram, in code:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

with DAG(
    dag_id="daily_orders",
    schedule="@daily",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args={"retries": 2},
) as dag:

    extract   = PythonOperator(task_id="extract",   python_callable=do_extract)
    transform = PythonOperator(task_id="transform", python_callable=do_transform)
    load_wh   = PythonOperator(task_id="load_warehouse", python_callable=load_warehouse)
    load_ml   = PythonOperator(task_id="load_ml",   python_callable=load_ml)
    notify    = PythonOperator(task_id="notify",    python_callable=send_alert)

    # dependencies: the shape of the DAG
    extract >> transform >> [load_wh, load_ml] >> notify

That last line is the whole workflow: run extract, then transform, then the two loads in parallel, then notify once both finish.

Scheduling and retries: what Airflow gives you

Airflow doesn't just run tasks — it schedules them, retries failures, and remembers what already ran. This is what separates it from cron:

• Scheduling — schedule="@daily" (or a cron expression) tells Airflow when to trigger a DAG run. Validate cron expressions with our free cron tester.
• Retries — retries=2 re-runs a failed task automatically before giving up.
• Backfill & catchup — Airflow can run a DAG for past dates it missed (catchup).
• State tracking — every task run is success, failed, running, or retrying, visible in the UI.
• Alerting — email or Slack on failure, so you know before your users do.

The scheduler + executor model: the scheduler continuously checks which tasks are ready (schedule met, upstream tasks succeeded) and queues them; an executor (Local, Celery, or Kubernetes) runs them on workers. You write DAGs; Airflow runs the machinery.

Airflow vs cron: why orchestration matters

Cron runs commands on a clock; Airflow runs pipelines with dependencies, recovery, and visibility. The comparison makes the value obvious:

A single nightly script? Cron is fine. A pipeline where load depends on transform depends on extract, across many datasets? That's exactly what Airflow was built for.

Try it yourself — read and design a DAG

1. Given a >> b >> c, if task b fails and its retries are exhausted, does c run? Why?
2. Write the dependency line for: run clean, then run report_sales and report_finance in parallel, then email.
3. You need a pipeline to run at 6 AM every day. What schedule value would you set?

Common mistakes with Airflow

• Putting heavy work directly in the DAG file. DAG files are parsed constantly by the scheduler — keep them lightweight and push real work into tasks/operators.
• Using Airflow to process data instead of orchestrate it. Airflow triggers jobs (Spark, SQL, dbt); it's not a data-processing engine itself.
• Forgetting catchup=False. Leaving catchup on can unexpectedly backfill months of runs the first time you deploy a DAG.
• No retries or alerting. A pipeline that fails silently at 3 AM is worse than no pipeline. Set retries and failure alerts.
• Creating cycles or hidden dependencies. A DAG must be acyclic; circular dependencies are invalid and break the graph.

Frequently Asked Questions

What is Apache Airflow?

Apache Airflow is an open-source platform for orchestrating data pipelines — authoring, scheduling, and monitoring workflows as code. You define workflows as DAGs in Python, and Airflow schedules the tasks, runs them in order, retries failures, and shows their status in a web UI. It's the most widely used orchestrator for coordinating ETL pipelines with dependencies.

What is a DAG in Airflow?

A DAG (Directed Acyclic Graph) is how Airflow represents a workflow: tasks with directional dependencies and no cycles. "Directed" means each dependency points one way (A before B); "acyclic" means no loops, so there's a clear start and end. You define a DAG in Python, and Airflow uses it to decide what to run and in what order.

What is the difference between Airflow and cron?

Cron runs a command on a time schedule with no awareness of dependencies, failures, or state. Airflow is a full orchestrator: it manages dependencies (run C only after A and B succeed), retries failures, backfills missed runs, tracks every run's state, and provides monitoring through a UI. Use cron for a single independent job; Airflow for many tasks with dependencies.

What is an operator in Airflow?

An operator is a template for a single task — it defines what one unit of work does. PythonOperator runs a Python function, BashOperator runs a shell command, and provider operators run SQL, trigger Spark, or call cloud services. You instantiate operators inside a DAG, and each becomes a task node with its own retries and state.

How does the Airflow scheduler work?

The scheduler continuously reads your DAG definitions, determines which task instances are ready based on schedule and upstream dependencies, and queues them. An executor (Local, Celery, or Kubernetes) runs the queued tasks on workers. The scheduler tracks each task's state — success, failed, running, retrying — so the pipeline advances correctly and failures are handled.

Conclusion

Airflow turns a pile of scripts into a coordinated pipeline: define tasks and their dependencies as a DAG in Python, and Airflow schedules them, runs them in order, retries failures, and shows you everything in a UI. Master DAGs, tasks, operators, and the >> dependency syntax, and you can orchestrate real data platforms — the layer that ties every other track in this course together.

Next in the Orchestration track of the free Learn Data Engineering course: dbt vs Airflow — how the two tools differ and where each fits. Standing up orchestration for your pipelines now? Get matched with a vetted data engineer on SolutionGigs — it's free to post a project.

Mohammed Yaseen

Founder, SolutionGigs

Mohammed orchestrates production data pipelines with Airflow and teaches the Orchestration track of the free Learn Data Engineering course. LinkedIn →