Anatomy of Airflow 3 — Components and the Task Execution API

This is Part 1 of the Airflow 3 in Practice series. If the previous part, An Overview of Airflow 3 & When to Use It, addressed "why 3," this part goes one level deeper to dissect what fits together and how it all turns. Before the next part, Building It as a Cluster, places these components onto actual nodes, the goal here is to first sketch a map in your head.

People running Airflow for the first time often say, "I thought one scheduler ran the whole thing." In 3, though, the picture splits fairly cleanly. If we name just the single biggest change up front, it's this.

In Airflow 3, workers (tasks) no longer connect directly to the metadata DB. All state reads and writes go through the Task Execution API on the API server.

We'll explain why this one line matters later in the article — first, let's look at the whole picture.

1. The Core Components at a Glance

Airflow 3 is made up of six pieces, each with a clear role. The key point is that the metadata DB is the source of truth, and surprisingly few components touch that DB directly.

Here's what each piece does.

If you remember 2.x, it'll feel odd not to see a "webserver" here. In Airflow 3, the webserver has been replaced by the API server. The UI and REST were merged into a single process, and the Task Execution API was layered on top.

Why the DAG processor was split off

In 2.x, the scheduler also handled DAG parsing. The problem is that a DAG file is arbitrary Python code. If someone does a heavy import inside a DAG, or calls an external API at the top level of a module, that burden flowed straight into the scheduler.

In 3, the DAG processor is split into an independent process. As a result, (1) parsing no longer blocks the scheduling loop, (2) arbitrary execution of DAG code is isolated outside the scheduler's trust boundary, and (3) where DAGs are fetched from is defined separately via DAG bundles (git and the like). In short, two different jobs — "parsing" and "scheduling" — are now handled by two different processes.

2. The Task Execution API — the Most Important Change

Now let's look at the change we previewed at the top. In 2.x, workers attached directly to the metadata DB to read and write task state. Over a long operational run, this creates two pain points.

• Security: Every worker has to hold metadata DB credentials. If a single worker is compromised, the entire DB is exposed. With remote or edge workers it's even worse — you'd have to open the DB port to the outside.
• Coupling: Task code is tied to the metadata DB schema and SQLAlchemy models, making it hard to write tasks in languages other than Python.

Airflow 3 inserts a thin HTTP layer in between, called the Task Execution API. Workers talk only to this API, through the Task SDK.

# Airflow 3: DAGs/tasks are imported from airflow.sdk
from airflow.sdk import dag, task
 
@dag(schedule="@daily", catchup=False)  # note that the catchup default changed to False
def etl_pipeline():
    @task
    def extract():
        return {"rows": 1000}
 
    @task
    def transform(payload: dict):
        # The worker running this code knows nothing about the metadata DB.
        # All communication — XCom push/pull, status reporting, etc. — flows
        # from the Task SDK to the Task Execution API.
        return payload["rows"] * 2
 
    transform(extract())
 
etl_pipeline()

The key takeaways.

In one sentence: the worker is now an "API client" that knows nothing about the DB, which makes remote execution and security isolation feel natural.

This change is precisely the foundation that makes the EdgeExecutor in the next section possible. Since workers talk over HTTP rather than to the DB, it no longer matters where outside the data center you place a worker.

3. The Executor — Where Tasks Actually Run

The Executor is the part that decides "where and how to actually run the tasks the scheduler placed on the queue." The representative choices available in Airflow 3 are as follows.

The EdgeExecutor is a new option built on the foundation of the Task Execution API. Because workers communicate with the control plane over HTTP only, you can place workers in places like beyond a firewall, in another region, or at an on-prem site.

And one more practical change in 3 — you can run multiple Executors at once (hybrid). For example, you can take ordinary lightweight tasks on Celery while sending heavy tasks that need a GPU to Kubernetes, splitting where tasks run on a per-task basis. Which workloads map to which Executor is covered, along with the actual configuration, in the next part, Cluster Configuration.

4. The State Transitions of a Single Task

Now that we've seen the components and the Executor, let's look at the states a single task passes through over its lifetime — the journey from the moment the scheduler decides "this is good to run" until it finishes.

Two things worth noting here.

• up_for_retry: Even when a task fails, if retries remain it doesn't die immediately but moves into a retry-waiting state. When the wait ends, it returns to scheduled and gets queued again.
• deferred: A deferrable operator (for example, a sensor waiting for an external job to finish) doesn't hold onto a worker slot to poll. Instead, it hands off to the Triggerer with "wake me when this condition is met" and drops into the deferred state. When the condition is satisfied, the Triggerer sends a signal and the task returns to scheduled. Since it doesn't occupy a worker slot, this saves a great deal of resources in large-scale waiting workloads.

5. The Lifecycle of a Single DAG Run

Now let's put the pieces on a timeline and follow the entire process of one DAG run executing.

Walking through it in order.

1. Parsing — The DAG processor reads the DAG source (DAG bundle), parses it, and writes the structure to the metadata DB.
2. Scheduling decision — The Scheduler looks at the metadata DB and decides which runs and tasks to execute.
3. Queuing — Runnable tasks are placed on the Executor queue (queued).
4. Assignment & execution — The Executor hands the task to a worker, and the worker runs the code (running).
5. State recording — The worker reports start, progress, and completion entirely through the Task Execution API, and the API server reflects that into the metadata DB.

In 2.x, where the worker would have written to the metadata DB directly at step 5, in 3 the decisive difference is that the API server steps in between and receives and handles all writes. Thanks to this, the worker doesn't need to know the DB at all, and the security boundary gathers cleanly inside the control plane.

Wrapping Up

If we summarize the architecture of Airflow 3 in one sentence, it's this.

The metadata DB is the source of truth; only the control plane (Scheduler, API server, DAG processor, Triggerer) touches that DB; and the worker has become a client that talks only through the Task Execution API.

Thanks to this structure, DAG parsing doesn't block scheduling, remote and edge workers feel natural, and you can mix multiple Executors in a single cluster. Once you have this map drawn in your head, it's time to place the components onto actual nodes.

In the next part, Building an Airflow 3 Cluster, we'll continue with concrete configurations — how to deploy these components separately and which Executor combinations to choose.

For deeper material, see the official Apache Airflow documentation.