Airflow 3 Configuration & Performance Tuning Guide

This is Part 3 of the Airflow 3 in Practice series. In the previous part, Setting Up a Cluster, you brought up the Scheduler, API server, DAG processor, Triggerer, and Worker. Now it's time to touch the knobs that decide how much work that cluster takes on, and where it stalls. Before moving on to the next part, The Right Way to Author DAGs, let's first tame the settings that — if set wrong — can paralyze the entire cluster.

Most Airflow performance problems aren't caused by "a slow server," but by "hitting a limit you didn't know about." If tasks are piling up in the queue while workers sit idle, it's almost always some concurrency limit somewhere choking the flow like a funnel. The goal of this article is to help you picture where those funnels are and how they interact, so that when you see a symptom you immediately know which knob to turn.

Performance tuning doesn't start with raising values — it starts with knowing where the bottleneck is.

Where Does Config Come From: Precedence First

A single Airflow setting can come from several sources, and if you don't know which one wins, you fall into the "I clearly changed it but nothing happened" trap. For the same key, precedence roughly follows this order (higher wins).

1. Environment variables AIRFLOW__SECTION__KEY (uppercase, with a double underscore __ as the separator)
2. The _CMD / _SECRET variants of environment variables (the value is fetched from a command's output or a secrets backend)
3. Values specified in the airflow.cfg file
4. Airflow's built-in defaults

The core rule is this: the parallelism key in the [core] section of airflow.cfg becomes AIRFLOW__CORE__PARALLELISM as an environment variable. Just uppercase the section name and key and join them with __.

# Exactly equivalent to the following two lines in airflow.cfg
# [core]
# parallelism = 64
# (this one takes precedence)
export AIRFLOW__CORE__PARALLELISM=64

For container deployments, the environment variable approach is effectively the standard. Don't bake airflow.cfg into the image; injecting AIRFLOW__... environment variables keeps per-environment (dev/stage/prod) branching clean.

If you're unsure which value is actually in effect at runtime, check it with a command. Giving airflow config get-value core parallelism a section and key tells you the value actually applied (the result after all precedence is resolved).

The Three Concurrency Tiers: The Funnels Where Work Gets Stuck

Airflow's concurrency isn't a single number but a series of funnels. Each tier throttles the flow independently, and the narrowest tier determines actual throughput. With CeleryExecutor, the worker-side limit adds one more tier.

The three core knobs are both defaults (config file/env var) and upper bounds, and you can throttle more tightly with per-DAG/per-task arguments (max_active_runs, max_active_tasks, and Pool). You can't widen beyond them — parallelism is the ceiling.

Here's a funnel diagram of how these limits gate, in sequence, before a single task actually runs.

The most common mistake here is the arithmetic across tiers not adding up. For example (these are illustrative numbers):

• 4 workers × worker_concurrency=16 = capacity to process 64 concurrent tasks, but
• if you're capped at parallelism=32 → half the workers are always idle.
• Conversely, if parallelism=256 but worker capacity is only 64 → the scheduler sees 256 as "running," but 192 are backlogged in the broker queue.

A rough starting point: set parallelism ≈ (number of workers × worker_concurrency), then adjust from there as you watch the load. The two numbers drifting apart is the most common mistuning.

Pool + priority_weight: Isolating Resources

max_active_tasks_per_dag is a knob that throttles "within a single DAG," but it's powerless when multiple DAGs hammer the same external resource. If 30 DAGs call the same external API at once, that API falls over first. This is where Pool comes in.

A Pool is a named bundle of slots. If you assign pool="external_api" to a task, the task waits in the queued state until a slot in that pool frees up. No matter which DAG it came from, all tasks sharing the same pool converge under a single limit.

from airflow.sdk import dag, task
import pendulum
 
 
@dag(schedule="@hourly", start_date=pendulum.datetime(2026, 1, 1), catchup=False)
def crm_sync():
    # The external CRM API only tolerates 5 concurrent requests → isolate with a dedicated pool
    @task(pool="external_api", pool_slots=1, priority_weight=10)
    def fetch_accounts():
        ...
 
    fetch_accounts()
 
 
crm_sync()

• pool_slots: You can make a heavy task grab 2 or more slots, so that instead of "one slot = one task," consumption scales with cost.
• priority_weight: When competing for a slot within the same pool, this decides who gets pulled first. The higher the number, the higher the priority. When a slot frees up, tasks with higher weight run first from the queue.

Here's the flow of how a Pool acts as a gate from request to execution, and where priority comes into play.

Tasks that call external systems (a DB, a payment gateway, an internal legacy API) should almost always go into a dedicated Pool. A Pool is the simplest safeguard for enforcing the promise "only N of this resource at a time" from outside the code. Deeper patterns for external system integration are covered in Part 8, External System Integration & Sync Calls.

DAG Parsing Performance: Giving the Scheduler Room to Breathe

In Airflow 3, DAG parsing is split off into an independent process called the DAG processor, so heavy parsing no longer directly steals from the scheduler's scheduling loop (for architectural background, see Part 1, Anatomy of the Architecture). Even so, if parsing is slow, new DAGs and changes are reflected late, and run creation is delayed.

But the highest-impact tuning isn't a setting — it's how you write the DAG code. Top-level code (code outside functions, executed right after import) runs every time the DAG file is parsed — that is, repeatedly at the intervals above. Do anything heavy there and the whole parse slows down.

# Bad: an external call runs at the top level on every parse
import requests
config = requests.get("https://config-server/limits").json()  # a network call on every parse!
 
 
# Good: keep heavy work inside the task (runs only once, at execution time)
from airflow.sdk import task
 
@task
def load_config():
    import requests
    return requests.get("https://config-server/limits").json()

Rule: don't do DB queries, API calls, or heavy imports at the top level of a DAG file. Do only the work of "defining the DAG," and put all the code that "does work" inside tasks. We dig deeper into this principle in Part 4, The Right Way to Author DAGs.

Metadata DB Connection Pool & Log Retention

The scheduler, DAG processor, and API server all hammer the metadata DB (in Airflow 3, workers/tasks don't connect directly to the DB but go through the Task Execution API, so DB connection pressure is mainly on the scheduler side). If the SQLAlchemy connection pool is too narrow, components stall waiting for a connection.

Caution: if you run multiple schedulers, connections multiply by the number of components. Make sure the DB's max_connections can handle (number of schedulers + DAG processor + API server) × (pool_size + max_overflow). Otherwise the DB rejects with "too many connections."

Logs, too, will fill the disk if left unattended. Airflow 3 has the airflow db clean command for cleaning up old metadata and logs, and the standard practice is to set a retention period and run it periodically.

# Clean up run/log metadata older than 90 days (run periodically via cron, etc.)
airflow db clean --clean-before-timestamp "2026-03-27 00:00:00+00:00"

A Checklist for Diagnosing Common Bottlenecks

This is a decision flow for going straight from a symptom to the knob to turn. It narrows down "where is it narrow?" from top to bottom.

To put it as a checklist you can run through quickly:

• Queue backlog + idle workers → check that parallelism isn't smaller than worker capacity. (parallelism ≈ number of workers × worker_concurrency)
• One specific pool always full → raise that pool's slots, or if it's truly a resource limit, leave it as is and use priority_weight to move urgent tasks up.
• Only one DAG slow as if serialized → check that the DAG's max_active_tasks / max_active_runs isn't pinned to 1 or some low value.
• New DAGs appear late / changes reflected slowly → removing heavy top-level code is the top priority, then check parsing_processes and min_file_process_interval.
• Scheduler CPU saturated / DB "too many connections" → verify the connection pool and DB max_connections are consistent.
• Long waits that could be made deferrable (sensors, etc.) occupying worker slots → move them to the Triggerer to free up slots (detailed patterns in Part 4).

The order of tuning is always observe → identify the narrowest funnel → one knob at a time. If you raise several values at once, you can't tell what made the difference, and soon something else (the DB or broker) becomes the new bottleneck.

Wrapping Up

Performance tuning in Airflow 3 isn't a single magic setting but the work of aligning a series of funnels. Know the config precedence (AIRFLOW__... environment variables > airflow.cfg), get the arithmetic of the three concurrency tiers and worker limits right, isolate external resources with Pools, empty out top-level code to keep parsing light, and manage DB connections and logs — and most "it's slow" problems are solved with a single knob.

Exact key names and defaults can change between versions, so before you finalize anything, double-check the official Airflow configuration reference once more. In the next part, The Right Way to Author DAGs, we'll take on in earnest what this article deferred: "how to empty out top-level code and how to structure tasks."