Apache Airflow

Apache Airflow is the most widely adopted open-source workflow orchestration platform, used by thousands of companies to programmatically author, schedule, and monitor data pipelines. In this Apache Airflow review, we examine how Airflow's Python-based DAG definitions, extensive operator library, and massive ecosystem make it the default choice for data engineering teams — and where its limitations push teams toward alternatives like Dagster, Prefect, and Kestra.

Overview

Apache Airflow was created at Airbnb in 2014 by Maxime Beauchemin (who also created Apache Superset) and became an Apache top-level project in 2019. It's now maintained by a community of 2,500+ contributors with 37,000+ GitHub stars, making it one of the most active open-source data projects in existence.

Airflow's core concept is the DAG (Directed Acyclic Graph) — a Python script that defines tasks and their dependencies. The scheduler executes tasks in the correct order, retries failures, and provides a web UI for monitoring. Airflow supports 1,000+ operators and hooks through provider packages, connecting to virtually every data system: databases (PostgreSQL, MySQL, SQL Server), cloud services (AWS, GCP, Azure), data warehouses (Snowflake, BigQuery, Redshift), and tools (dbt, Spark, Kubernetes).

Airflow 2.x (current generation) introduced the TaskFlow API for cleaner Python-native DAG authoring, a revamped scheduler with horizontal scaling, and improved security with fine-grained RBAC. Airflow 3.0 is in development with features like event-driven scheduling and improved data-aware scheduling.

Key Features and Architecture

Python-Based DAG Authoring

DAGs are defined as Python code, providing unlimited flexibility — loops, conditionals, dynamic task generation, and integration with any Python library. The TaskFlow API (Airflow 2.0+) simplifies common patterns with Python decorators: @task for task definitions and automatic XCom passing between tasks.

Scheduler and Executor Architecture

The scheduler parses DAG files, determines which tasks are ready to run, and dispatches them to an executor. Airflow supports multiple executors: LocalExecutor (single machine), CeleryExecutor (distributed workers via Redis/RabbitMQ), KubernetesExecutor (one pod per task), and CeleryKubernetesExecutor (hybrid). This flexibility allows Airflow to scale from a single laptop to thousands of concurrent tasks.

Web UI and Monitoring

The built-in web interface provides DAG visualization (graph and tree views), task instance logs, execution history, Gantt charts for performance analysis, and manual trigger/retry controls. The UI is functional for operations but not as polished as newer tools like Dagster's Dagit.

1,000+ Provider Packages

Airflow's provider ecosystem includes operators, hooks, and sensors for AWS (S3, EMR, Glue, Redshift, Lambda), GCP (BigQuery, Dataflow, GCS, Composer), Azure (Blob Storage, Data Factory, Synapse), databases (PostgreSQL, MySQL, MongoDB, Cassandra), and tools (dbt, Spark, Kubernetes, Docker, Slack, PagerDuty). This is the largest operator ecosystem of any orchestration tool.

Connections and Variables Management

Centralized management of database connections, API credentials, and configuration variables through the UI or CLI. Connections support secret backends (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) for production-grade credential management.

Data-Aware Scheduling (Datasets)

Airflow 2.4+ introduced Datasets — a mechanism for DAGs to declare which datasets they produce and consume. When a producing DAG updates a dataset, consuming DAGs are automatically triggered. This enables event-driven pipeline orchestration without external triggers.

Ideal Use Cases

ETL/ELT Pipeline Orchestration

The most common use case: scheduling and monitoring data pipelines that extract data from sources, transform it, and load it into warehouses. Airflow orchestrates the sequence of dbt runs, Spark jobs, API calls, and SQL queries that make up a modern data pipeline.

ML Pipeline Scheduling

Data science teams use Airflow to schedule model training, feature engineering, and batch inference jobs. The KubernetesExecutor is particularly useful for ML workloads that need GPU instances for training tasks but CPU instances for data preparation.

Cross-System Workflow Automation

Organizations automate complex workflows that span multiple systems — triggering a Snowflake query after an S3 file lands, then sending results to Slack and updating a dashboard. Airflow's operator library connects these systems without custom integration code.

Data Quality and Monitoring Pipelines

Teams schedule data quality checks (Great Expectations, Soda, dbt tests) as Airflow tasks, gating downstream processing on quality validation. Sensor operators wait for upstream data to arrive before triggering dependent pipelines.

Pricing and Licensing

Apache Airflow is free and open-source under the Apache 2.0 license. Managed offerings:

Self-hosted Airflow requires a metadata database (PostgreSQL recommended), a message broker (Redis or RabbitMQ for CeleryExecutor), the webserver, scheduler, and worker processes. A production-grade setup with CeleryExecutor on AWS typically costs $500–$1,500/month in infrastructure. Managed services eliminate operational overhead at a premium.

Pros and Cons

Pros

• Largest ecosystem — 1,000+ operators, 2,500+ contributors, 37,000+ GitHub stars; the most battle-tested orchestration platform
• Unlimited flexibility — Python-based DAGs can express any workflow logic; no limitations from visual or YAML-based approaches
• Multiple managed options — Astronomer, MWAA, Cloud Composer, and Azure all offer managed Airflow, reducing operational burden
• Massive job market — Airflow experience is the most requested orchestration skill in data engineering job postings
• Data-aware scheduling — Datasets feature enables event-driven orchestration without external triggers
• Active development — Airflow 3.0 in progress with significant improvements to scheduling and developer experience

Cons

• Operational complexity — self-hosted Airflow requires managing scheduler, webserver, workers, database, and message broker; upgrades can be painful
• DAG complexity at scale — large Airflow deployments with 500+ DAGs suffer from slow DAG parsing, scheduler bottlenecks, and difficult debugging
• No built-in data lineage — Airflow tracks task dependencies but doesn't natively understand data lineage; requires integration with OpenLineage or external catalogs
• Testing is difficult — unit testing DAGs requires mocking Airflow's execution context; no built-in testing framework comparable to Dagster's
• Scheduler latency — the file-based DAG parsing model introduces latency between code changes and scheduler recognition; not ideal for rapid iteration
• Legacy patterns — many tutorials and existing DAGs use outdated Airflow 1.x patterns; the codebase carries significant backward compatibility burden

Alternatives and How It Compares

Dagster

Dagster is the strongest challenger to Airflow, offering a software-engineering-first approach with typed inputs/outputs, built-in testing, asset-based orchestration, and a polished UI (Dagit). Dagster Cloud starts at $0 (free tier) with paid plans from $100/month. Dagster is better for teams that value testability and data lineage; Airflow wins on ecosystem breadth and job market demand.

Prefect

Prefect provides Python-native orchestration with a simpler API than Airflow — flows are regular Python functions decorated with @flow and @task. Prefect Cloud offers a generous free tier (10,000 task runs/month). Prefect is easier to learn than Airflow but has a smaller operator ecosystem and community.

Kestra

Kestra is a declarative orchestration platform using YAML workflow definitions instead of Python code. It's designed for broader teams (not just Python developers) and offers a visual workflow editor. Kestra is newer and less proven than Airflow but simpler for teams that prefer configuration over code.

dbt Cloud

dbt Cloud provides scheduling and orchestration specifically for dbt transformations. It's not a general-purpose orchestrator — it only runs dbt jobs. Teams often use dbt Cloud for transformation scheduling and Airflow for everything else, or replace dbt Cloud with Airflow-triggered dbt runs.

AWS Step Functions

Step Functions is AWS's serverless workflow orchestrator using JSON state machine definitions. It's tightly integrated with AWS services and requires no infrastructure management. Step Functions is simpler than Airflow for AWS-only workflows but lacks Airflow's flexibility, Python expressiveness, and multi-cloud support.