AutoGineer
Generated banner for Airflow Dag Patterns
Generated logo for Airflow Dag Patterns

Airflow Dag Patterns

Seth HobsonCommunityCreated by Seth Hobson

  • Skill
  • Guidelines
  • Cloud and DevOps
  • MIT
  • No issues found
Download ZIP

From the author

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.
— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to developing Apache Airflow data pipelines. It covers essential design principles such as idempotency, atomicity, and observability, while offering practical patterns for DAG construction, including the TaskFlow API and dynamic DAG generation. The content focuses on standardizing pipeline development, managing task dependencies, and implementing robust error handling and deployment strategies.

What it does

  • Defines core design principles for reliable DAG development
  • Provides patterns for TaskFlow API implementation
  • Demonstrates techniques for dynamic DAG generation
  • Outlines best practices for operators, sensors, and task dependencies
  • Offers guidance on testing and debugging DAG runs

When to use it

  • When designing new data pipeline architectures in Airflow
  • When standardizing DAG development across a team
  • When implementing complex task dependencies or branching logic
  • When refactoring existing pipelines for better maintainability
  • When setting up production-grade scheduling and monitoring

Advantages

  • Promotes consistent coding standards for data workflows
  • Reduces common pitfalls through established design patterns
  • Simplifies complex pipeline logic using modern Airflow features
  • Improves pipeline observability and error handling

Usage examples

“How do I implement a simple ETL pipeline using the TaskFlow API?”
The agent provides a code template using the @dag and @task decorators to define an ETL flow with automatic XCom data passing.
“What are the best practices for handling task retries in Airflow?”
The agent explains the use of default_args, including retry counts, exponential backoff, and retry delays to ensure pipeline resilience.
“How can I generate multiple similar DAGs without duplicating code?”
The agent provides a dynamic DAG factory pattern that iterates over a configuration list to instantiate multiple DAG objects programmatically.
“What should I avoid when writing Airflow DAGs?”
The agent lists specific anti-patterns such as hardcoding dates, using global state, or placing heavy logic directly inside the DAG file.

Install

  1. Download the skill as a ZIP:

    https://mp-staging.autogineer.com/agent-skills/wshobson/airflow-dag-patterns/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip airflow-dag-patterns.zip -d .agents/skills

Documentation by Seth Hobson

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

When to Use This Skill

  • Creating data pipeline orchestration with Airflow

  • Designing DAG structures and dependencies

  • Implementing custom operators and sensors

  • Testing Airflow DAGs locally

  • Setting up Airflow in production

  • Debugging failed DAG runs

Core Concepts

1. DAG Design Principles
PrincipleDescription
IdempotentRunning twice produces same result
AtomicTasks succeed or fail completely
IncrementalProcess only new/changed data
ObservableLogs, metrics, alerts at every step
2. Task Dependencies
# Linear
task1 >> task2 >> task3

# Fan-out
task1 >> [task2, task3, task4]

# Fan-in
[task1, task2, task3] >> task4

# Complex
task1 >> task2 >> task4
task1 >> task3 >> task4

Quick Start

# dags/example_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
    'retry_exponential_backoff': True,
    'max_retry_delay': timedelta(hours=1),
}

with DAG(
    dag_id='example_etl',
    default_args=default_args,
    description='Example ETL pipeline',
    schedule='0 6 * * *',  # Daily at 6 AM
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=['etl', 'example'],
    max_active_runs=1,
) as dag:

    start = EmptyOperator(task_id='start')

    def extract_data(**context):
        execution_date = context['ds']
        # Extract logic here
        return {'records': 1000}

    extract = PythonOperator(
        task_id='extract',
        python_callable=extract_data,
    )

    end = EmptyOperator(task_id='end')

    start >> extract >> end

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

  • Use TaskFlow API - Cleaner code, automatic XCom

  • Set timeouts - Prevent zombie tasks

  • Use mode='reschedule' - For sensors, free up workers

  • Test DAGs - Unit tests and integration tests

  • Idempotent tasks - Safe to retry

Don'ts

  • Don't use depends_on_past=True - Creates bottlenecks

  • Don't hardcode dates - Use {{ ds }} macros

  • Don't use global state - Tasks should be stateless

  • Don't skip catchup blindly - Understand implications

  • Don't put heavy logic in DAG file - Import from modules

Files

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Seth Hobson
Publisher
Seth Hobson (Community)
Source
wshobson/agents · plugins/data-engineering/skills/airflow-dag-patterns · commit 4bde316
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Seth Hobson.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download airflow-dag-patterns.zip