airflow-dag-slot-booking-cc 0.0.1

Airflow plugin for DAG schedule slot booking — supports Airflow 2.x (Flask) and 3.x (FastAPI)

Airflow DAG Slot Booking Plugin

A visual scheduling plugin for Apache Airflow that shows which time slots are already used and which are available — through an interactive 7-day x 24-hour grid UI inside Airflow.

The Problem

When you have dozens (or hundreds) of DAGs running on Airflow, scheduling conflicts become a real issue:

• Multiple DAGs scheduled at the same time compete for workers, database connections, and API rate limits
• This leads to failures, timeouts, and slow runs
• There's no built-in way in Airflow to visualize which time slots are crowded and which are free
• Teams end up hardcoding cron schedules in DAG scripts without knowing what's already running at that time

---

Airflow Version Compatibility

Supports both Airflow 2.x and Airflow 3.x automatically — no configuration needed.

The plugin detects your Airflow version at startup and registers itself using the correct interface:

Airflow Version	Plugin Interface	URL
2.x (e.g. 2.8, 2.9, 2.10)	Flask Blueprint	http://<host>/dag_slot_booking
3.x (e.g. 3.0, 3.1+)	FastAPI app	http://<host>/dag_slot_booking/

You don't need to do anything differently — just install the package and restart Airflow. The right interface is chosen automatically.

Note for Airflow 3.x users: The URL has a trailing slash — use /dag_slot_booking/ not /dag_slot_booking.

---

Approach 1: View Available Slots (All Users — Zero Config)

Just install and open. The plugin reads your existing DAG schedules from Airflow's own metadata.

Step-by-step setup:

your-airflow-project/
├── dags/
│   ├── my_etl_dag.py          # your existing DAGs (no changes needed)
│   ├── my_ml_pipeline.py
│   └── ...
├── docker-compose.yml          # or Helm chart
└── requirements.txt            # add the plugin here

Step 1: Install the plugin

# Option A: Add to requirements.txt
echo "airflow-dag-slot-booking-cc" >> requirements.txt

# Option B: Add to docker-compose.yml env
_PIP_ADDITIONAL_REQUIREMENTS: "airflow-dag-slot-booking-cc"

# Option C: Add to Dockerfile
RUN pip install airflow-dag-slot-booking-cc

Step 2: Restart your Airflow webserver

# Docker Compose (Airflow 2.x)
docker-compose restart airflow-webserver

# Kubernetes (Airflow 2.x)
kubectl rollout restart deployment/airflow-webserver -n airflow

# Kubernetes (Airflow 3.x — pod is called api-server)
kubectl rollout restart deployment/airflow-api-server -n airflow

Step 3: Open the UI

# Airflow 2.x
http://<your-airflow-host>/dag_slot_booking

# Airflow 3.x (note the trailing slash)
http://<your-airflow-host>/dag_slot_booking/

Done. You'll see a grid showing all your DAGs' schedules. No database, no variables, no DAG changes needed.

Features:

• A 7-day x 24-hour grid showing every DAG's cron schedule
• Colored slots = occupied, gray slots = available
• Click any colored slot to see which DAGs are scheduled at that time
• All times in UTC (Airflow's default), hover for your local time
• Filter by day of week or DAG type
• Stats: total DAGs, full slots, available slots

Your existing DAGs stay the same:

# Nothing changes — your DAGs work as before
from airflow import DAG
from datetime import datetime

dag = DAG(
    'my_etl_pipeline',
    schedule_interval='0 3 * * 1',  # hardcoded is fine for this approach
    start_date=datetime(2024, 1, 1),
)

The plugin just reads these schedules and shows them visually. Use it to check what's free before adding a new DAG.

---

Approach 2: Book Slots from UI + Read Schedule from DB (Advanced)

This is how I use it in production with 150+ web scrapers. Instead of hardcoding cron schedules in DAG scripts, you store them in a database table and book new slots from the UI.

Why this approach is better:

• No more hardcoded schedules — DAG scripts read their cron from the DB at runtime
• No merge conflicts — team members book slots from the UI instead of editing DAG files
• Capacity control — prevents overloading a time slot (max 5 DAGs per slot)
• Visibility — everyone sees who booked what and when
• Dynamic — change a schedule in the UI, DAG picks it up automatically without code changes

Step-by-step setup:

your-airflow-project/
├── dags/
│   ├── my_batch_dag.py         # reads schedule from DB (see example below)
│   └── ...
├── docker-compose.yml
└── requirements.txt            # add: airflow-dag-slot-booking-cc, psycopg2-binary

Step 1: Install the plugin (same as Approach 1)

pip install airflow-dag-slot-booking-cc

Step 2: Create the schedule table in your PostgreSQL database

CREATE TABLE "YOUR_SCHEMA"."DAG_SCHEDULE_CONFIG" (
    "DAG_NAME"       VARCHAR(255),
    "SCHEDULE_TIME"  VARCHAR(100),   -- cron expression in UTC (e.g., '0 3 * * 1')
    "OWNER"          VARCHAR(255),   -- who scheduled this DAG
    "COMMAND_NAME"   VARCHAR(255),   -- optional: spider name, script name, etc.
    "TYPE"           VARCHAR(50),    -- e.g., 'Scrapy', 'Python', 'Spark', 'ETL'
    "BATCHES"        VARCHAR(255),   -- optional: batch group name
    "SELENIUM"       VARCHAR(10),    -- optional: 'TRUE' if browser-based
    "IS_ACTIVE"      VARCHAR(10)     -- 'TRUE' or 'FALSE'
);

Step 3: Create an Airflow Variable named EXTRACTION_CONFIG

Go to Airflow UI > Admin > Variables > + and add:

• Key: EXTRACTION_CONFIG
• Value (JSON):

{
  "DATABASE_HOST": "your-db-host",
  "DATABASE_USER": "your-db-user",
  "DATABASE_NAME": "your-db-name",
  "DATABASE_PASSWORD": "your-db-password",
  "DATABASE_PORT": "5432"
}

Step 4: Write your DAG script to read schedule from DB

Instead of hardcoding the schedule:

# OLD WAY — hardcoded, no visibility, merge conflicts
dag = DAG('my_spider', schedule_interval='0 3 * * 1')

Read it from the database:

# NEW WAY — schedule comes from DB, booked via the plugin UI
import psycopg2
from airflow import DAG
from airflow.models import Variable
from airflow.operators.bash import BashOperator
from datetime import datetime

config = Variable.get("EXTRACTION_CONFIG", deserialize_json=True)

def get_schedule(dag_name):
    """Read cron schedule from DAG_SCHEDULE_CONFIG table."""
    conn = psycopg2.connect(
        host=config["DATABASE_HOST"],
        user=config["DATABASE_USER"],
        dbname=config["DATABASE_NAME"],
        password=config["DATABASE_PASSWORD"],
        port=config["DATABASE_PORT"],
    )
    cursor = conn.cursor()
    cursor.execute("""
        SELECT "SCHEDULE_TIME"
        FROM "YOUR_SCHEMA"."DAG_SCHEDULE_CONFIG"
        WHERE "DAG_NAME" = %s AND "IS_ACTIVE" = 'TRUE'
    """, (dag_name,))
    row = cursor.fetchone()
    cursor.close()
    conn.close()
    return row[0] if row else None

schedule = get_schedule('my_spider')

if schedule:
    dag = DAG(
        'my_spider',
        schedule_interval=schedule,
        start_date=datetime(2024, 1, 1),
        catchup=False,
    )
    run_task = BashOperator(
        task_id='run_spider',
        bash_command='scrapy crawl my_spider',
        dag=dag,
    )

Step 5: Book slots from the UI

1. Open http://<your-airflow-host>/dag_slot_booking (or /dag_slot_booking/ on Airflow 3.x)
2. Click a free (gray) slot on the grid
3. Fill in: DAG name, owner, type
4. Click Schedule DAG
5. The schedule is saved to your DB table
6. Your DAG script picks it up on the next scheduler refresh — no code changes needed

The full workflow:

Team member opens plugin UI
        ↓
Sees the 7x24 grid — finds a free slot
        ↓
Clicks the free slot → booking form appears
        ↓
Fills in DAG name, owner, type → clicks "Schedule DAG"
        ↓
Schedule saved to DB table: DAG_SCHEDULE_CONFIG
        ↓
DAG script reads schedule from DB at runtime
        ↓
Airflow runs the DAG at the booked time

---

How the Plugin Auto-Detects the Mode

EXTRACTION_CONFIG variable	Mode	UI
Not set	Airflow Mode	Read-only grid (Approach 1)
Set with DB credentials	DB Mode	Grid + booking panel (Approach 2)

No code changes needed — the plugin switches automatically based on whether the variable exists.

---

Feature Comparison

Feature	Approach 1 (View Only)	Approach 2 (DB + Booking)
Visual 7x24 schedule grid	Yes	Yes
UTC times with local time on hover	Yes	Yes
Click slot to see DAG details	Yes	Yes
Filter by day / type	Yes	Yes
Live stats dashboard	Yes	Yes
Book new DAGs from UI	—	Yes
Slot capacity limits (max 5/slot)	—	Yes
Batch management	—	Yes
Cancel/deactivate DAGs	—	Yes
DAGs read schedule from DB	—	Yes

---

Real-World Example

I built this for my team's web scraping infrastructure:

• 150+ Scrapy spiders scheduled across the week
• Each spider is a DAG that reads its cron schedule from DAG_SCHEDULE_CONFIG
• Team members use the plugin UI to book free slots for new spiders
• Spiders are grouped into batches (max 5 per batch) for coordinated execution
• No one edits DAG files to change schedules — it's all done from the UI

Tech Stack

• Backend: FastAPI (Airflow 3.x) / Flask Blueprint (Airflow 2.x) — auto-detected at startup
• Frontend: Bootstrap 4, vanilla JavaScript
• Database: PostgreSQL via psycopg2 (optional, for Approach 2 only)
• Compatibility: Apache Airflow 2.x and 3.x, Python 3.8+

Author

Jeevini Manohar

License

MIT