📖 Liberty Airflow

A Scalable and Extensible FastAPI and React Scheduler prebuilt with Airflow

Airflow Setup Guide

📌 Prerequisites

Before installing Airflow, ensure you have the following installed on your system:

• Python 3.12
• PostgreSQL (for database storage)
• Git (if needed for repository usage)

🔧 Step 1: Create a .env File

Airflow requires environment variables to be set. Create a .env file in the project root and configure it as follows:

# Airflow and Python Version
AIRFLOW_VERSION=3.0.1
PYTHON_VERSION=3.12
AIRFLOW_INIT=true

# PostgreSQL Server
POSTGRES_HOST=localhost
POSTGRES_PORT=5432

# PostgreSQL Airflow
POSTGRES_AIRFLOW_DB=airflow
POSTGRES_AIRFLOW_USER=airflow
POSTGRES_AIRFLOW_PASSWORD=your_password_here  # Replace with a secure password

# PostgreSQL Liberty
POSTGRES_LIBERTY_DB=liberty
POSTGRES_LIBERTY_USER=liberty
POSTGRES_LIBERTY_PASSWORD=your_password_here  # Replace with a secure password

# PostgreSQL Liberty Airflow Repository
POSTGRES_LIBNARF_DB=libnarf
POSTGRES_LIBNARF_USER=libnarf
POSTGRES_LIBNARF_PASSWORD=your_password_here  # Replace with a secure password

# PostgreSQL Liberty Airflow Data
POSTGRES_NOMAARF_DB=nomaarf
POSTGRES_NOMAARF_USER=nomaarf
POSTGRES_NOMAARF_PASSWORD=your_password_here  # Replace with a secure password

# Admin PostgreSQL Config
POSTGRES_ADMIN_DB=liberty
POSTGRES_ADMIN_USER=liberty
POSTGRES_ADMIN_PASSWORD=your_admin_password_here  # Replace with a secure password

# Airflow Config
AIRFLOW_HOME="./"
AIRFLOW__CORE__LOAD_EXAMPLES="False"
AIRFLOW__DATABASE__LOAD_DEFAULT_CONNECTIONS="False"
AIRFLOW__WEBSERVER__EXPOSE_CONFIG="True"
AIRFLOW__CORE__EXECUTOR="LocalExecutor"
AIRFLOW__API__PORT=8081 # Replace with your port
AIRFLOW__API__BASE_URL="http://localhost:8081/airflow" # Replace with your URL
PYTHONWARNINGS="ignore::SyntaxWarning"
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
AIRFLOW__API_AUTH__JWT_SECRET=your_secret_key_here  # Replace with a secure secret key
AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth
AIRFLOW__CORE__AUTH_MANAGER="airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager"
SCARF_ANALYTICS=false

# Default Airflow Admin User
AIRFLOW_ADMIN_USER=admin
AIRFLOW_ADMIN_EMAIL=admin@example.com
AIRFLOW_ADMIN_PASSWORD=your_admin_password_here  # Replace with a secure password
AIRFLOW_ADMIN_FIRSTNAME=Admin
AIRFLOW_ADMIN_LASTNAME=User

# FastAPI Config
FASTAPI_HOST="localhost"
FASTAPI_PORT=8082

⚠ IMPORTANT: Do not commit the .env file to version control. Keep credentials secure.

▶ Step 2: Install Liberty Airflow

pipx is recommended due to dependencies between liberty framework and airflow

pip install pipx
pipx ensurepath
pipx install liberty-airflow

This script will preconfigure the environment for Airflow installation

  installed package liberty-airflow, installed using Python 3.12.4
  These apps are now globally available
    - airflow-install
    - airflow-start
    - airflow-stop
    - liberty-airflow
done! ✨ 🌟 ✨

▶ Step 3: Install and configure Airflow

To install Airflow and set up the database, run:

airflow-install

This script will:

• Install Airflow and required dependencies
• Create the PostgreSQL database and user (if not already created)
• Initialize the Airflow database
• Create a default Airflow admin user

▶ Step 4: Start Liberty Airflow

To start Airflow services, run:

liberty-airflow

This will:

• Start the Airflow Scheduler
• Start the Airflow API Server
• Start the Airflow Dag Processor
• Start FastAPI

Once started, you can access at: 👉 Airflow UI: http://localhost:8081/airflow 👉 Frontend: http://localhost:8082 👉 API: http://localhost:8082/api

⏹ Step 5: Stop Airflow

To stop all running Airflow processes, run:

airflow-stop

This will:

• Stop the Airflow Scheduler
• Stop the Airflow API Server
• Stop the Airflow Dag Processor

🛠 Troubleshooting

• If Airflow does not start, check the logs in the logs/ directory.
• Ensure PostgreSQL is running and accessible

✅ Next Steps

• Configure DAGs inside $AIRFLOW_HOME/dags/.
• Add Git integration using the configured GIT_URL.
• Secure your instance by updating passwords and secret keys.

✅ Prebuilt Dags

Daily DAGs

• airflow-purge-daily-1: Purges old Airflow logs and metadata on a daily schedule (@daily).
• database-backup-daily-1: Backs up databases every day at 01:00 AM (00 1 * * *).

Weekly DAGs

• database-purge-weekly-1: Performs database cleanup and purging on a weekly schedule (@weekly).

Unscheduled DAGs

• airflow-sync-1: Synchronizes repositories as needed (manually triggered).

✅ Variables

The DAGs rely on Airflow Variables for configuration. These can be set in the Airflow UI (Admin -> Variables) or via the Airflow CLI.

✅ Connections

The DAGs require predefined Airflow Connections to interact with external systems. These can be configured via Airflow UI (Admin -> Connections).

Git Connection (git_conn)

• Connection Type: generic
• Description: Used for performing Git operations (pull, push, etc.)
• Required Fields: login, password, host

PostgreSQL Connection (liberty_conn)

• Connection Type: postgres
• Description: Database connection for Liberty Framework
• Required Fields: login, password, host, port, schema
• Defaults:
  • login: liberty
  • host: localhost
  • port: 5432

💖 Sponsorship

If you find Liberty Ariflow useful and would like to support its development, consider sponsoring us. Your contributions help maintain the project, add new features, and improve the documentation. Every contribution, big or small, is greatly appreciated!

To sponsor, visit: GitHub Sponsors or reach out to us directly.

📜 License

Liberty Airflow is open-source software licensed under the AGPL License.

📧 Contact & Support

If you have questions or need support:

• Email: franck.blettner@nomana-it.fr
• GitHub Issues: Report an issue
• Discussions: Join the conversation in the GitHub Discussions section.

⭐ If you find Liberty Airflow useful, consider giving it a star on GitHub!

git clone https://github.com/fblettner/liberty-airflow.git
cd liberty-framework

🚀 Let's build the future of business applications together! 🚀