Apache Mesos Provider
Project description
Apache Mesos Provider for Apache Airflow 2.x
This provider for Apache Airflow contain the following features:
- MesosExecuter - A scheduler to run Airflow DAG's on mesos
- MesosOperator - To executer Airflow tasks on mesos. (TODO)
Requirements
- Airflow 2.x
- Apache Mesos minimum 1.6.x
How to install and configure
On the Airflow Server, we have to install the mesos provider.
pip install avmesos_airflow_provider
Then we will configure Airflow.
vim airflow.cfg
executor = avmesos_airflow_provider.executors.mesos_executor.MesosExecutor
[mesos]
master = leader.mesos:5050
framework_name = Airflow
checkpoint = True
failover_timeout = 604800
command_shell = True
task_cpu = 1
task_memory = 20000
authenticate = True
default_principal = <MESOS USER>
default_secret = <MESOS PASSWORD>
docker_image_slave = <AIRFLOW DOCKER IMAGE>
docker_volume_driver = local
docker_volume_dag_name = airflowdags
docker_volume_dag_container_path = /home/airflow/airflow/dags/
docker_sock = /var/run/docker.sock
docker_volume_logs_name = airflowlogs
docker_volume_logs_container_path = /home/airflow/airflow/logs/
docker_environment = '[{ "name":"<KEY>", "value":"<VALUE>" }, { ... }]'
DAG example with mesos executor
from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators.dummy_operator import DummyOperator
from airflow.providers.docker.operators.docker import DockerOperator
from airflow.operators.python import PythonOperator
default_args = {
'owner' : 'airflow',
'description' : 'Use of the DockerOperator',
'depend_on_past' : True,
}
with DAG('docker_dag2', default_args=default_args, schedule_interval="*/10 * * * * ", catchup=True, start_date=datetime.now()) as dag:
t2 = DockerOperator(
task_id='docker_command',
image='centos:latest',
api_version='auto',
auto_remove=False,
command="/bin/sleep 600",
docker_url='unix:///var/run/docker.sock',
executor_config={
"MesosExecutor": {
"cpus": 2.0,
"mem_limit": 2048
}
}
)
t2
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for avmesos_airflow_provider-0.1.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9eb75c0b914ed33b7ab0f7de741254ca9e540b9acc9af5bb0d521b2936cb904b |
|
MD5 | 95cb6472307b1216427246eb3cbf7a5c |
|
BLAKE2b-256 | 6b4a73770a1b86383c2d8652e7961705655bb182abc890e4fcf127cc57baf40a |
Close
Hashes for avmesos_airflow_provider-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9de8b511acaf185f8757b921d029c7eca6e2d9c62e78be1acf4ba1ee4abc4fce |
|
MD5 | 4dcbdb03c57380b9799e57cd17868295 |
|
BLAKE2b-256 | 0250b243ff7df478f52a6669dd5720d0dd3e5e01828f6f42e69958f4756c097b |