airflow-python-sdk 1.0.1

Airflow API (Stable)

airflow-python-sdk

Overview

To facilitate management, Apache Airflow supports a range of REST API endpoints across its objects. This section provides an overview of the API design, methods, and supported use cases.

Most of the endpoints accept JSON as input and return JSON responses. This means that you must usually add the following headers to your request:

Content-type: application/json
Accept: application/json

Resources

The term resource refers to a single type of object in the Airflow metadata. An API is broken up by its endpoint's corresponding resource. The name of a resource is typically plural and expressed in camelCase. Example: dagRuns.

Resource names are used as part of endpoint URLs, as well as in API parameters and responses.

CRUD Operations

The platform supports Create, Read, Update, and Delete operations on most resources. You can review the standards for these operations and their standard parameters below.

Some endpoints have special behavior as exceptions.

Create

To create a resource, you typically submit an HTTP POST request with the resource's required metadata in the request body. The response returns a 201 Created response code upon success with the resource's metadata, including its internal id, in the response body.

Read

The HTTP GET request can be used to read a resource or to list a number of resources.

A resource's id can be submitted in the request parameters to read a specific resource. The response usually returns a 200 OK response code upon success, with the resource's metadata in the response body.

If a GET request does not include a specific resource id, it is treated as a list request. The response usually returns a 200 OK response code upon success, with an object containing a list of resources' metadata in the response body.

When reading resources, some common query parameters are usually available. e.g.:

v1/connections?limit=25&offset=25

Query Parameter	Type	Description
limit	integer	Maximum number of objects to fetch. Usually 25 by default
offset	integer	Offset after which to start returning objects. For use with limit query parameter.

Update

Updating a resource requires the resource id, and is typically done using an HTTP PATCH request, with the fields to modify in the request body. The response usually returns a 200 OK response code upon success, with information about the modified resource in the response body.

Delete

Deleting a resource requires the resource id and is typically executing via an HTTP DELETE request. The response usually returns a 204 No Content response code upon success.

Conventions

• Resource names are plural and expressed in camelCase.

• Names are consistent between URL parameter name and field name.

• Field names are in snake_case.

{
    \"name\": \"string\",
    \"slots\": 0,
    \"occupied_slots\": 0,
    \"used_slots\": 0,
    \"queued_slots\": 0,
    \"open_slots\": 0
}

Update Mask

Update mask is available as a query parameter in patch endpoints. It is used to notify the API which fields you want to update. Using update_mask makes it easier to update objects by helping the server know which fields to update in an object instead of updating all fields. The update request ignores any fields that aren't specified in the field mask, leaving them with their current values.

Example:

  resource = request.get('/resource/my-id').json()
  resource['my_field'] = 'new-value'
  request.patch('/resource/my-id?update_mask=my_field', data=json.dumps(resource))

Versioning and Endpoint Lifecycle

• API versioning is not synchronized to specific releases of the Apache Airflow.
• APIs are designed to be backward compatible.
• Any changes to the API will first go through a deprecation phase.

Summary of Changes

Airflow version	Description
v2.0	Initial release

Trying the API

You can use a third party client, such as curl, HTTPie, Postman or the Insomnia rest client to test the Apache Airflow API.

Note that you will need to pass credentials data.

For e.g., here is how to pause a DAG with curl, when basic authorization is used:

curl -X POST 'https://example.com/api/v1/dags/{dag_id}?update_mask=is_paused' \\
-H 'Content-Type: application/json' \\
--user \"username:password\" \\
-d '{
    \"is_paused\": true
}'

Using a graphical tool such as Postman or Insomnia, it is possible to import the API specifications directly:

1. Download the API specification by clicking the Download button at top of this document
2. Import the JSON specification in the graphical tool of your choice.

• In Postman, you can click the import button at the top
• With Insomnia, you can just drag-and-drop the file on the UI

Note that with Postman, you can also generate code snippets by selecting a request and clicking on the Code button.

Authentication

To be able to meet the requirements of many organizations, Airflow supports many authentication methods, and it is even possible to add your own method.

If you want to check which auth backend is currently set, you can use airflow config get-value api auth_backend command as in the example below.

$ airflow config get-value api auth_backend
airflow.api.auth.backend.basic_auth

The default is to deny all requests.

For details on configuring the authentication, see API Authorization.

Errors

We follow the error response format proposed in RFC 7807 also known as Problem Details for HTTP APIs. As with our normal API responses, your client must be prepared to gracefully handle additional members of the response.

Unauthenticated

This indicates that the request has not been applied because it lacks valid authentication credentials for the target resource. Please check that you have valid credentials.

PermissionDenied

This response means that the server understood the request but refuses to authorize it because it lacks sufficient rights to the resource. It happens when you do not have the necessary permission to execute the action you performed. You need to get the appropriate permissions in other to resolve this error.

BadRequest

This response means that the server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing). To resolve this, please ensure that your syntax is correct.

NotFound

This client error response indicates that the server cannot find the requested resource.

MethodNotAllowed

Indicates that the request method is known by the server but is not supported by the target resource.

NotAcceptable

The target resource does not have a current representation that would be acceptable to the user agent, according to the proactive negotiation header fields received in the request, and the server is unwilling to supply a default representation.

AlreadyExists

The request could not be completed due to a conflict with the current state of the target resource, meaning that the resource already exists

Unknown

This means that the server encountered an unexpected condition that prevented it from fulfilling the request.

This Python package is automatically generated by the OpenAPI Generator project based on the specs:

• API version: 1.0.0
• Package version: 1.0.1
• Build package: org.openapitools.codegen.languages.PythonClientCodegen For more information, please visit https://github.com/zachliu

Requirements.

Python >= 3.6

Installation & Usage

pip install

The python package is hosted on PyPI, you can install directly using:

pip install airflow-python-sdk

Then import the package:

import airflow_python_sdk

Setuptools

Install via Setuptools.

python setup.py install --user

(or sudo python setup.py install to install the package for all users)

Then import the package:

import airflow_python_sdk

Getting Started

Please follow the installation procedure and then run the following:

import time
import airflow_python_sdk
from pprint import pprint
from airflow_python_sdk.api import config_api
from airflow_python_sdk.model.config import Config
from airflow_python_sdk.model.error import Error
# Defining the host is optional and defaults to http://localhost/api/v1
# See configuration.py for a list of all supported configuration parameters.
configuration = airflow_python_sdk.Configuration(
    host = "http://localhost/api/v1"
)

# The client must configure the authentication and authorization parameters
# in accordance with the API server security policy.
# Examples for each auth method are provided below, use the example that
# satisfies your auth use case.

# Configure HTTP basic authorization: Basic
configuration = airflow_python_sdk.Configuration(
    host = "https://<your-airflow-2.0.0>/api/v1",
    username = 'YOUR_USERNAME',
    password = 'YOUR_PASSWORD'
)


# Enter a context with an instance of the API client
with airflow_python_sdk.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = config_api.ConfigApi(api_client)

    try:
        # Get current configuration
        api_response = api_instance.get_config()
        pprint(api_response)
    except airflow_python_sdk.ApiException as e:
        print("Exception when calling ConfigApi->get_config: %s\n" % e)

Documentation for API Endpoints

All URIs are relative to http://localhost/api/v1

Class	Method	HTTP request	Description
ConfigApi	get_config	GET /config	Get current configuration
ConnectionApi	delete_connection	DELETE /connections/{connection_id}	Delete a connection
ConnectionApi	get_connection	GET /connections/{connection_id}	Get a connection
ConnectionApi	get_connections	GET /connections	List connections
ConnectionApi	patch_connection	PATCH /connections/{connection_id}	Update a connection
ConnectionApi	post_connection	POST /connections	Create a connection
DAGApi	get_dag	GET /dags/{dag_id}	Get basic information about a DAG
DAGApi	get_dag_details	GET /dags/{dag_id}/details	Get a simplified representation of DAG
DAGApi	get_dag_source	GET /dagSources/{file_token}	Get a source code
DAGApi	get_dags	GET /dags	List DAGs
DAGApi	get_task	GET /dags/{dag_id}/tasks/{task_id}	Get simplified representation of a task
DAGApi	get_tasks	GET /dags/{dag_id}/tasks	Get tasks for DAG
DAGApi	patch_dag	PATCH /dags/{dag_id}	Update a DAG
DAGApi	post_clear_task_instances	POST /dags/{dag_id}/clearTaskInstances	Clear a set of task instances
DAGApi	post_set_task_instances_state	POST /dags/{dag_id}/updateTaskInstancesState	Set a state of task instances
DAGRunApi	delete_dag_run	DELETE /dags/{dag_id}/dagRuns/{dag_run_id}	Delete a DAG run
DAGRunApi	get_dag_run	GET /dags/{dag_id}/dagRuns/{dag_run_id}	Get a DAG run
DAGRunApi	get_dag_runs	GET /dags/{dag_id}/dagRuns	List DAG runs
DAGRunApi	get_dag_runs_batch	POST /dags/~/dagRuns/list	List DAG runs (batch)
DAGRunApi	post_dag_run	POST /dags/{dag_id}/dagRuns	Trigger a new DAG run
EventLogApi	get_event_log	GET /eventLogs/{event_log_id}	Get a log entry
EventLogApi	get_event_logs	GET /eventLogs	List log entries
ImportErrorApi	get_import_error	GET /importErrors/{import_error_id}	Get an import error
ImportErrorApi	get_import_errors	GET /importErrors	List import errors
MonitoringApi	get_health	GET /health	Get instance status
MonitoringApi	get_version	GET /version	Get version information
PoolApi	delete_pool	DELETE /pools/{pool_name}	Delete a pool
PoolApi	get_pool	GET /pools/{pool_name}	Get a pool
PoolApi	get_pools	GET /pools	List pools
PoolApi	patch_pool	PATCH /pools/{pool_name}	Update a pool
PoolApi	post_pool	POST /pools	Create a pool
TaskInstanceApi	get_extra_links	GET /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/links	List extra links
TaskInstanceApi	get_log	GET /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/logs/{task_try_number}	Get logs
TaskInstanceApi	get_task_instance	GET /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}	Get a task instance
TaskInstanceApi	get_task_instances	GET /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances	List task instances
TaskInstanceApi	get_task_instances_batch	POST /dags//dagRuns//taskInstances/list	List task instances (batch)
VariableApi	delete_variable	DELETE /variables/{variable_key}	Delete a variable
VariableApi	get_variable	GET /variables/{variable_key}	Get a variable
VariableApi	get_variables	GET /variables	List variables
VariableApi	patch_variable	PATCH /variables/{variable_key}	Update a variable
VariableApi	post_variables	POST /variables	Create a variable
XComApi	get_xcom_entries	GET /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/xcomEntries	List XCom entries
XComApi	get_xcom_entry	GET /dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/xcomEntries/{xcom_key}	Get an XCom entry

Documentation For Models

• ClassReference
• ClearTaskInstance
• CollectionInfo
• Color
• Config
• ConfigOption
• ConfigSection
• Connection
• ConnectionAllOf
• ConnectionCollection
• ConnectionCollectionAllOf
• ConnectionCollectionItem
• CronExpression
• DAG
• DAGCollection
• DAGCollectionAllOf
• DAGDetail
• DAGDetailAllOf
• DAGRun
• DAGRunCollection
• DAGRunCollectionAllOf
• DagState
• Error
• EventLog
• EventLogCollection
• EventLogCollectionAllOf
• ExtraLink
• ExtraLinkCollection
• HealthInfo
• HealthStatus
• ImportError
• ImportErrorCollection
• ImportErrorCollectionAllOf
• InlineResponse200
• InlineResponse2001
• ListDagRunsForm
• ListTaskInstanceForm
• MetadatabaseStatus
• Pool
• PoolCollection
• PoolCollectionAllOf
• RelativeDelta
• SLAMiss
• ScheduleInterval
• SchedulerStatus
• Tag
• Task
• TaskCollection
• TaskCollectionAllOf
• TaskExtraLinks
• TaskInstance
• TaskInstanceCollection
• TaskInstanceCollectionAllOf
• TaskInstanceReference
• TaskInstanceReferenceCollection
• TaskState
• TimeDelta
• TriggerRule
• UpdateTaskInstancesState
• Variable
• VariableAllOf
• VariableCollection
• VariableCollectionAllOf
• VariableCollectionItem
• VersionInfo
• WeightRule
• XCom
• XComAllOf
• XComCollection
• XComCollectionAllOf
• XComCollectionItem

Documentation For Authorization

Basic

• Type: HTTP basic authentication

Kerberos

Author

[e-mail]

Notes for Large OpenAPI documents

If the OpenAPI document is large, imports in airflow_python_sdk.apis and airflow_python_sdk.models may fail with a RecursionError indicating the maximum recursion limit has been exceeded. In that case, there are a couple of solutions:

Solution 1: Use specific imports for apis and models like:

• from airflow_python_sdk.api.default_api import DefaultApi
• from airflow_python_sdk.model.pet import Pet

Solution 1: Before importing the package, adjust the maximum recursion limit as shown below:

import sys
sys.setrecursionlimit(1500)
import airflow_python_sdk
from airflow_python_sdk.apis import *
from airflow_python_sdk.models import *