Archived

This project has been archived by its maintainers, and is no longer receiving any updates.

⚠️ Discontinuation of project

This project is no longer actively maintained by Astronomer, but we'd love to see it live on in the community. It has been deprecated since May 2025. Astronomer is not accepting new contributions, bug fixes or releases, but the code is still here for you to explore, fork and adapt under the terms of its license. Please note that it may not work with the latest dependencies or platforms, and it could contain security vulnerabilities. Astronomer can't offer guarantees or warranties for its use. If you're interested in adopting or stewarding this project, we'd be happy to chat, reach us at oss@astronomer.io. Thanks for being part of the open-source journey and helping keep great ideas alive!

astro

workflows made easy

Astro Python SDK is a Python SDK for rapid development of extract, transform, and load workflows in Apache Airflow. It allows you to express your workflows as a set of data dependencies without having to worry about ordering and tasks. The Astro Python SDK was maintained by Astronomer.

Prerequisites

Apache Airflow >= 2.1.0.

Install

The Astro Python SDK is available at PyPI. Use the standard Python installation tools.

To install a cloud-agnostic version of the SDK, run:

pip install astro-sdk-python

You can also install dependencies for using the SDK with popular cloud providers:

pip install astro-sdk-python[amazon,google,snowflake,postgres]

Quickstart

Ensure that your Airflow environment is set up correctly by running the following commands:
```
export AIRFLOW_HOME=`pwd`
export AIRFLOW__CORE__XCOM_BACKEND=astro.custom_backend.astro_custom_backend.AstroCustomXcomBackend
export AIRFLOW__ASTRO_SDK__STORE_DATA_LOCAL_DEV=true
airflow db init
```
Note: AIRFLOW__CORE__ENABLE_XCOM_PICKLING no longer needs to be enabled for astro-sdk-python. This functionality is now deprecated as our custom xcom backend handles serialization.

The AIRFLOW__ASTRO_SDK__STORE_DATA_LOCAL_DEV should only be used for local development. The XCom backend docs give further details about how to set this up in non-local environments.

Currently, custom XCom backends are limited to data types that are json serializable. Since Dataframes are not json serializable, we need to enable XCom pickling to store dataframes.

The data format used by pickle is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can’t represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.

Read more: enable_xcom_pickling and pickle:

Create a SQLite database for the example to run with:

# The sqlite_default connection has different host for MAC vs. Linux
export SQL_TABLE_NAME=`airflow connections get sqlite_default -o yaml | grep host | awk '{print $2}'`
sqlite3 "$SQL_TABLE_NAME" "VACUUM;"

Copy the following workflow into a file named calculate_popular_movies.py and add it to the dags directory of your Airflow project:

https://github.com/astronomer/astro-sdk/blob/d5aa768b2d4bca72ef98f8d533fe3f99624b172f/example_dags/calculate_popular_movies.py#L1-L37

Alternatively, you can download calculate_popular_movies.py
```
 curl -O https://raw.githubusercontent.com/astronomer/astro-sdk/main/python-sdk/example_dags/calculate_popular_movies.py
```

Run the example DAG:

airflow dags test calculate_popular_movies `date -Iseconds`

Check the result of your DAG by running:

sqlite3 "$SQL_TABLE_NAME" "select * from top_animation;" ".exit"

You should see the following output:

$ sqlite3 "$SQL_TABLE_NAME" "select * from top_animation;" ".exit"
Toy Story 3 (2010)|8.3
Inside Out (2015)|8.2
How to Train Your Dragon (2010)|8.1
Zootopia (2016)|8.1
How to Train Your Dragon 2 (2014)|7.9

Supported technologies

Databases
Databricks Delta
Google BigQuery
Postgres
Snowflake
SQLite
Amazon Redshift
Microsoft SQL
DuckDB

File types
CSV
JSON
NDJSON
Parquet

File stores
Amazon S3
Filesystem
Google GCS
Google Drive
SFTP
FTP
Azure WASB
Azure WASBS

Available operations

The following are some key functions available in the SDK:

load_file: Load a given file into a SQL table
transform: Applies a SQL select statement to a source table and saves the result to a destination table
drop_table: Drops a SQL table
run_raw_sql: Run any SQL statement without handling its output
append: Insert rows from the source SQL table into the destination SQL table, if there are no conflicts
merge: Insert rows from the source SQL table into the destination SQL table, depending on conflicts:
- ignore: Do not add rows that already exist
- update: Replace existing rows with new ones
export_file: Export SQL table rows into a destination file
dataframe: Export given SQL table into in-memory Pandas data-frame

For a full list of available operators, see the SDK reference documentation.

Documentation

The documentation is a work in progress--we aim to follow the Diátaxis system:

Getting Started Tutorial: A hands-on introduction to the Astro Python SDK
How-to guides: Simple step-by-step user guides to accomplish specific tasks
Reference guide: Commands, modules, classes and methods
Explanation: Clarification and discussion of key decisions when designing the project

Changelog

The Astro Python SDK follows semantic versioning for releases. Check the changelog for the latest changes.

Release managements

To learn more about our release philosophy and steps, see Managing Releases.

Contribution guidelines

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

Read the Contribution Guideline for a detailed overview on how to contribute.

Contributors and maintainers should abide by the Contributor Code of Conduct.

License

Apache Licence 2.0

Metadata

Release files for astro-sdk-python 1.9.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for astro-sdk-python 1.9.0
File	Size	Uploaded
astro_sdk_python-1.9.0.tar.gz	117.0 kB	Sep 30, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for astro-sdk-python 1.9.0
File	Interpreter	ABI	Platform	Reset
astro_sdk_python-1.9.0-py3-none-any.whl 161.8 kB Sep 30, 2026	Python 3	none	any	Details

Total release size: 278.8 kB

Release files / astro_sdk_python-1.9.0.tar.gz

Download URL	astro_sdk_python-1.9.0.tar.gz
Size	117.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`0e111f31f55b41e0ac539b25a51126fced355448a79d6257c04a3ec996df89e4`
BLAKE2b-256 checksum How to use checksums	`43aec2a62636558334f335f87fcf6c248356f9e40cd4a2acaff5f0d01cdacc0e`
Upload date	Sep 30, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.12`

Release files / astro_sdk_python-1.9.0-py3-none-any.whl

Download URL	astro_sdk_python-1.9.0-py3-none-any.whl
Size	161.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`cd59b4bbae28f65bb100ab6b1b0bfeaf5381f0c821be384df87673abe14cb96f`
BLAKE2b-256 checksum How to use checksums	`8bd3794fb287d6d5318051ec933d592a765e8a9770eab9f44f67d73042906133`
Upload date	Sep 30, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.12.12`