FastETL framework, modern, versatile, does almost everything.
Este texto também está disponível em português: 🇧🇷LEIAME.md.
FastETL is a plugins package for Airflow for building data pipelines for a number of common scenarios.
Main features:
- Full or incremental replication of tables in SQL Server, Postgres and MySQL databases
- Load data from GSheets and from spreadsheets on Samba/Windows networks
- Extracting CSV from SQL
- Clean data using custom data patching tasks (e.g. for messy geographical coordinates, mapping canonical values for columns, etc.)
- Using a Open Street Routing Machine service to calculate route distances
- Using CKAN or dados.gov.br's API to update dataset metadata
- Using Frictionless Tabular Data Packages to write OpenDocument Text format data dictionaries
This framework is maintained by a network of developers from many teams at the Ministry of Management and Innovation in Public Services and is the cumulative result of using Apache Airflow, a free and open source tool, starting in 2019.
For government: FastETL is widely used for replication of data queried via Quartzo (DaaS) from Serpro.
Installation in Airflow
FastETL implements the standards for Airflow plugins. To install it,
simply add the apache-airflow-providers-fastetl package to your
Python dependencies in your Airflow environment.
Or install it with
pip install apache-airflow-providers-fastetl
To see an example of an Apache Airflow container that uses FastETL, check out the airflow2-docker repository.
To ensure appropriate results, please make sure to install the
msodbcsql17 and unixodbc-dev libraries on your Apache Airflow workers.
Tests
The test suite uses Docker containers to simulate a complete use environment, including Airflow and the databases. For that reason, to execute the tests, you first need to install Docker and docker-compose.
For instructions on how to do this, see the official Docker documentation.
To build the containers:
make setup
To run the tests, use:
make setup && make tests
To shutdown the environment, use:
make down
Usage examples
The main FastETL feature is the DbToDbOperator operator. It copies data
between postgres and mssql databases. MySQL is also supported as a
source.
Here goes an example:
from datetime import datetime
from airflow import DAG
from fastetl.operators.db_to_db_operator import DbToDbOperator
default_args = {
"start_date": datetime(2023, 4, 1),
}
dag = DAG(
"copy_db_to_db_example",
default_args=default_args,
schedule_interval=None,
)
t0 = DbToDbOperator(
task_id="copy_data",
source={
"conn_id": airflow_source_conn_id,
"schema": source_schema,
"table": table_name,
},
destination={
"conn_id": airflow_dest_conn_id,
"schema": dest_schema,
"table": table_name,
},
destination_truncate=True,
copy_table_comments=True,
chunksize=10000,
dag=dag,
)
More detail about the parameters and the workings of DbToDbOperator
can bee seen on the following files:
How to contribute
To be written on the CONTRIBUTING.md document (issue
#4).
5 maintainers
Metadata
5 maintainers
Release files for apache-airflow-providers-fastetl 0.2.14
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| apache_airflow_providers_fastetl-0.2.14.tar.gz | 80.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| apache_airflow_providers_fastetl-0.2.14-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 165.5 kB
Release files / apache_airflow_providers_fastetl-0.2.14.tar.gz
| Download URL | apache_airflow_providers_fastetl-0.2.14.tar.gz |
|---|---|
| Size | 80.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bef8b946c057024d45708016d9733d43e96bb6bee8d06667d3b6c81b46d7b99c
|
|
BLAKE2b-256 checksum How to use checksums |
b19e97de5e7e6ced8d3695fd1a2e3350530513f65d6d4d2130c290c713f37a16
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 29, 2026.
Transparency logRelease files / apache_airflow_providers_fastetl-0.2.14-py3-none-any.whl
| Download URL | apache_airflow_providers_fastetl-0.2.14-py3-none-any.whl |
|---|---|
| Size | 84.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
85f817282aeced14f4b264919d57df902677bb79f8f1cc322c97121b2ab655cf
|
|
BLAKE2b-256 checksum How to use checksums |
0925ae271fab1af67fe21ed0d1c6cace6b73c09e1bbca7f66373c0fa8a34d56e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Apr 29, 2026.
Transparency log
