BeETL: Extensible Python/Polars-based ETL Framework

BeETL was born from a job as Integration Developer where a majority of the integrations we develop follow the same pattern - get here, transform a little, put there (with the middle step frequently missing altogether).

After building our 16th integration between the same two systems with another manual template, we decided to build BeETL. BeETL is currently limited to one datasource per source and destination per sync, but this will be expanded in the future. One configuration can contain multiple syncs.

Note: Even though some of the configuration below is in YAML format, you can also use JSON or a python dictionary.

Minimal example

# Syncing users from one table to another in the same database
from src.beetl.beetl import Beetl, BeetlConfig
config = BeetlConfig({
    "version": "V1"
    "sources": [
        {
            "name": "Sqlserver",
            "type": "Sqlserver",
            "connection": {
                "settings": {
                    "connection_string": "Server=myServerAddress;Database=myDataBase;User Id=myUsername;Password=myPassword;"
                }
            }
        },
    "sync": [
        {
            "name": "Sync between two tables in a sql server",
            "source": "Sqlserver",
            "sourceConfig": {
                "query": "SELECT id, name, email FROM users"
            }
            "destination": "SqlServer",
            "destinationConfig": {
                "table": "users",
                "unique_columns": ["id"]
            }
            "comparisonColumns": [
                {
                    "name": "id",
                    "type": "Int32",
                    "unique": True
                },
                {
                    "name": "name",
                    "type": "Utf8"
                },
                {
                    "name": "email",
                    "type": "Utf8"
                }
            ]
        }
    ]
})

Beetl(config).sync()

Installation

From PyPi

#/bin/bash
python -m pip install beetl

# If you need to use xsl transformations
python -m pip install beetl[xsl]

From Source

#/bin/bash
# Clone and enter the repository
git clone https://github.com/Hoglandets-IT/beetl.git
cd ./beetl
# Install the build tools
python -m pip install build
# Build beetl
python -m build
# Install beetl from locally built package
python -m pip install ./dist/*.tar.gz

Getting Started

All the latest information about how to use beetl is located at the official docs.

Development Environment

The easiest way to get started is to use the included devcontainer.

Requirements

Docker
Visual Studio Code

Steps

Clone the repository.
Open the repository in Visual Studio Code.
Install the recommended extensions.
Using the command palette (ctrl+shift+p) search for reopen in container and run it.
- The devcontainer will now be provisioned in your local docker instance and vscode will automatically connect to it.
You can now use the included launch profiles to either open the docs or run the tests file.
You can also use the built-in test explorer to run the available test.

Metadata

Release files for beetl 1.4.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for beetl 1.4.2
File	Size	Uploaded
beetl-1.4.2.tar.gz	58.5 kB	Oct 16, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for beetl 1.4.2
File	Interpreter	ABI	Platform	Reset
beetl-1.4.2-py3-none-any.whl 88.8 kB Oct 16, 2026	Python 3	none	any	Details

Total release size: 147.3 kB

Release files / beetl-1.4.2.tar.gz

Download URL	beetl-1.4.2.tar.gz
Size	58.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`9c5467a27d002d9b2c8a927a6be72e553cbdd1913922c7e65a5f2d69fc94414c`
BLAKE2b-256 checksum How to use checksums	`690723b5156af49935135480643cc941df1bfdf9253562390ee5e48044f10644`
Upload date	Oct 16, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.10.22`

Release files / beetl-1.4.2-py3-none-any.whl

Download URL	beetl-1.4.2-py3-none-any.whl
Size	88.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`e3439b3ed57f697a71b69376ca6c6e09d0c175a794fea34c929738553a48563e`
BLAKE2b-256 checksum How to use checksums	`f302dab99e5212912c19b790092a2bc890f611b8522a0c77bf0bfc0c0b621a75`
Upload date	Oct 16, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/7.0.0 CPython/3.10.22`