jett 0.0.2

just a engine template tool

Jett

Just a Engine Template Tool that easy to use and develop for Data Engineer. This project support the ETL template for multiple DataFrame engine like PySpark, Duckdb, Polars, etc.

Supported Features:

• Dynamic Supported Engines via YAML template
• JSON Schema Validation support

📦 Installation

uv pip install -U jett

Engine Supported:

Name	Status	Description
Pyspark	✅	Pyspark and Spark submit CLI for distributed workload
DuckDB	✅	DuckDB and Spark API DuckDB
Polars	✅	Polars for Python workload
Arrow	✅	Arrow for Python workflow with Columnar
Daft	❌	Daft for Python distributed workload
DBT	❌	DBT for SQL workload
GX	❌	Great Expectation for data quality

[!NOTE] Version Tracking:

Package	Version	Next Support
Python	3.10.13	>=3.11.0
Spark	3.4.2	>=4.0.0
Hadoop	3	3
Java	openjdk@11	openjdk@17
Pyspark	3.4.1	>=4.0.0
Scala	2.12.17	2.12.17
DuckDB	1.3.2
Polars	1.32.0
Arrow	21.0.0

📝 Usage

For example, making file, etl.polars.tool (I use .tool be file extension for validate it with the JSON schema with pattern *.tool), for ETL state like:

type: polars
name: Load CSV to GGSheet
app_name: load_csv_to_ggsheet
master: local

# 1) 🚰 Load data from source
source:
  type: local
  file_format: csv
  path: ./assets/data/customer.csv

# 2) ⚙️ Transform this data.
transforms:
  - op: rename_to_snakecase
  - op: group
    transforms:
      - op: expr
        sql: "CAST(id AS string)"

# 3) 🎯 Sink result to target
sink:
  type: local
  file_type: google_sheet
  path: ./assets/landing/customer.gsheet

# 4) 📩 Metric that will send after execution.
metric:
  - type: console
    convertor: basic
  - type: restapi
    convertor: basic
    host: "localhost"
    port: 1234

Use by Python API:

from jett import Tool

tool = Tool(path="./etl.spark.tool")
tool.execute(allow_raise=True)

📖 Documents

This project will reference emoji from the Pipeline Emojis.

💬 Contribute

I do not think this project will go around the world because it has specific propose, and you can create by your coding without this project dependency for long term solution. So, on this time, you can open the GitHub issue on this project 🙌 for fix bug or request new feature if you want it.