DataForge helps data analysts and engineers build and extend data solutions by leveraging modern software engineering principles.
Understanding DataForge
DataForge enables writing of inline functions using single-column SQL expressions rather than CTEs, procedural scripts, or set-based models.
For an overview of the underlying concepts, check out this introduction blog.
Each function:
- is pure, with no side effects
- returns single column
- is composable with other functions
DataForge software engineering principles:
These principles allow DataForge projects to be easy to modify and extend - even with thousands of integrated pipelines.
Explore the Core CLI or learn more about how Core powers DataForge Cloud.
Requirements
Dataforge Core is a code framework and command line tool to develop transformation functions and compile them into executable Spark SQL.
To run the CLI you will need:
- Java 8 or higher
- Amazon Corretto is a great option
- A PostgreSQL v14+ server with a dedicated empty database
- Check out our friends over at Tembo
- Python version 3.12+
The CLI also includes an integration to run the code in Databricks. To support this you need:
Installation and Quickstart
-
Open a new command line window
-
Validate Java and Python are installed correctly:
> java --version openjdk 21.0.3 2024-04-16 LTS> python --version Python 3.12.3 -
Install Dataforge by running:
> pip install dataforge-core Collecting dataforge-core... Installing collected packages: dataforge-core Successfully installed dataforge-core... -
Validate installation:
> dataforge --version dataforge-core 1.0.0 -
Configure connections and credentials to Postgres and optionally Databricks
> dataforge --configure Enter postgres connection string: postgresql://postgres:<postgres-server-url>:5432/postgres Do you want to configure Databricks SQL Warehouse connection (y/n)? y Enter Server hostname: <workspace-url>.cloud.databricks.com Enter HTTP path: /sql/1.0/warehouses/<warehouse-guid> Enter access token: <token-guid> Enter catalog name: <unity_catalog_name> Enter schema name: <schema_in_catalog_name> Connecting to Databricks SQL Warehouse <workspace-url>.cloud.databricks.com Databricks connection validated successfully Profile saved in C:\Users... -
Navigate to an empty folder and initialize project structure and sample files:
> dataforge --init Initialized project in C:\Users... -
Deploy dataforge structures to Postgres
> dataforge --seed All objects in schema(s) log,meta in postgres database will be deleted. Do you want to continue (y/n)? y Initializing database.. Database initialized -
Build sample project
> dataforge --build Validating project path C:\Users... Started import with id 1 Importing project files... <list of files> Files parsed Loading objects... Objects loaded Expressions validated Generated 8 source queries Generated 1 output queries Generated run.sql Import completed successfully -
Execute in Databricks
> dataforge --run Connecting to Databricks SQL Warehouse <workspace-url>.cloud.databricks.com Executing query Execution completed successfully
Commands
| -h, --help | Display this help message and exit |
| -v, --version | Display the installed DataForge version |
| -c, --configure | Connect to Postgres database and optionally Databricks SQL Warehouse |
| -s, --seed | Deploy tables and scripts to postgres database |
| -i, --init [Project Path] | Initialize project folder structure with sample code |
| -b, --build [Project Path] | Compile code, store results in Postgres, and generate target SQL files |
| -r, --run [Project Path] | Run compiled project on Databricks SQL Warehouse |
| -p, --profile [Profile Path] | Update path of stored credentials profile file |
Links
2 maintainers
Release files for dataforge-core 1.6.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dataforge_core-1.6.2.tar.gz | 71.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dataforge_core-1.6.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 146.8 kB
Release files / dataforge_core-1.6.2.tar.gz
| Download URL | dataforge_core-1.6.2.tar.gz |
|---|---|
| Size | 71.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4997bb9c594755f69ab511cc9b64cdf01e85551b76bbe0d42a4305303875fd04
|
|
BLAKE2b-256 checksum How to use checksums |
3fb855abd2e068c3452b70baeee5f2020175c05b88dc16b5490212902e02788c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 18, 2026.
Transparency logRelease files / dataforge_core-1.6.2-py3-none-any.whl
| Download URL | dataforge_core-1.6.2-py3-none-any.whl |
|---|---|
| Size | 75.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b4346055794250d3b3b0e4955884f3a7e607e6824ef25948b1d2d160826118b6
|
|
BLAKE2b-256 checksum How to use checksums |
8b9265e34f7c6900e328d4f7556cedf37c84fc81b213e77200fb5d4900dfeb16
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 18, 2026.
Transparency log
