dataenginex

The Python library that powers DEX Studio — an open-source, self-hosted, local-first Data + ML + AI workbench. Use the library directly when you want code, not a UI.

Pre-1.0 status. 0.4.0 is honest about that. See the scope reset CHANGELOG for the rationale.

Install

pip install dataenginex                    # lean base — DuckDB, structlog, Pydantic, Click, pyarrow

Optional integrations — install only what you need:

pip install 'dataenginex[postgres]'        # asyncpg-backed lineage, persistence
pip install 'dataenginex[qdrant]'          # Qdrant vector store backend
pip install 'dataenginex[queue]'           # ARQ async job queue (pulls redis)
pip install 'dataenginex[cloud]'           # S3, GCS, BigQuery storage backends
pip install 'dataenginex[ml]'              # scikit-learn, xgboost, sentence-transformers
pip install 'dataenginex[tracking]'        # MLflow integration
pip install 'dataenginex[data]'            # PySpark, databricks-cli

LiteLLM: install separately — it pins python-dotenv==1.0.1 which conflicts with our >=1.2.2:
pip install 'litellm>=1.83.3' --no-deps

Quick start

from pathlib import Path
from dataenginex.engine import DexEngine

# Load config and initialize all backends
engine = DexEngine(Path("dex.yaml"))

# Data — run pipelines defined in dex.yaml
engine.run_pipeline("clean_users")

# ML — train, register, predict
models = engine.model_registry.list_models()
result = engine.model_registry.predict("churn_model", features)

# AI — chat with an agent over your data
response = engine.agents["assistant"].chat("summarise the latest pipeline run")

# Persistence — query SQLite-backed history
runs = engine.store.list_pipeline_runs(limit=10)

Smaller surfaces — use only what you need:

from dataenginex.config import load_config
cfg = load_config("dex.yaml")

from dataenginex.core.interfaces import BaseConnector
from dataenginex.core.registry import BackendRegistry

from dataenginex.ml import ModelRegistry
from dataenginex.ai.llm import get_llm_provider
from dataenginex.ai.vectorstore import VectorStoreBackend

Submodules

Module	Description
`dataenginex.engine`	`DexEngine` — single entry point; loads config, inits store, wires backends
`dataenginex.store`	`DexStore` — SQLite (WAL) persistence (`.dex/store.duckdb`)
`dataenginex.config`	`dex.yaml` schema, loader, env-var resolution
`dataenginex.core`	Exceptions, `Base*` ABCs, `BackendRegistry`
`dataenginex.cli`	`dex` CLI (`validate`, `version`, `init`)
`dataenginex.data`	Connectors (CSV, Parquet, DuckDB, HTTP, …), pipeline runner, schema registry
`dataenginex.ml`	Classical ML — training, model registry, serving, drift
`dataenginex.ai`	LLM providers, agents, RAG, vector store, memory, observability
`dataenginex.orchestration`	Scheduler, background workers
`dataenginex.middleware`	structlog config, Prometheus metrics
`dataenginex.lakehouse`	Storage backends, catalog, partitioning
`dataenginex.warehouse`	Transforms, lineage tracking
`dataenginex.secops`	PII detection, masking, audit logging
`dataenginex.api`	Pydantic response models (no HTTP server bundled)
`dataenginex.plugins`	Entry-point plugin discovery

Want the UI?

dataenginex is the engine. The web UI lives in a separate repo: github.com/TheDataEngineX/dex-studio. It imports dataenginex directly — no separate API server.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dataenginex-0.5.1.tar.gz (583.3 kB view details)

Uploaded Jul 21, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

dataenginex-0.5.1-py3-none-any.whl (248.9 kB view details)

Uploaded Jul 21, 2026 Python 3

File details

Details for the file dataenginex-0.5.1.tar.gz.

File metadata

Download URL: dataenginex-0.5.1.tar.gz
Upload date: Jul 21, 2026
Size: 583.3 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dataenginex-0.5.1.tar.gz
Algorithm	Hash digest
SHA256	`fcf0172f2672b85607d85d24f9aaf7b51c721a877e813f785215ea9b7f34348f`
MD5	`56e7adecc666cece8c7af2c23f617549`
BLAKE2b-256	`11dfad77472a162cf86c22b872f64224a3782232b9dfbf0a7727e8508290a797`

See more details on using hashes here.

Provenance

The following attestation bundles were made for dataenginex-0.5.1.tar.gz:

Publisher: release.yml on TheDataEngineX/dataenginex

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: dataenginex-0.5.1.tar.gz
- Subject digest: fcf0172f2672b85607d85d24f9aaf7b51c721a877e813f785215ea9b7f34348f
- Sigstore transparency entry: 2212455519
- Sigstore integration time: Jul 21, 2026
Source repository:
- Permalink: TheDataEngineX/dataenginex@e1dfaf76f41adcfc4f05d073a728a58bfedba6ea
- Branch / Tag: refs/heads/main
- Owner: https://github.com/TheDataEngineX
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@e1dfaf76f41adcfc4f05d073a728a58bfedba6ea
- Trigger Event: workflow_dispatch