dataenginex

The Python library that powers DataEngineX Studio — an open-source, self-hosted, local-first Data + ML + AI workbench for individuals and small teams. Use the library directly when you want code; install DataEngineX Studio when you want a UI.

Install

pip install dataenginex                  # lean base — DuckDB, structlog, Pydantic, Click, pyarrow

from dataenginex.engine import DexEngine

engine = DexEngine("dex.yaml")           # loads config, inits SQLite store, wires backends
engine.run_pipeline("clean_users")       # execute a pipeline
models = engine.model_registry.list_models()
response = engine.agents["assistant"].chat("summarise the latest run")

Optional integrations install only what you need:

LiteLLM must be installed separately due to a python-dotenv pin conflict:

pip install 'litellm>=1.83.3' --no-deps

What it does

dex.yaml describes a project. DexEngine reads it and wires the pieces:

Local-first by default. A fresh pip install dataenginex requires no external services — DuckDB is embedded; nothing reaches the network unless you explicitly configure it (or call a hosted LLM).

Project structure

src/dataenginex/
├── ai/                 # LLM providers, routing, agents, retrieval, tools, memory, workflows
├── api/                # Pydantic response models, GraphQL schema (no HTTP server)
├── application/        # Domain services (governance, projects, runs, resources)
├── bootstrap/          # Engine wiring and startup
├── cli/                # `dex` CLI: validate, version, init
├── config/             # dex.yaml schema, loader, env-var resolution
├── core/               # Core abstractions and shared utilities
├── data/               # Connectors (CSV, Parquet, DuckDB, REST, HTTP, Kafka), transforms
├── domains/            # Domain logic (ai, analytics, data, execution, governance, ml, plugins, security)
├── duckdb/             # DuckDB connection and helpers
├── engine.py           # DexEngine — entry point
├── engines/            # Engine implementations and registry (DuckDB, Spark, base)
├── foundation/         # Base ABCs, plugin contracts, errors, events, resources
├── interfaces/         # Gateway, embedded, external service interfaces
├── lakehouse/          # Bronze/Silver/Gold lakehouse storage and catalog
├── middleware/         # structlog, Prometheus metrics
├── ml/                 # ML training, registry, serving, drift, features
├── orchestration/      # Scheduler, background workers, queue backends
├── orm/                # SQLAlchemy ORM models
├── plugins/            # Entry-point plugin discovery
├── providers/          # Backend implementations (connectors, vector, tracking)
├── runtime/            # Compiler, registry, ML runtime, sandbox
├── secops/             # PII detection, masking, audit
├── spark/              # PySpark connector (SQL, streaming, ML, catalog, lineage)
├── store.py            # DexStore — DuckDB (WAL) persistence
└── warehouse/          # Warehouse operations

Development

git clone https://github.com/TheDataEngineX/dataenginex && cd dataenginex
uv sync
uv run poe check-all          # lint + typecheck + tests
uv run poe test-cov           # tests + coverage
uv run poe lint-fix           # auto-fix lint issues
dex validate dex.yaml         # validate a config file
dex version                   # show version + environment

Want the full workbench?

dataenginex is the library. The web UI is dex-studio — it imports dataenginex directly, no HTTP hop.

Ecosystem

Documentation

License: MIT • Python: 3.13+ • Status: Pre-1.0 • Version: 0.7.0