det — Data Engineering Toolkit

by det Labs

A modular, Python-first data engineering framework built by det Labs for modern analytical workflows.

Current Status: Version 0.1.0 (Catalog MVP Released)

det (developed by det Labs) has completed its Version 0.1.0 milestone with the release of the Catalog Module (det.catalog):

• Concurrency-safe, atomic YAML metadata persistence with file locks.
• Automated schema discovery for Parquet, CSV, TSV, JSON, NDJSON, Excel, and directory datasets using Polars and PyArrow.
• Leverages Python 3.15 stable native lazy import syntax for instantaneous engine load times.
• Full Python programmatic Command API (CatalogService).
• Interactive Local Browser Web GUI with vertical category explorer, draggable resizer, quick circular action buttons, Discover Schema modal, inline column description editing, and a 3-way theme switcher (Light / Dark / Auto System).
• Typer CLI (det catalog ... and det-gui).

Architectural Vision & Module Roadmap

det is built from the ground up by det Labs to be modular, extensible, and Python-native:

┌─────────────────────────────────────────────────────────────┐
│                    User Interfaces                         │
│         CLI (`det`)   │   Web GUI (`det-gui`)   │   API     │
└──────────────┬──────────────────┬──────────────────┬────────┘
               │                  │                  │
┌──────────────▼──────────────────▼──────────────────▼────────┐
│                 det Core Modules (det Labs)                 │
│                                                             │
│  [ Released v0.1 ]                                          │
│  ├── det.catalog     Metadata, schemas, YAML store & GUI    │
│                                                             │
│  [ Planned Roadmap ]                                        │
│  ├── det.engine      Execution abstraction (DuckDB, Polars) │
│  ├── det.pipeline    DAG orchestration & SQL/Python models  │
│  ├── det.quality     Schema contracts, profiling & tests    │
│  ├── det.lineage     Table & column-level dependency graphs │
│  └── det.connectors  Ingestion & sync (Iceberg, Delta, S3)  │
└─────────────────────────────────────────────────────────────┘

Module Roadmap

Quickstart

Prerequisites

• Python: >= 3.15 (Stable released October 9, 2026)

Installation

From PyPI:

pip install det-core

Or from source:

# Clone the repository
git clone https://github.com/det-Labs/det.git
cd det

# Create virtual environment with Python 3.15
py -3.15 -m venv .venv
.venv\Scripts\activate       # On Windows
# source .venv/bin/activate  # On Linux/macOS

# Install in editable mode with development tools
pip install -e ".[dev]"

# Verify test suite
pytest

1. Launch the Catalog Web GUI

Experience the interactive visual catalog manager directly in your local browser:

det-gui

Navigate to http://127.0.0.1:8501:

• Group datasets into categories / namespaces.
• Point to local files or folders and click Discover Schema to auto-infer columns and types.
• Drag the sidebar border to resize the explorer bar manually (double-click to reset).
• Switch themes between Light, Dark, or Auto (System Default).
• Edit column descriptions inline by clicking on them.
• Switch between different YAML catalog files or create brand new catalogs.

2. Using the CLI

# View system & version info
det version
det info

# Scan a dataset to inspect its inferred schema without registering
det catalog scan sample_data/orders.parquet

# List all registered tables in active catalog
det catalog list

# Set a catalog file as the persistent default
det catalog set-default catalogs/main.yaml

3. Programmatic Python API

from det.catalog import CatalogService, DatasetMetadata

# Connect to catalog
service = CatalogService("catalogs/main.yaml")

# Discover schema from a Parquet or CSV file
scan = service.scan_path("data/customers.parquet")

# Register into catalog
service.register_dataset(DatasetMetadata(
    id="crm.customers",
    name="customers",
    namespace="crm",
    format="parquet",
    location="data/customers.parquet",
    description="Customer master records",
    columns=scan.columns,
    row_count=scan.row_count,
))

# Retrieve and inspect
dataset = service.get_dataset("crm.customers")
print(f"Table {dataset.name} has {len(dataset.columns)} columns and {dataset.row_count} rows.")

Testing & Quality

All code adheres to strict standards:

• PEP 257 docstrings on all public functions, classes, and methods.
• Explicit type annotations.
• Python 3.15 lazy import integration.
• Fast, automated pytest test suite:

pytest -v
ruff check .

Publishing a Release

Releases are published to PyPI only after the package passes tests and the pypi GitHub Environment deployment is approved. To release:

1. Update the version in pyproject.toml and src/det/__init__.py.
2. Push the changes to main and create a GitHub Release with a matching tag (for example, v0.1.0a2).
3. After the workflow builds and tests the release, approve its deployment to the pypi environment in GitHub Actions. The workflow then publishes the distributions using PyPI Trusted Publishing (OIDC).

The pypi environment and the matching Trusted Publisher must be configured once in GitHub and PyPI, respectively.

License

Distributed under the Apache License 2.0. Copyright (c) 2026 det Labs. See LICENSE for details.