SibiFlux: The Production Data Toolkit

SibiFlux is a production-grade data engineering toolkit designed for Python 3.11+. It provides a unified ecosystem for project scaffolding, declarative datacube management, automated resource generation, and agentic interoperability through the Model Context Protocol (MCP).

Key Features

• Declarative Datacubes: Define data structures in YAML and auto-generate specialized Python classes.
• Automated Init Engine: Transform modular YAML configs and .env files into strongly-typed Pydantic settings.py.
• High-Performance Storage: Unified fsspec and PyArrow storage registry with depot isolation and path traversal protection.
• Agentic Ready: Native MCP (Model Context Protocol) integration to expose your data pipelines directly to AI agents.
• Scalable Workloads: First-class support for Dask and distributed computing.
• Geospatial Power: Integrated OSMnx helpers and PBF handling for complex GIS workflows.

Tech Stack

• Core: Python 3.11+
• Configuration: Pydantic Settings v2
• CLI Framework: Typer
• Data Engineering: Pandas, PyArrow, Dask
• Storage Layer: fsspec, s3fs
• Databases: SQLAlchemy, Clickhouse, MySQL, PostgreSQL
• Web/API: FastAPI, Uvicorn, MCP
• Orchestration: Docker, Docker Compose, uv

Prerequisites

• Python 3.11+
• uv: Extremely fast Python package manager.
• Docker & Docker Compose: For local services (Postgres, Clickhouse).
• Poethepoet: Task runner (installed via dev dependencies).

Getting Started

1. Clone & Install

git clone https://github.com/lvalverdeb/sibi-flux.git
cd sibi-flux
uv sync --all-extras

2. Environment Setup

Copy the example environment file:

cp .env.linux .env

Initialize the project configuration:

# This generates conf/settings.py and conf/credentials/
sibi-flux env --env-file .env

3. Start Local Infrastructure

docker-compose up -d

4. Project Scaffolding

# Initialize a new project structure
sibi-flux init my_project --app

# Create a specific application within the project
sibi-flux create_app my_app

5. Datacube Workflow

# Propose cubes from a database domain
sibi-flux propose-cubes my_db_domain my_app

# Generate the app-specific extensions
sibi-flux create-cubes my_app

Architecture

Directory Structure

├── src/sibi_flux/
│   ├── init/             # Project bootstrapping & logic generation
│   ├── datacube/         # Datacube orchestration, discovery & mapping
│   ├── storage/          # Unified storage manager (fsspec + pyarrow)
│   ├── mcp/              # MCP Router & Resource registration
│   ├── osmnx_helper/     # Geospatial PBF handling & Graph loading
│   ├── config/           # Unified settings & config management
│   ├── parquet/          # Optimized parquet I/O with gatekeeping
│   ├── df_helper/        # Rich DataFrame manipulation utils
│   └── orchestration/    # Dask/Distributed workload management
├── conf/                 # Generated configuration (settings.py)
├── solutions/            # Reference implementations & examples
├── test_prj/             # Standard test project layout
└── Dockerfile            # Production-grade container image

Data Lifecycle

1. Declarative Phase: Developers define discovery rules and datacube mappings in conf/discovery_params.yaml.
2. Generation Phase: sibi-flux dc commands scan data sources and generate Python wrappers.
3. Runtime Phase:
   • ProjectService detects the root and loads environment-specific settings.py.
   • StorageManager establishes connections to S3/WebDAV/Local.
   • Datacube instances provide high-level abstractions over PyArrow/Dask.
4. Exposure Phase: The MCP Router registers datacubes as resources for external consumption (e.g., by LLM agents).

Unified Storage Registry

SibiFlux implements a robust storage registry that isolates distinct "depots" (e.g., bronze, silver, gold).

• Isolation: Each depot is mapped to a designated path.
• Performance: Automatic switching between fsspec and Native C++ PyArrow S3FileSystem.
• Security: Built-in path traversal protection (blocking .. in joined segments).

Environment Configuration

Required Variables

Modular YAMLs

The init engine populates conf/ with modular YAML files:

• storage.yaml: S3 layouts and filesystem definitions.
• databases.yaml: Database connection mapping.
• osmnx.yaml: Geospatial storage references.

Available Scripts (Poe)

Testing

SibiFlux uses complex snapshot-based testing for code generation verification.

# Run all tests
pytest tests/

# Update snapshots (Golden Masters)
UPDATE_SNAPSHOTS=1 pytest

Deployment

Docker

The project includes a production-ready multi-stage Dockerfile.

docker build -t sibi-flux:latest .

Docker Compose

The docker-compose.yml orchestrates:

• API: The main SibiFlux web service (FastAPI).
• Dask Scheduler: For distributed task execution.
• Postgres/Clickhouse: Local data stores for development.

Troubleshooting

• ImportError: Ensure you have run uv sync to install all dependencies.
• Connection Error: Check your .env variables and ensure Docker services are running.
• Storage Risk: If you see "Security Risk: Path traversal detected," ensure your dynamic paths do not contain .. or absolute prefixes.

Contributing

1. Follow the Conventional Commits standard.
2. Ensure all tests pass (poe test).
3. Add snapshots for any new code generation logic.