shkit 1.2.21

AUTOMODEL — universal data discovery, relationship intelligence, and warehouse architecture (published as the shkit distribution).

AUTOMODEL

A universal data discovery, relationship-intelligence, and data-architecture framework.

AUTOMODEL inspects a data source, profiles every column, infers primary/foreign keys and relationships, discovers business domains and metrics, builds a knowledge graph, assesses data quality and warehouse readiness, makes architecture recommendations, traces lineage, and generates a model in the architecture of your choice — star, snowflake, normalized ER, Data Vault, or a full dbt project.

The core is deterministic, rule-based, and statistically driven — identical input always yields identical output. Every inferred artifact carries a confidence score and structured evidence, so the platform is explainable and auditable. (AI is an optional explanation layer, never part of the inference path.)

Status: 0.2. Implemented + tested: SQLite/CSV connectors, profiling, PK/FK detection, fact/dim classification; pluggable modeling strategies (dimensional/snowflake/normalized/data_vault/dbt); domain, metric, knowledge-graph, data-quality, readiness, recommendation, and lineage engines; the interactive Architect Explorer HTML report + named JSON/SVG catalogs; CLI. See docs/ARCHITECTURE.md for the full design and roadmap.

Choose your architecture

automodel analyze -s examples/shop.db --strategy data_vault    # or dimensional|snowflake|normalized|dbt
automodel analyze -s examples/shop.db --config examples/automodel.yaml

import automodel
bp = automodel.analyze("sqlite:///examples/shop.db", modeling_strategy="snowflake")
print(bp.domains, bp.metrics, bp.readiness.overall)
bp.knowledge_graph.related("customer")   # "show all assets related to Customer"

Install

pip install -e .            # core (zero third-party deps)
pip install -e ".[dev]"     # + pytest/ruff
pip install -e ".[all]"     # + every connector/UI extra

The core runs on the Python standard library alone. Heavy backends (Postgres, Snowflake, BigQuery, Parquet, Excel, …) are opt-in extras.

Quick start

# Build the bundled sample retail database
python examples/build_sample_db.py

# Full analysis → JSON + Markdown + HTML + DDL in ./automodel_out
automodel analyze --source examples/shop.db --out automodel_out

Or from Python:

import automodel
bp = automodel.analyze("sqlite:///examples/shop.db")
print(bp.model.facts, bp.model.dimensions)

CLI

Command	Output
automodel analyze	full pipeline → all reports + catalogs + SVGs + DDL
automodel profile / discover	metadata + profiling (+ PK/FK) JSON
automodel generate	warehouse model (per --strategy) + DDL
automodel domains / metrics	domain / metric catalog (JSON)
automodel quality / readiness / recommend	intelligence catalogs (JSON)
automodel graph --query Revenue	knowledge graph (JSON/SVG) + related-assets query
automodel dbt	a complete dbt project tree
automodel report --format html|markdown|json	one report
automodel visualize --kind star|lineage|graph	an SVG

Common flags: --source/-s, --out/-o, --strategy, --config <yaml>, --sample N (0 = full scan), --seed, --log-level.

Sources

Built-in (core): SQLite, CSV. Planned extras: PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, BigQuery, Excel, Parquet, JSON, REST APIs — added via the plugin system (automodel.connectors entry-point group).

How it works

connect → metadata → profile → primary keys → foreign keys → graph →
classify (fact/dim) → entities → warehouse model → blueprint

Each stage is an isolated, testable engine. Scoring formulas (PK/FK/fact/dimension) live in docs/ARCHITECTURE.md §7 and every tunable threshold is in automodel.core.config.Config — no magic numbers in engine code.

Development

pytest            # run the suite
ruff check .      # lint

License

Apache-2.0