DataCoolie Studio

DataCoolie Studio is a local web app for exploring DataCoolie projects. Use it to manage sources, edit metadata, inspect lineage, and monitor extract, transform, and load (ETL) runs.

What you can do

• Organize sources by Project and Environment
• Read and edit metadata in JSON, YAML, or XLSX format
• Inspect lineage from metadata, SQL queries, and Python code
• Monitor Dataflow, Job, and System logs
• Connect to Local, S3, MinIO, ADLS, OneLake, GCS, and Databricks storage
• Store cloud credentials in the operating system credential store

Studio keeps source files as the source of truth. Metadata saves validate the document and create a backup before replacing the original file. Lineage combines evidence for display without creating a merged metadata file.

Screenshots

The screenshots below show the main Studio workflow using a populated local DataCoolie environment.

Projects and Environment

Projects provides a single workspace view for project readiness, environment navigation, and source coverage.

Environment Overview brings Metadata, Lineage, Monitoring, freshness, and next actions together in one screen.

Metadata, Assets, Lineage, and Sources

Metadata presents connections, dataflows, schema hints, and ordered source-defined transform configuration in an editable workspace.

Assets provides an inventory of discovered assets and references, including resolution and usage context.

Lineage connects metadata, SQL, and Python evidence into an interactive graph with filters and run-status context.

Sources shows Local and cloud bindings, readable/cache status, scheduled Log refresh, and one-click path copying.

When scanning a project, Studio always uses automatic layout discovery. It keeps the configured functions root and also recognizes common Python layouts from a root pyproject.toml. A project with src/<package>/**/*.py is indexed with import names such as package.module, without executing or installing the project. .py, .zip, and .whl files found below the code root become separate code artifacts, so their provenance and validation state remain independent. Unsupported or ambiguous namespace layouts are reported for manual module-root/module-prefix configuration. The API still accepts code_discovery_mode=explicit for advanced integrations that need to limit a scan to the configured folder.

Monitoring

Monitoring is split into nine focused pages so operational questions can be answered without leaving the Environment.

Open all 9 Monitoring pages

Overview Health KPIs, trends, runtime context, and attention signals.	Jobs Job status, duration, runtime context, and drill-in evidence.
Dataflows Dataflow filtering, execution status, timings, and source/destination context.	Failures Failure categories, repeated failures, and investigation entry points.
Freshness Source freshness, event time, watermarks, and stale-data signals.	Performance Duration percentiles, phase contribution, pressure, and candidates.
Volume Rows, bytes, files, workload trends, and file-churn candidates.	Maintenance Maintenance operations, destination impact, and performance signals.
Diagnostics Bounded diagnostic aggregates and investigation evidence.

Install and run

DataCoolie Studio requires Python 3.11 or later.

pip install datacoolie-studio
datacoolie-studio

The launcher starts Studio at http://127.0.0.1:8765, creates its local workspace on first run, and opens your browser. On a wide interactive terminal it prints a large DataCoolie wordmark, links to the DataCoolie and DataCoolie Studio repositories, and the URL it is starting. Narrow terminals and redirected output use a compact version of the introduction.

Use --no-banner when only the server logs are needed:

datacoolie-studio --no-banner

The startup introduction looks like this:

 ____        _         ____            _ _
|  _ \  __ _| |_ __ _ / ___|___   ___ | (_) ___
| | | |/ _` | __/ _` | |   / _ \ / _ \| | |/ _ \
| |_| | (_| | || (_| | |__| (_) | (_) | | |  __/
|____/ \__,_|\__\__,_|\____\___/ \___/|_|_|\___|

Studio v<version>
Explore DataCoolie metadata, lineage, and ETL logs.

DataCoolie - Metadata-driven ETL framework
  https://github.com/datacoolie/datacoolie
DataCoolie Studio - Local web app for DataCoolie projects
  https://github.com/datacoolie/datacoolie-studio

Starting at: http://127.0.0.1:8765
Press Ctrl+C to stop.

Install only the cloud integrations you need:

pip install "datacoolie-studio[s3]"
pip install "datacoolie-studio[minio]"
pip install "datacoolie-studio[adls]"
pip install "datacoolie-studio[onelake]"
pip install "datacoolie-studio[gcs]"

Databricks SDK support is included in the base installation. Use pip install "datacoolie-studio[cloud]" to install every other cloud integration. For a complete environment with every cloud integration and the development, test, and packaging tools, use:

pip install "datacoolie-studio[all]"

Create your first workspace

1. Create a Project
2. Add an Environment such as dev, test, or prod
3. Add a metadata file or scan a DataCoolie project
4. Add ETL logs for Monitoring
5. Add Python code artifacts when metadata references Python functions
6. Open Metadata, Assets, Lineage, or Monitoring

Metadata is required. Logs and code artifacts are optional.

Configure Studio

Studio stores persistent local state under ~\.datacoolie\datacoolie-studio\ by default. Set --data-dir or DATACOOLIE_STUDIO_DATA_DIR to move the entire local root:

<data-dir>/
db\studio.db
backups\
cache\read-models.sqlite3
cache\analytics.duckdb
cache\source-materializations\
logs\

Common launcher options:

datacoolie-studio --port 8765
datacoolie-studio --host 127.0.0.1
datacoolie-studio --data-dir .\.scratch\datacoolie-studio
datacoolie-studio --database-url "postgresql+psycopg://user:password@host:5432/datacoolie_studio"
datacoolie-studio --result-cache-url "sqlite:///D:/data/datacoolie-studio/cache/read-models.sqlite3"
datacoolie-studio --no-open

You can also configure storage with environment variables:

CLI values take precedence over environment variables. The workspace URL accepts SQLite URLs such as sqlite:///D:/data/datacoolie-studio/db/studio.db and other SQLAlchemy URLs supported by the installed drivers. The result-cache provider currently accepts SQLite URLs and memory://; remote Redis/PostgreSQL result cache backends are not enabled.

--db and DATACOOLIE_STUDIO_DB are not part of the current configuration contract. Configure a local deployment with --data-dir or DATACOOLIE_STUDIO_DATA_DIR; use --database-url or DATACOOLIE_STUDIO_DATABASE_URL only when an explicit workspace database URL is required.

Studio binds to 127.0.0.1 by default. Choose a shared database and review network access before hosting it for multiple users.

Develop from source

Run the backend directly from src. This assumes the active Python environment already contains the dependencies declared in pyproject.toml.

$env:PYTHONPATH = "$PWD\src"
python -m uvicorn datacoolie_studio.main:app `
  --reload `
  --host 127.0.0.1 `
  --port 8765

Run the frontend in another terminal:

cd frontend
npm install
npm run dev

Open http://127.0.0.1:5173. Vite sends API requests to the backend at http://127.0.0.1:8765.

Build the frontend into the Python package:

cd frontend
npm run build

Run repository checks:

.\scripts\verify.ps1
.\scripts\verify.ps1 -Mode Full

The default check covers architecture, packaged static assets, security, API contracts, frontend tests, and the production build. Full mode also runs the complete backend test suite.