datalore-cli 0.4.1

Deterministic local data analysis CLI for coding agents.

Datalore

Datalore is a deterministic local data-analysis CLI designed to amplify coding agents.

The core idea is simple: when Codex, Claude Code, or a human operator needs a reliable answer about a local dataset, they should call a stable binary with explicit arguments and parse structured output instead of improvising pandas, sklearn, and matplotlib every time.

What It Does

• Inspect local CSV, Excel, and JSON datasets
• Profile columns, dtypes, missing values, and duplicates
• Generate structured descriptive statistics
• Generate structured correlation reports
• Run deterministic linear regression
• Apply deterministic cleaning and export operations
• Compare two datasets structurally and at the row level
• Generate reproducible plot artifacts
• Diagnose runtime/install issues with a doctor command
• Emit either human-readable text or machine-readable JSON
• Optionally persist the full JSON envelope to a report file
• Keep outputs under artifacts/ by default

Install

Current install flow, from any machine where you want to analyze local data:

python3 -m venv .venv
source .venv/bin/activate
pip install 'git+https://github.com/micic-mihajlo/datalore-cli.git'

After that, the public interface is just:

datalore --help

The repo-local ./datalore wrapper is only a development convenience for contributors working inside this repository.

If you are developing Datalore itself, install it in editable mode from the repo root:

pip install -e .

Use In Your Own Project

Move into the folder that contains your data:

cd /path/to/my/data
datalore init

That creates:

• AGENTS.md
• CLAUDE.md
• ./datalore
• artifacts/
• a .gitignore entry for artifacts/

After that, both you and your coding agent can work from that folder directly:

./datalore files . --format json
./datalore profile sales.csv --format json
./datalore clean sales.csv --rename sales=>revenue --output artifacts/datasets/sales_clean.csv --format json

Why the wrapper matters:

• humans can keep using datalore
• agents are usually safer using ./datalore because it binds the project to the exact Python interpreter that installed Datalore

CLI Usage

Inspect a dataset:

datalore inspect data.csv

Profile a dataset in JSON mode for agent consumption:

datalore profile data.csv --format json

Run a simple regression:

datalore regression data.csv --target last --format json

Constrain the predictors:

datalore regression data.csv --target revenue --predictors spend,visits,leads --format json

Create a plot with a stable output path:

datalore plot data.csv --kind histogram --x revenue --output artifacts/plots/revenue_hist.png --format json

Generate a correlation heatmap:

datalore plot data.csv --kind correlation-heatmap --format json

Compare two datasets:

datalore compare before.csv after.csv --format json
datalore compare before.csv after.csv --key-columns id --format json

Generate structured summary statistics:

./datalore summary data.csv --format json

Generate a structured correlation report:

./datalore correlate data.csv --target revenue --min-abs-correlation 0.3 --format json

Clean and export a dataset deterministically:

datalore clean data.csv --fill-numeric median --drop-duplicates --output artifacts/datasets/data_clean.csv --format json

Apply deterministic transforms without falling back to ad hoc pandas:

datalore clean data.csv \
  --rename sales=>revenue \
  --derive sales_per_customer=>revenue\|/\|customers \
  --filter 'customers>=24' \
  --limit 100 \
  --output artifacts/datasets/data_prepared.csv \
  --format json

Write the full structured result to disk for downstream tooling:

datalore profile data.csv --format json --report-file artifacts/reports/profile.json

Find local datasets, including gitignored files in the repo:

datalore files . --format json

Inspect the runtime/install state:

datalore doctor --format json

Initialize a non-repo project for Codex and Claude Code:

datalore init
datalore init --agent codex

Command Surface

datalore inspect

• Quick shape, dtype, missing-value, duplicate, and preview summary.

datalore profile

• Adds per-column detail on top of inspect.

datalore regression

• Runs a deterministic linear regression using numeric columns only.

datalore summary

• Emits machine-readable descriptive statistics for numeric and categorical columns.

datalore correlate

• Emits a machine-readable correlation matrix, strongest pairs, and optional target-column ranking.

datalore clean

• Applies deterministic cleaning and export operations such as filter, derive, rename, select, fill, drop NA, dedupe, sort, limit, and standardize.
• The transform pipeline runs in a fixed order: rename -> derive -> filter -> select -> fill -> drop NA -> dedupe -> standardize -> sort -> limit.

datalore compare

• Compares two datasets for shape, schema, dtype, missing-value, duplicate, and row-level changes. With --key-columns, it can report changed rows keyed by business identifiers.

datalore files

• Discovers local dataset files by walking the workspace, including gitignored inputs that rg --files would skip.

datalore plot

• Generates histogram, scatter, line, bar, or correlation heatmap artifacts.

datalore doctor

• Reports interpreter, artifact-root, and dependency status for install/runtime debugging.

datalore init

• Scaffolds AGENTS.md, CLAUDE.md, a local ./datalore wrapper, artifacts/, and an artifacts/ ignore rule in the current project.

Reliability

• JSON output uses a stable envelope with schema_version, command, result, and error
• every command supports --report-file for downstream automation
• datalore works from any folder after installation
• ./datalore prefers the repo venv automatically when developing inside this repo
• datalore doctor gives a stable diagnostics entrypoint when the runtime is suspect
• startup no longer crashes on --help if scientific dependencies are missing
• CI runs editable install, distribution builds, and the CLI test suite on multiple Python versions via .github/workflows/ci.yml

Agent-Friendly Conventions

In a normal user project, prefer:

./datalore <command> ... --format json

Why:

• JSON output is stable and easy to parse
• exit codes are predictable
• artifact paths are explicit
• the analysis method is fixed rather than improvised
• --report-file gives you a stable on-disk result for chained workflows
• datalore files avoids the common failure mode where the dataset is gitignored and invisible to rg --files
• datalore init gives Codex and Claude Code repo-local instructions plus a reliable local wrapper in the user's own data project

Inside the Datalore repository itself, contributors can still prefer:

./datalore <command> ... --format json

Repo-native instructions are included for both ecosystems:

• AGENTS.md for Codex
• CLAUDE.md for Claude Code
• .agents/skills/ for Codex skills
• .claude/skills/ for Claude Code skills

These instructions tell agents when to use Datalore instead of writing ad hoc analysis code.

Artifacts

Generated outputs go under artifacts/ by default:

• artifacts/plots/
• artifacts/datasets/
• artifacts/reports/
• artifacts/scripts/

Override the root with:

export DATALORE_ARTIFACT_DIR=/path/to/artifacts

Publishing

The repository is set up for PyPI Trusted Publishing through GitHub Actions.

Release flow:

git tag v0.4.1
git push origin v0.4.1
gh release create v0.4.1 --generate-notes

Before the first release, create or register the datalore-cli project on PyPI and add a Trusted Publisher that matches:

• owner: micic-mihajlo
• repository: datalore-cli
• workflow: .github/workflows/publish.yml
• environment: pypi

If the project does not exist yet on PyPI, use PyPI's pending publisher flow for the first release. After that, publishing happens from GitHub Actions without local API tokens.

Testing

Run the CLI test suite:

.venv/bin/python -m unittest discover -s tests -v

Smoke test the fixture dataset manually:

./datalore files tests --format json
./datalore inspect tests/fixtures/mini_dataset.csv --format json
./datalore summary tests/fixtures/mini_dataset.csv --format json
./datalore correlate tests/fixtures/mini_dataset.csv --target sales --format json
./datalore regression tests/fixtures/mini_dataset.csv --target sales --predictors marketing,customers --format json
./datalore clean tests/fixtures/mini_dataset.csv --rename sales=>revenue --derive sales_per_customer=>revenue\|/\|customers --filter 'customers>=24' --limit 3 --format json
./datalore clean tests/fixtures/mini_dataset_changed.csv --fill-numeric median --output artifacts/datasets/mini_clean.csv --format json
./datalore compare tests/fixtures/mini_dataset.csv tests/fixtures/mini_dataset_changed.csv --key-columns month --format json
./datalore doctor --format json

Smoke test the installed package from outside the repo:

mkdir -p /tmp/datalore-demo
cp tests/fixtures/mini_dataset.csv /tmp/datalore-demo/
cd /tmp/datalore-demo
datalore init
./datalore files . --format json
./datalore profile mini_dataset.csv --format json