DataMine Lib

Shared data-mining library for the causal research suite — one facade that profiles columns, mines associations, extracts KPIs (tabular + optional vision), exports morpheme catalogs, runs auto feature engineering (propose → build → test), and emits markdown/JSON reports.

Package: datamine · PyPI-style name: datamine-lib · version 0.5.0

from datamine import DataMiner, MineReport, AutoFeatureEngine, auto_features, library_help

miner = DataMiner.from_csv("x.csv", prefer_autocausal=False)  # also from_df, from_sqlalchemy, from_public_join
report = miner.profile().associate().kpi().morphemes().insights().report()
# or: report = miner.run()

print(report.to_markdown())
ac = miner.to_autocausal()   # optional, if AutoCausalLib installed
civ = miner.to_causaliv()    # CausalIV-shaped KPI catalog dict

# Auto feature engineering (miners → SLM guide → coding agent → test)
miner.auto_features(
    target="conversion",
    research_problem="What drives conversion?",
    use_slm=False,
    use_causal_adapter=True,  # optional causal-heuristic seeds (≠ identification)
    coding_llm="builders",    # offline; or "auto" (cloud → local → builders)
    coding_harness="sys",     # offline; or "auto" (openclaw → hermes → sys)
)
print(miner.report().auto_features["kept"])

# or convenience → AutoFeatureReport
from datamine import auto_features
fe = auto_features(
    df, target="conversion", research_problem="...", use_slm=False,
    use_causal_adapter=True, coding_llm="builders", coding_harness="sys",
    prefer_autocausal=False,
)
print(fe.to_markdown())

print(library_help(module="features"))

Exploratory FE / association testing ≠ causal identification. Significance vs a target is associational evidence only. Causal heuristics (when enabled) seed proposals — they are not causal identification.

See docs/auto_features.md for the full pipeline and call path.

Install

cd research/DataMineLib
pip install -e ".[dev]"

# optional extras
pip install -e ".[vision]"   # opencv
pip install -e ".[sql]"      # sqlalchemy
pip install -e ".[fe]"       # sklearn, scipy, statsmodels, nltk, gensim, spaCy, transformers
pip install -e ../AutoCausalLib   # preferred tabular backend
pip install -e ../CausalIVSuite   # preferred KPI backend

Pipeline overview

Classic mine
  profile → associate → kpi → morphemes → insights → MineReport

Auto features
  miners (+ optional causal heuristics / morpheme+vision seeds)
       → optional SLM guide (default off)
       → coding harness (openclaw|hermes|sys) + coding LLM
       → significance / keep-reject (+ optional refine-on-failure)
       → adaptive multi-round on augmented work frame → AutoFeatureReport

CLI

python -m datamine --help
python -m datamine help --all
python -m datamine help --module features
python -m datamine help --api
python -m datamine help --cli

python -m datamine mine --csv path/to/data.csv
python -m datamine mine --csv path/to/data.csv --json -o report.json
python -m datamine mine --csv data.csv --with-features --target conversion
python -m datamine guides
python -m datamine adapters

# Auto features: propose → build → significance-test (offline-safe flags)
python -m datamine features --csv path/to/data.csv --target conversion \
  --problem "What drives conversion?" --causal-heuristics \
  --coding-llm builders --coding-harness sys --show-backends \
  --feature-frame-out feats.csv --max-refine 1
python -m datamine features --parquet data.parquet --target y
python -m datamine pipeline --csv data.csv --target y --coding-llm builders
python -m datamine features --csv data.csv --target y --json -o fe.json

Examples

Runnable offline demos under examples/:

Script	Pipeline
examples/01_classic_mine.py	Fluent classic mine
examples/02_auto_features.py	Full auto-features (builders + sys)
examples/03_causal_heuristics.py	+ use_causal_adapter=True
examples/04_coding_harness.py	coding_llm="auto" / coding_harness="auto"
examples/run_cli_examples.ps1	CLI walkthrough (Windows)

python examples/01_classic_mine.py
python examples/02_auto_features.py
python examples/04_coding_harness.py --force-builders

Auto feature engineering

Pipeline: mine associations / association rules / KPIs → advanced statistical miners (+ optional causal heuristics adapter) seed proposals → optional SLM re-rank → coding agent LLM (OpenAI / Claude / Grok / local / builders) → whitelist builders → significance + distribution tests → iterate.

Coding LLM (coding_llm="auto"): cloud if API key present → best on-device → deterministic builders-only. Keys via env (OPENAI_API_KEY, ANTHROPIC_API_KEY, XAI_API_KEY / GROK_API_KEY, DATAMINE_LLM_*) — never hardcode.

Coding harness (coding_harness="auto"): OpenClaw → Hermes → built-in sys. Soft-degrades when externals are missing. See docs/auto_features.md and docs/coding_harness.md.

Advanced miners (soft deps; sources on FeatureProposal):

source	what it does
mi	Mutual information / dependence ranks (sklearn)
anova	ANOVA / Kruskal for categorical↔numeric
cramers_v	Chi-square / Cramér's V for categorical–categorical
partial_corr	Partial / residualized associations
dist_transform	Skew/kurtosis → log / rank / bin
outlier	Extreme-value indicator flags
stratified_lift	Level lift vs base rate
vif	Multicollinearity notes → ratio proposals
causal_heuristic / autocausal_adapter	Optional causal-role heuristics (treatment/confounder/…) — seeds only, ≠ identification

Also: correlation pairs, association rules (equals_value for target-linked rules), research-problem term overlap, NLP term features. Enable causal seeds with use_causal_adapter=True or --causal-heuristics.

Environment variables

Variable	Role
OPENAI_API_KEY	OpenAI coding LLM
ANTHROPIC_API_KEY	Claude coding LLM
XAI_API_KEY / GROK_API_KEY	Grok coding LLM
DATAMINE_LLM_API_KEY + DATAMINE_LLM_BASE_URL	OpenAI-compatible endpoint
DATAMINE_LLM_MODEL	Model id for compatible endpoint
DATAMINE_CODING_LLM	Default provider (auto / builders / …)
DATAMINE_CODING_HARNESS	Default harness (auto / sys / …)
DATAMINE_OPENCLAW_BIN / DATAMINE_HERMES_BIN	External harness CLI paths
DATAMINE_OPENCLAW_CMD / DATAMINE_HERMES_CMD	Optional command templates
DATAMINE_LOCAL_LLM_MODEL	Local on-device model id

Full table: docs/coding_harness.md · docs/auto_features.md.

Guides

datamine.guides detects installed suite packages and recommends which miner/backend to use (AutoCausal, CausalIV AutoKPI, VisionKPI, NextFrameSeq image_mine, MorphemeStudio, EquityIV, FactorIV, CausalSearch).

Product adapters

Thin optional adapters live in each product as <pkg>.datamine_adapter. Soft-import datamine — products keep working without it. See docs/ADAPTERS.md.

Docs

• auto_features.md — propose → SLM guide → coding harness/LLM → test + causal heuristics + env vars
• coding_harness.md — OpenClaw / Hermes / sys harness
• ARCHITECTURE.md
• ADAPTERS.md
• examples/README.md — examples index

Help

python -m datamine --help
python -m datamine help --all|--module features|--api|--cli

from datamine import library_help
print(library_help(all=True))
print(library_help(api=True))

Catalog covers: DataMiner, auto_features, AutoFeatureEngine, CodingAgent, harness, causal heuristics.

Tests

pip install -e ".[dev]"
pytest -q

CausalBridge

Registered as datamine in the CausalBridge catalog. Workflow:

python -m causalbridge workflow datamine_all --dry-run

GitHub

https://github.com/ehallford11714/datamine-lib