cubest

English · 简体中文 · Español · हिन्दी · العربية · বাংলা · Português · Русский · 日本語 · ਪੰਜਾਬੀ

7-22× fewer tokens per tool response (measured across 7 real scenarios, reproducible via examples/run_all.sh). A single-pass OLAP aggregator that turns any text stream — code, logs, CSV, JSONL, XML, HTML, SDD artefacts — into a compact cube. Built for Claude Code, Cursor, Codex, Aider, Windsurf, Cline, Continue.dev and any tool-calling agent that pays per input token.

🧠 Why an AI agent should care

Measured across 7 realistic scenarios (see examples/):

Cubest wins on large streams and hierarchical data. On very small tabular data (300-row CSV) a plain awk chain is already compact enough, and cubest actually loses. Where it matters — logs, code trees, sitemap crawls — 5-25× fewer tokens land in the agent's context.

At $3–15 per million input tokens (Claude Sonnet 4.6 / Opus 4.7) and 1000 agent sessions per day, that's thousands of dollars per month saved on tool-response ingestion alone — and long sessions stop hitting the context wall. Run examples/run_all.sh yourself.

⚡ Before / After

Question the agent needs to answer: "What does this project do, what endpoints does it expose, and where's the tech debt?"

# ❌ Before — the agent burns 30-60k tokens on raw files:
find . -type f -name '*.py' | xargs cat        # 40k tokens
grep -rn 'TODO\|FIXME' .                        # 8k tokens
grep -rn '@app\|@router' .                      # 3k tokens

# ✅ After — one Python file, three OLAP cuts, 200 tokens back:
cubest --profile file_tree .
cubest --profile api_routes .
cubest --profile tech_debt .

🎯 What it actually does

One Python file (cubest.py, ~1800 lines, PyYAML optional) that:

1. Streams a text source — files, directories, .gz archives, stdin
2. Extracts records via regex or one of 10 built-in presets
3. Aggregates them into an in-memory hierarchical OLAP cube (dimensions × measures: count, sum, avg, min, max, p50, p90, p95, p99 via reservoir sampling)
4. Renders the cube in one of 13 formats — from compact tree to standalone interactive HTML dashboard

Zero database. Zero LLM. Zero tree-sitter. Zero external services. Just python3 cubest.py --profile ... <path>.

🚀 Install

# Option A — pip from PyPI (PyYAML bundled)
pip install cubest
cubest --profile file_tree .

# Option B — npx from npm (thin wrapper, delegates to python3)
npx cubest --profile file_tree .

# Option C — plain download, single file, no venv
curl -O https://raw.githubusercontent.com/BaryshevS/cubest/main/cubest.py
pip install PyYAML          # only needed for YAML profiles / output
python3 cubest.py --profile file_tree .

# Option D — ephemeral via uv, no venv, no install
uv run --with pyyaml \
  https://raw.githubusercontent.com/BaryshevS/cubest/main/cubest.py \
  --profile file_tree .

pip install cubest and npx cubest pull PyYAML automatically. The curl variant runs standalone but needs PyYAML for YAML profiles.

🔌 Install once — use in every AI agent

Every block below installs cubest into the user-global skill space of the harness so it becomes available across all your projects, not just the current one. No project-scoped copies, no per-repo setup.

Step 0 — the binary itself. Every harness spawns a shell, so cubest just needs to be on $PATH once:

pip install cubest        # PyPI (bundles PyYAML)
# or
npm install -g cubest     # npm (thin wrapper, delegates to python3)

Verify: cubest --profile file_tree . should print an ASCII tree.

Claude Code — ~/.claude/skills/cubest/

Claude Code auto-discovers skills in your home directory. One command enables cubest in every session on this machine:

mkdir -p ~/.claude/skills && \
  git clone --depth 1 https://github.com/BaryshevS/cubest \
    ~/.claude/skills/cubest

Claude reads SKILL.md's frontmatter (USE WHEN + 31-profile catalog) and routes to cubest automatically when your prompt matches a trigger.

Verify — in any Claude Code session:

Use the cubest skill to map this repo — top directories × extension × size.

Expected: Claude invokes cubest --profile file_tree .. The skill appears in the sidebar as Skill: cubest.

Cursor — ~/.cursor/rules/cubest.mdc

Cursor loads user-global rules from ~/.cursor/rules/. Create the file:

mkdir -p ~/.cursor/rules && cat > ~/.cursor/rules/cubest.mdc <<'EOF'
---
description: OLAP aggregator for code / logs / CSV / SDD / sitemaps
alwaysApply: false
---
When the user asks for a repo overview, endpoint inventory, log rollup,
CSV pivot, or spec catalog, prefer `cubest --profile <name> <path>`
over chains of grep/cat/find/wc. It streams input in one pass and
returns a compact OLAP tree — 7-22× cheaper in tokens than raw file
content. 31 built-in profiles: `cubest --help`.
Repo & full profile list: https://github.com/BaryshevS/cubest
EOF

Verify — in Cursor chat (any repo):

Map this project structure with cubest — top dirs, extensions, size.

Cursor will propose cubest --profile file_tree . and ask to run it.

OpenAI Codex CLI — ~/.codex/AGENTS.md

Codex CLI reads AGENTS.md from ~/.codex/ (user-global) and merges it with any project-level AGENTS.md. Global install:

mkdir -p ~/.codex && cat >> ~/.codex/AGENTS.md <<'EOF'

## Prefer cubest for aggregation

When aggregating over many files, a long log, or a large CSV, use
`cubest --profile <name> <path>` — it streams input and returns a
compact OLAP tree instead of raw content (7-22× fewer tokens).

Common profiles: file_tree, tech_debt, api_routes, loc_counter,
nginx_access, csv_analytics, mr_impact, seo_audit, k8s_resources,
openapi_endpoints, sdd_specs, call_graph. Full list: cubest --help.

Docs: https://github.com/BaryshevS/cubest
EOF

Verify:

codex "count lines of code per language with cubest"

Codex will propose cubest --profile loc_counter ..

Aider — ~/.aider.conf.yml

Aider reads ~/.aider.conf.yml (user-global) plus any local override. Point it at a cubest hint that all sessions load:

mkdir -p ~/.aider && cat > ~/.aider/cubest-hint.md <<'EOF'
Prefer `cubest --profile <name> <path>` (or `/run cubest ...`) over
chains of grep/cat/find for repo, log, CSV, or spec inventories.
7-22× cheaper in tokens than raw content. 31 built-in profiles.
https://github.com/BaryshevS/cubest
EOF

# Register it globally so every project sees it
cat >> ~/.aider.conf.yml <<'EOF'

read:
  - ~/.aider/cubest-hint.md
EOF

Verify:

/run cubest --profile file_tree .

Aider passes the tree back into its own context.

Windsurf (Codeium) — ~/.codeium/windsurf/memories/global_rules.md

Windsurf's Cascade reads user-global memories from ~/.codeium/windsurf/memories/global_rules.md. Add a rule:

mkdir -p ~/.codeium/windsurf/memories && \
  cat >> ~/.codeium/windsurf/memories/global_rules.md <<'EOF'

## cubest — OLAP over text streams

When the user asks about repo structure, endpoints, TODOs, logs (incl.
`.gz`), CSV rollups, sitemap taxonomy, or SDD/spec artefacts, prefer
`cubest --profile <name> <path>` over multi-file reads. Aggregation is
7-22× cheaper than raw content in the context window. 31 built-in
profiles: `cubest --help`. Repo: https://github.com/BaryshevS/cubest
EOF

Verify — in Cascade chat:

Show me all TODOs in this repo, grouped by kind and directory.

Cascade will propose cubest --profile tech_debt ..

Cline (VS Code) — VS Code global settings

Cline reads its custom-instructions field from your VS Code user settings. Paste the following into Settings → Cline → Custom Instructions (or settings.json):

{
  "cline.customInstructions": "For repo/log/CSV/SDD inventories, prefer `cubest --profile <name> <path>` — OLAP tree, not raw content. 31 built-in profiles; run `cubest --help` for the list. 7-22× fewer input tokens than grep+cat chains. https://github.com/BaryshevS/cubest"
}

Verify — any Cline session:

Use cubest to show top nginx endpoints by status × p95 latency in logs/access.log.gz.

Continue.dev — ~/.continue/config.json

Continue's config is user-global by default. Add a custom slash-command:

// ~/.continue/config.json  →  extend the "customCommands" array
{
  "customCommands": [{
    "name": "cubest",
    "description": "OLAP over code / logs / CSV / SDD via cubest",
    "prompt": "Pick the cubest profile that best answers: {{{ input }}}. Then propose `cubest --profile <name> <path>`. Profiles: file_tree, code_stats, tech_debt, api_routes, loc_counter, nginx_access, csv_analytics, seo_audit, mr_impact, call_graph, k8s_resources, openapi_endpoints, sdd_specs (31 total)."
  }]
}

Verify:

/cubest map the repository top-level structure

Any other tool-calling agent

Wrap cubest -p '<inline JSON>' <path> as a shell tool. The clever bit: the agent generates the inline JSON profile itself, tailored to the exact user question — no pre-baked prompt per query, no rigid schema. See SKILL.md for the profile grammar.

Universal smoke-test

Regardless of harness, this one prompt proves the wiring works:

Use cubest to show the file tree of the current directory — top directories, extensions, total size. Just run the command and paste the output.

If the agent returns an ASCII tree starting with directory names and count=…, bytes=… per leaf — cubest is installed and reachable. If you see "command not found", the agent's shell can't see $PATH (usually a sandbox / container issue, not cubest).

⚡ Quick start

# Map an unfamiliar repo (30 lines instead of 3000)
cubest --profile file_tree .

# Nginx access.log.gz — top URLs × status × avg duration + p95/p99
cubest --profile nginx_access /var/log/nginx/access.log.gz

# Count lines of code by language (drop-in for scc/tokei/cloc)
cubest --profile loc_counter .

# Approximate call graph → interactive HTML dashboard
cubest --profile call_graph src/ > graph.html && open graph.html

# CSV → OLAP → ECharts dashboard (one HTML file, no server needed)
cubest -p '{
  "dimensions": ["campaign", "device"],
  "measures": [{"name":"impressions","type":"sum","field":"impressions"}],
  "extract": [{"type":"preset","preset":"csv"}],
  "output": {"format":"echarts","chart_type":"sankey"}
}' ads.csv > ads.html

# MR/PR impact map from git diff
git diff --name-only origin/main...HEAD | \
  cubest -F - --profile mr_impact .

📊 What you get

13 output formats — pick the one that matches your audience:

31 built-in profiles — pick or customize:

🧪 Benchmarks

Measured on CPython 3.8, laptop-class hardware (July 2026):

Streaming stays flat-memory — 10 TB of logs is bound by I/O, not RAM.

🔁 Replaces common tools

Not a full replacement, but covers 80% of typical scenarios with one file instead of installing a whole zoo:

👥 Roles

🧑‍🍳 Cookbook — before → after

# Before
find . -name "*.py" -not -path "./venv/*" | xargs wc -l | tail -1
find . -name "*.js" -not -path "./node_modules/*" | xargs wc -l | tail -1
# ...repeat for every language

# After
cubest --profile loc_counter .

# Before
zcat access.log.gz | awk '$9 ~ /^5/' | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

# After
cubest -p '{
  "dimensions":["path_root","status"],
  "measures":[{"name":"hits","type":"count"}],
  "extract":[{"type":"regex","pattern":"\"(?P<method>GET|POST) /(?P<path_root>[^/? ]+)[^ ]* HTTP/[\\\\d.]+\" (?P<status>5\\\\d\\\\d)"}],
  "output":{"format":"flat","top_n":20}
}' access.log.gz

# Before: custom awk that sorts everything and eats RAM, or install GoAccess

# After
cubest -p '{
  "dimensions":["path_root"],
  "measures":[
    {"name":"hits","type":"count"},
    {"name":"p50","type":"p50","field":"duration"},
    {"name":"p95","type":"p95","field":"duration"},
    {"name":"p99","type":"p99","field":"duration"}
  ],
  "extract":[{"type":"regex","pattern":" /(?P<path_root>[^/? ]+)[^ ]* HTTP.* (?P<duration>[0-9.]+)$"}],
  "scan":{"stream":true},
  "output":{"format":"flat","top_n":20}
}' access.log.gz

cubest -p '{
  "dimensions":["campaign","device"],
  "measures":[
    {"name":"impressions","type":"sum","field":"impressions"},
    {"name":"cost","type":"sum","field":"cost"},
    {"name":"cost_p95","type":"p95","field":"cost"}
  ],
  "extract":[{"type":"preset","preset":"csv"}],
  "output":{"format":"echarts","chart_type":"sankey"}
}' report.csv > report.html

cubest --profile seo_audit ./crawl/          # md-table of title/desc/H1/schema
cubest --profile seo_semantic_tree ./crawl/  # interactive sunburst of H1-H6
cubest --profile sitemap_map sitemap.xml     # URL taxonomy treemap

git diff --name-only origin/main...HEAD | \
  cubest -F - --profile mr_impact . \
    -p '{"output":{"format":"md_table"}}' > /tmp/impact.md
gh pr comment ${{ github.event.number }} --body-file /tmp/impact.md

cubest --profile call_graph src/ | dot -Tsvg > graph.svg
# or interactive:
cubest --profile call_graph src/ -p '{"output":{"format":"echarts","chart_type":"graph"}}' > graph.html

# Every endpoint across all OpenAPI YAML/JSON specs
cubest --profile openapi_endpoints ./api/

# Only /admin/* endpoints, output as md-table for a PR comment
cubest --profile openapi_endpoints ./api/ \
  -p '{"filters":["path.startswith(\"/admin\")"],"output":{"format":"md_table"}}'

# Fast diff — what endpoints changed between two branches?
git checkout main && cubest -p openapi_endpoints ./api/ \
  -p '{"output":{"format":"json"}}' > /tmp/base.json
git checkout -   && cubest -p openapi_endpoints ./api/ \
  -p '{"output":{"format":"json"}}' > /tmp/head.json
diff /tmp/base.json /tmp/head.json

💰 Business impact (industry benchmarks)

Figures below are industry benchmarks for observability/AIOps in general (Forrester, Research Square, Rootly 2025, incident.io ROI calc). Cubest doesn't replace Datadog / New Relic — it fills the gap between grep and a data warehouse.

• MTTR reduction: manual log investigation consumes 60-80% of MTTR; aggregation cuts it materially (Forrester: up to 50%)
• Cost savings: at $10k/hour downtime and 60→30-minute MTTR, mid- size enterprise saves ~$250k+/year
• Post-mortem archaeology: 60-90 min → 10-15 min per incident
• LLM token cost: measured 212× at typical repo size
• Warehouse compute: moves work out of BigQuery/Snowflake/Athena (paid per TB scanned) into a local aggregator ($0)

🕳️ Related projects (honest comparison)

GitHub search for the exact combination (single-pass + OLAP + CLI + Python + regex) returned zero direct competitors. Partial overlaps:

🧪 Tests

python3 tests/run_tests.py     # 57 unit tests
python3 tests/bench.py         # quick load test
HEAVY=1 python3 tests/bench.py # 5M records, 200k files (~30s, ~500 MiB RSS)

📜 License & attribution

Apache License 2.0 — see LICENSE and NOTICE.

Attribution requirement (Apache 2.0 §4d): if you redistribute cubest — in derivative works, embedded in your product, as part of a hosted service, container image, CLI wrapper, IDE plugin, agent template — you MUST include the NOTICE file (or its readable contents) preserving the upstream URL:

https://github.com/BaryshevS/cubest

Placement options: a NOTICE / THIRD_PARTY_NOTICES / ATTRIBUTION file in your distribution, your documentation, or an "About" / "Credits" / "Powered by" screen.

🗺️ Roadmap

See ROADMAP.md. Highlights:

• examples/ directory with self-contained scripts + input data + expected output for each cookbook recipe
• Exact percentiles via optional t-digest
• --diff mode comparing two cubes for CI regressions
• Streaming CSV parser for > 1 GB inputs
• More AI-agent integration snippets in examples/agents/

🤝 Contributing

Issues and PRs welcome. For substantive changes please open a discussion first. All contributions are accepted under the Apache 2.0 license.

💖 Support

If cubest saves you tokens in daily agent workflows or shortens an incident, consider sponsoring — it directly funds roadmap items (t-digest, streaming CSV, agent snippets) and infra:

• GitHub Sponsors → https://github.com/sponsors/BaryshevS
• Open Collective → https://opencollective.com/baryshevsv

Even $3/month keeps the lights on. Sponsors get priority on issue triage and are credited in release notes.

⭐ Star this repo

If cubest saves you a chunk of the AI budget or shrinks an SRE incident by an hour — a star helps others find it. That's the whole ask.