Balarama

Balarama is a local in-memory hybrid analytical index for Python.

The core idea is simple:

• keep rows as the source of truth
• build sorted per-column access paths
• load data once
• run many repeated local queries and analytics very quickly

The primary use case now is a code corpus slice explorer for AI-assisted software development. You load function metadata once, then repeatedly filter by language, repo, split, code_len_chars, doc_len_chars, code_line_count, and similar fields for corpus prep, eval slicing, and prompt/example selection.

What Works Today

The implemented path is centered on the original hybrid design:

• native C table engine with one row store and one sorted index array per column
• bulk-friendly load-once path that appends rows first and builds indexes once
• Python HybridTable API for equality/range filtering, counts, sums, row reconstruction, and value counts
• higher-level CodeCorpusExplorer API for the CodeSearchNet metadata workflow
• sample code corpus demo that runs locally from the repo
• public CodeSearchNet metadata benchmark path with DuckDB as the first external local comparator

The earlier vector-search path is still in the repo as an experiment, but it is no longer the main product story.

Quick Example

from pathlib import Path

import balarama as br

csv_path = Path("examples/data/code_corpus_sample.csv")
explorer = br.CodeCorpusExplorer.from_csv(csv_path)

python_tests = explorer.examples(
    language="python",
    split="test",
    has_docstring=True,
    limit=3,
)

docstring_rich_train_chars = explorer.sum_code_length(
    language="python",
    split="train",
    has_docstring=True,
    min_doc_len=40,
)

top_repos = explorer.top_repos(
    has_docstring=True,
    min_doc_len=20,
    top_n=5,
)

Run It

Install the editable package into a virtualenv:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .

For a standard release install after PyPI publish:

python -m pip install balarama

Run the code corpus sample app:

make demo-code-corpus

Run the full test suite:

make test

Run the public CodeSearchNet metadata benchmark:

make benchmark-codesearchnet-hybrid

The benchmark prepares a public CodeSearchNet metadata CSV, loads it into Balarama’s hybrid table engine, and compares repeated slice/count/sum/example/top-repo workloads against DuckDB on the same corpus.

Why This Direction

The strongest product story here is not “another database” and not “another vector store”.

It is:

• load a large structured dataset once
• keep it in memory
• slice it repeatedly with indexed filters
• reconstruct rows quickly
• do this locally from Python without running a service

That is a better match for the original architecture and for real engineering workflows around dataset prep, eval slicing, and example selection.

Current Strengths We Intend To Preserve

• contiguous in-memory data structures
• explicit ownership and predictable behavior
• sorted per-column access paths
• stable rowid bridging rows and indexes
• benchmarkability
• small, understandable APIs

Non-Goals For The Main Product Story

• no hosted service
• no distributed clustering
• no mutable operational database story
• no ANN-first or vector-first product identity
• no broad SQL engine ambitions

Repository Guide

• docs/product/hybrid_load_once_vision.md: the reset vision for the original hybrid design
• docs/research/hybrid_use_case_research.md: use-case shortlist and the recommended code-corpus story
• docs/research/hybrid_storage_landscape.md: research on related hybrid row/column systems and where Balarama fits
• docs/product/quickstart.md: the main quick start for the hybrid Python path
• docs/product/code_corpus_slice_explorer.md: user-facing walkthrough of the code corpus explorer
• docs/product/python_api_reference.md: Python API reference for the hybrid and explorer APIs
• docs/product/go_to_market_plan.md: sequential launch plan for the current library and product surfaces
• docs/product/repo_launch_surface.md: recommendation for the clean launch-facing repo surface and what remains experimental
• docs/product/codex_plugin_plan.md: detailed design, implementation, testing, and docs plan for the Codex plugin
• docs/product/pypi_release_runbook.md: concrete same-day release steps for GitHub plus PyPI
• docs/product/release_notes_v0_1_0.md: release note for the first public Python launch
• docs/product/example_output.md: representative output from the user-style sample program
• docs/benchmarking/public_dataset_strategy.md: recommended public datasets and why they fit
• docs/benchmarking/codesearchnet_hybrid_results.md: latest one-million-row public benchmark results for the hybrid path
• docs/benchmarking/duckdb_gap_plan.md: the concrete plan to close the performance gap with DuckDB on the right benchmark family
• docs/research/codex_productization_strategy.md: recommendation for packaging Balarama into Codex as a plugin plus skill plus MCP server
• benchmarks/python/README.md: benchmark entry points and artifact locations
• docs/product/marketing_copy.md: short product copy for posts and repo blurbs
• docs/product/linkedin_post.md: outward-facing post for the current product story
• docs/architecture/design_and_principles.md: hybrid-table design principles
• docs/architecture/detailed_design.md: current table/data-structure details
• docs/architecture/hybrid_storage_layers.md: diagrams and explanation of the row store, sorted column indexes, and future hot column arrays

Experimental vector-path docs remain available here:

• docs/product/vector_retrieval_direction.md
• docs/product/v1_collection_api.md
• docs/architecture/vector_engine_design.md