MooFile

A lightweight, embedded, single-file document store with a developer-friendly query API.
No server. No infrastructure. Just a file and a library.
🦀 Rust core available — 2-24× faster than pure Python.
🧠 On-device autoembedding — local embedding models for semantic search.
🔀 Multi-process friendly — a background worker and a web app can share one file.

from moofile import Collection, count, mean

with Collection("mydata.bson", 
                indexes=["email", "age"],
                vector_indexes={"embedding": 1024},
                text_indexes=["content"]) as db:
    
    db.insert({
        "name": "Alice", 
        "email": "alice@example.com", 
        "age": 30,
        "content": "Machine learning and data science expert",
        "embedding": embedding_vector,   # 1024 floats, from your embedding model
    })

    # Traditional query
    results = db.find({"age": {"$gt": 25}}).sort("age").to_list()
    
    # Vector similarity search
    similar = db.find({}).vector_search("embedding", query_vector, limit=5).to_list()
    
    # BM25 text search
    text = db.find({}).text_search("content", "machine learning", limit=10).to_list()
    
    # Hybrid search — BM25 + cosine, fused with Reciprocal Rank Fusion
    results = db.find({}).hybrid_search("content", "embedding",
                                        "data science", query_vector, 10).to_list()

Prefer not to manage embeddings yourself? Autoembedding runs a local GGUF model on-device, filling in the vector on insert and embedding your query text at search time. It needs the Rust core (pip install moofile ships it for most platforms).

Why MooFile?

Target dataset size: megabytes to single-digit gigabytes.

Sharing a file between processes

Like SQLite, several processes can keep the same file open — the usual setup being a long-running worker that writes and a web app that reads:

# worker.py — writes events forever
with Collection("app.bson", indexes=["kind"]) as db:
    for event in stream:
        db.insert({"kind": "event", **event})

# web.py — reads them, and writes the occasional setting
with Collection("app.bson", indexes=["kind"]) as db:
    recent = db.find({"kind": "event"}).sort("_id", descending=True).limit(50).to_list()
    db.insert({"kind": "config", "theme": "dark"})

Readers pick up new writes automatically, writes are serialized so nothing is lost or interleaved, and duplicate _ids are caught across processes. Best suited to one writer with many readers — writes take a brief exclusive lock, so many simultaneous writers will queue.

Installation

pip install moofile

On Linux (x86_64/ARM64), macOS (Apple Silicon) and Windows (x86_64) this installs the Rust-powered wheel. Elsewhere it installs the pure-Python fallback, which warns at import. See Native install below.

Quick Start

from datetime import datetime, timezone
from bson import Binary

from moofile import Collection

db = Collection("users.bson", 
                indexes=["email", "status"],
                text_indexes=["bio"],
                vector_indexes={"profile_vec": 128})

# Insert — any BSON type: datetimes, binary, ObjectId, Decimal128, nested docs
alice = db.insert({"name": "Alice", "email": "a@ex.com", "age": 30, "status": "active",
                   "joined": datetime(2025, 1, 15, tzinfo=timezone.utc),
                   "avatar": Binary(b"...")})
db.insert_many([...])

# Query — ranges work on dates too
active = db.find({"status": "active"}).to_list()
young  = db.find({"age": {"$lt": 30}}).sort("age").to_list()
recent = db.find({"joined": {"$gte": datetime(2025, 1, 1, tzinfo=timezone.utc)}}).to_list()
one    = db.find_one({"email": "alice@example.com"})

# Vector search
similar = db.find({}).vector_search("profile_vec", query_vector, limit=3).to_list()
for doc, score in similar:
    print(f"{doc['name']}: {score:.3f}")

# Text search
results = db.find({}).text_search("bio", "machine learning", limit=5).to_list()

# Update & Delete
db.update_one({"email": "a@ex.com"}, set={"age": 31})
db.update_many({"status": "trial"}, set={"status": "expired"})
db.delete_one({"email": "c@ex.com"})
db.delete_many({"status": "expired"})

Autoembedding

MooFile can run a local GGUF embedding model on-device, so text is embedded on insert and query text is embedded at search time — no external embedding API.

from moofile import Collection

db = Collection("papers.bson",
    indexes=["year", "category"],
    vector_indexes={"embedding": 1024},
    auto_embed={
        "abstract": {                             # source text field
            "model": "hf:jsonMartin/voyage-4-nano-gguf:voyage-4-nano-q8_0.gguf",
            "target": "embedding",                # target vector field
            "dims": 1024,
            "precision": "int8",                  # f32 | int8 | uint8 | binary
        },
    })

# Insert — auto-embeds abstract → embedding (1 KB/doc at int8)
db.insert({"title": "Quantum ML", "abstract": "Quantum computing for ML...", "year": 2025})

# Semantic search — the query text is embedded with the same model
for doc, score in db.find({"year": 2025}).semantic("abstract", "quantum algorithms", 5).to_list():
    print(f"{doc['title']}: {score:.3f}")

# Hybrid search — pass None for the query vector and the vector leg auto-embeds
db.find({}).hybrid_search("abstract", "embedding", "quantum", None, 10).to_list()

Requires the Rust core. The pure-Python fallback cannot run a GGUF model — it raises NotImplementedError from both auto_embed and semantic(). Everything else works there unchanged.

The same config block is accepted verbatim by every other binding, as JSON:

{
  "vector_indexes": {"embedding": 1024},
  "auto_embed": {
    "abstract": {"model": "hf:jsonMartin/voyage-4-nano-gguf:voyage-4-nano-q8_0.gguf",
                 "target": "embedding", "dims": 1024, "precision": "int8"}
  }
}

The model is downloaded from HuggingFace on first use (~355 MB for Q8_0) and cached. Autoembedding is on by default; building with --no-default-features drops it and ~300 transitive crates (~8.3 MB → ~2.8 MB), after which auto_embed and semantic search return a clear "not available" error and everything else works unchanged.

Native Install (Rust Core)

When the Rust native extension is installed, import moofile transparently uses it — same API, 2-24× faster.

From source (requires Rust)

# Install Rust: https://rustup.rs
curl --proto '=https' --tls v1.2 -sSf https://sh.rustup.rs | sh

# Build and install the native extension — from the REPO ROOT.
# The root pyproject.toml is what points maturin at bindings/python and adds
# the moofile/ package; running maturin inside bindings/python builds a wheel
# containing only the compiled module, with no Python package in it.
pip install maturin
maturin develop --release      # or: maturin build --release

Prebuilt wheels

GitHub Actions builds one abi3 wheel per platform on tag push — a single wheel that works on every CPython from 3.10 up, rather than one per minor version:

Anything else — musl/Alpine, Intel macOS — gets the pure-Python wheel, which has no autoembedding and is several times slower. That fallback emits a RuntimeWarning at import naming the reason; set MOOFILE_PURE_PYTHON=1 to silence it if you are on it deliberately.

CLI Tools

moosh users.bson --indexes email,age
moo2json users.bson users.json
moo2json --import users.json users.bson --indexes email
moo2mongo users.bson --uri mongodb://localhost/mydb --collection users
moo2sqlite users.bson users.db --table people

Full Documentation

• Specification — file format, architecture, design decisions
• API Reference — complete Python API, filter operators, aggregation
• Language Bindings — C, C++, Node.js, Go, Java, C#
• Building & Testing — toolchain setup for every language
• bench_native.py — Python vs Rust head-to-head benchmark

Language Bindings

MooFile is implemented in Rust with a Python binding (via PyO3). A C shared library (libmoofile.so) exposes the full API via extern "C" functions, and all other languages consume that:

Plus 8 cross-backend parity scenarios comparing pure-Python, PyO3 and C.

Every binding passes documents as JSON strings across the FFI boundary. The autoembedding feature (local GGUF embedding models) works in every binding — model loading and inference happen entirely inside the Rust core, so the auto_embed config block is identical in all of them. The one exception is the pure-Python fallback, which cannot run a model at all.

See bindings/README.md for build instructions, usage examples, and test results for each language.

Development

Setting up a machine from scratch — including the toolchains for all seven language bindings — is covered in BUILDING.md. Once installed, ./scripts/test-all.sh runs every suite and prints a summary, skipping any language whose toolchain is absent.

# Unit tests (PYTHONPATH=. so you test this checkout, not an installed copy)
PYTHONPATH=. pytest tests/ -v

# Cross-implementation tests — runs both backends
PYTHONPATH=. pytest tests-cross/ -v

# Rust core tests
export PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
cd core && cargo test

# Rust benchmark
cd core && cargo run --example bench --release

# Python vs Rust benchmark
PYTHONPATH=. python bench_native.py

Project layout

moofile/
├── core/                    # Rust engine (cargo build)
│   ├── src/{lib,storage,index,query,text,cache,embed,errors}.rs
│   └── examples/bench.rs    # Pure-Rust benchmark
├── bindings/                # Language bindings — see bindings/README.md
│   ├── python/              # PyO3 binding (maturin build)
│   ├── c/                   # C ABI (cdylib) + C++ header-only wrapper
│   ├── node/                # Node.js via koffi
│   ├── go/                  # Go via cgo
│   ├── java/                # Java via the Foreign Function & Memory API
│   └── csharp/              # C# via P/Invoke
├── moofile/                 # Python package
│   ├── __init__.py          # Auto-detects Rust, falls back to Python
│   ├── _rust_adapter.py     # Adapts Rust NativeCollection → Python API
│   ├── collection.py        # Pure-Python reference implementation
│   ├── query.py, index.py, storage.py, ...
│   └── cli/                 # moosh, moo2json, moo2mongo, moo2sqlite
├── tests/                   # Python test suite
├── tests-cross/             # Cross-implementation validation
├── docs/README.md           # Full Python API reference
├── moofile-spec.md          # File format & architecture spec
└── pyproject.toml           # Python package config

Other languages

Beyond Python, MooFile ships bindings for C, C++, Node.js, Go, Java and C#, all layered on one C ABI (bindings/c). They share the same file format, query language and semantics.

cargo build -p moofile-c --release   # build the shared library first

See bindings/README.md for per-language setup, usage, and the ABI contract (error conventions, ownership rules, no-match semantics).

License

MIT — see LICENSE.