Storage Researcher System

Database compression system evolving per-column encoding recipes and a molecular (DNA) storage layer.

Applies the Main Researcher System (pattern extraction → combinatorial recombination → dual evaluation → recursive archive) to structured database compression and molecular (DNA) storage encoding. This repository provides a general-purpose, production-ready CLI for compressing arbitrary CSVs, alongside the full evolutionary research code.

Read the full writeup in paper.md for related work, system design, and results.

🚀 Key Results (Real, Tested Data)

• 18.07x Compression: Tested on a 300,000-row (130 MB) synthetic database. Evolved optimal per-column encodings and verified byte-exact on every schema column.
• 10.0x General-Purpose CLI Compression: Zero hardcoded schemas. Automatically measures and applies the best encoding per column on an arbitrary held-out 50,000-row CSV.
• Biochemically-Valid DNA Encoding Layer: Output is successfully re-encoded into a base sequence (homopolymer ≤3, GC ~50%) featuring erasure-coded redundancy. Demonstrated 100% fountain-decode success across repeated 10%-droplet-loss trials.

🛠️ Quick Start

Install required dependencies:

pip install zstandard numpy

Self-Test & Production CLI

# Run the block-level self-tests (round-trips every encoder/decoder)
python3 blocks.py

# Compress any CSV — auto-detects the best encoding per column
python3 pipeline.py compress your_data.csv out.mrsc

# Compress with the DNA molecular encoding layer
python3 pipeline.py compress your_data.csv out.mrsc --molecular

# Decompress and verify
python3 pipeline.py decompress out.mrsc restored.csv
python3 pipeline.py inspect out.mrsc

Reproduce the Research Run

Reproduce the full evolutionary search (evolutionary search -> full-scale apply -> moonshot layer -> report), phased so each step is checkpointed:

python3 run_research.py gen
python3 run_research.py search
python3 run_research.py finalize
python3 run_research.py moonshot
python3 run_research.py report

📂 Architecture & Files

🧬 What "Gram-Scale" Actually Means

The --molecular flag produces a real, tested, round-trip-verified DNA base sequence in software. It does not synthesize physical DNA — that requires wet-lab equipment beyond the scope of this repository. Every gram figure mentioned is calculated arithmetically against published density figures (see paper.md §2.2 and §6), based on the compressed byte count.