nearlossless-context

A research library for memory-efficient long-context inference under fixed VRAM. It implements critical-span retention, query-unknown discovery, and measured KV-cache quality/peak-memory tradeoffs for Hugging Face models.

Install

pip install nearlossless-context

The distribution name is nearlossless-context; the import package is nearlossless_context. Install a CUDA-enabled PyTorch build appropriate for your system when GPU execution is required.

For development or paper reproduction, install from source:

git clone https://github.com/nilsperssonsuorra/nearlossless-context.git
cd nearlossless-context
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -e .

Quick start

Run the packaged end-to-end example. The first run downloads Qwen/Qwen2.5-0.5B-Instruct (about 0.5B parameters):

nearlossless-example

It automatically uses CUDA when available and otherwise runs on CPU with a shorter prompt. The command reports the original, peak, and retained cache-token counts before printing the generated answer. Override the defaults with, for example, nearlossless-example --device cuda --prompt-tokens 4096.

The equivalent library API is:

from nearlossless_context import greedy_generate, prefill_auto

# Default online path: sticky surface-novelty discovery.
past, logits, info = prefill_auto(
    model,
    input_ids,
    mode="stream",
    tokenizer=tokenizer,
)

tokens = greedy_generate(
    model,
    past,
    logits,
    max_new,
    eos_id=tokenizer.eos_token_id,
    next_position=input_ids.shape[-1],
)

Use discovery="query_hold" for the measured LongBench peak/quality tradeoff, or mode="posthoc" when query-aware scorer quality matters more than peak prefill memory. See USAGE.md for model-family floors, multi-document settings, and generation requirements.

Story in one line

Near-lossless training-free KV compression ≈ keep critical local neighborhoods; online stream fails when query-unknown discovery is weak; sticky surface novelty closes most of that gap on a multi-seed retrieval suite through 40k at flat peak cache ~1k; on public LongBench, posthoc query-aware ≈ full F1 while online quality is a peak/quality Pareto (query_hold ~0.92× full @ ~2.5k peak).

Results (headline)

On multi-seed retrieval (primary Qwen3-4B, RTX 3090):

Default path: prefill_auto(..., mode="stream", discovery="novelty") — see USAGE.md.

What we do not claim

General long-context SOTA, full RULER/LongBench leaderboards, production int8 kernels, or that surface novelty is universal “importance.” Suite is retrieval/needle-class; open long-doc QA is a Pareto, not free lunch at peak~1k. See paper §Limitations.

Goal

Run a small open model (~4B) with much more context than naive inference allows, on consumer hardware, with little or no quality loss vs full key–value (KV) cache on retrieval-critical tasks.

Formally:

Maximize (L_\varepsilon) s.t. quality ≥ ((1-\varepsilon)\times) full-KV on suite (\mathcal{S}), under ≤24 GB and usable speed.

• ε → 0 on retrieval-critical tasks (exact facts).
• Theory + falsification first — not “stack known KV tricks.”

Hardware & model

Workstation policy: long (L) uses streaming (peak cache ~ budget+chunk). Full 40k prefill can thrash WDDM; prefer novelty stream for long jobs.

Experiments

Paper reproduction (core)

python experiments\bench_paper_rigor.py --seeds 0,1,2,3,4 --ctx 4096
python experiments\bench_capsules.py --novelty --oracle-online --skip-capsules --budgets 512
python experiments\bench_novelty_longL.py --ctx 16384,24576,32768,40960 --arms novelty --budgets 512
python experiments\bench_external_slice.py --source longbench --n 60 --arms full,posthoc --budgets 512,1024,2048
python experiments\plot_paper_figures.py

Outputs: results/*.csv + *.json (gitignored). Narrative: results/FINDINGS.md.

Status

Core arc is measured and written up: mechanism → discovery gap → sticky novelty → long-(L) + stresses → public LongBench decomposition (posthoc UB + query_hold Pareto).

This is an experimental alpha API intended for research and evaluation. It is not yet a production serving backend. Interfaces and measured operating points may change as integration work expands.

Roadmap

The project is moving from a reproducible research prototype toward an integration-ready long-context inference library. Unchecked items are planned work, not claims about current capabilities.

Near term

• Validate wheel and source-distribution installation in clean environments
• Publish the package to PyPI after TestPyPI verification
• Expand unit tests and CPU-compatible public-API smoke tests
• Stabilize the compression-policy interface and runnable examples
• Add lightweight reproducibility checks to CI

Integration work

• Prototype integration with a vLLM-compatible KV-cache or attention extension point
• Investigate an SGLang integration path
• Add real quantized KV storage and kernel support
• Extend evaluation across additional models, context lengths, and public datasets

Longer term

• Add scheduled GPU benchmarking with regression tracking
• Evaluate batching, concurrency, and production-serving constraints
• Prepare upstream draft PRs once interfaces and benchmarks are sufficiently mature

Priorities may change as benchmark evidence and upstream interfaces evolve. Contributions and reproducible issue reports are welcome.

Docs

Related work is crowded (eviction + KV quant). Success here is raising (L) at ε≈0 with honest multi-seed claims and a clear discovery diagnosis—not renaming SnapKV.

Repo layout

nearlossless-context/
  experiments/     # benches + novelty + adaptive API
  papers/          # main.tex / main.pdf / figures
  results/         # FINDINGS.md (+ local CSVs gitignored)
  USAGE.md
  requirements.txt

GitHub: nilsperssonsuorra/nearlossless-context

License

The software in this repository is licensed under the Apache License 2.0. The paper and figures are copyright © 2026 Nils Persson Suorra and licensed under CC BY 4.0. Models, datasets, and other third-party materials remain subject to their respective licenses.