flashrec 0.1.0

Features | Quickstart | Documentation | Examples | Architecture | API | Evaluation | FAQ | 简体中文

An inference engine for generative recommendation, based on mini-sglang:
wide beam search over a semantic-ID catalog, executed inside CUDA graphs.

Generative recommendation (GenRec) formulates item retrieval as the generation of semantic IDs (SIDs): short, fixed-depth token sequences that index an item catalog. A typical request decodes 3–5 steps at a beam width of 50–512 or more, with every continuation restricted to the valid-SID catalog. General-purpose LLM engines are optimized for long-sequence, single-path decoding; a wide-beam request occupies the engine, and throughput does not scale with concurrency. FlashRec is designed for this workload.

Features

The engine targets short SID depth, wide beam, and catalog-constrained decoding. Throughput scales with beam width n and concurrency; the illegal-SID rate is 0 under the trie constraint. Measurements are in Evaluation.

Decoding

• CUDA graphs. Beam widths from 50 to 512+ run in captured graphs, including the beam-expansion step. Capture sizes extend to multiples of the configured width, so a wide beam is a single graph replay.
• SID constraint. A fused CUDA kernel (dense or CSR-sparse trie) restricts decoding to a catalog of valid semantic IDs. lm_head is evaluated only over the SID token range. Illegal SIDs are never candidates, so invalid_rate is 0; open-vocabulary baselines leave ~17–27% of beams illegal.
• Scheduling. Requests are admitted into decode waves between steps under a beam-row slot budget, so beam rows from different requests share a step. A radix prefix KV cache with longest-prefix-match scheduling reduces the cost of shared prompts; aging prevents starvation.

Serving

• In-process serving. HTTP, scheduling, weights, and the KV pool share one process and one address space.
• FP8 by default. W8A8 per-channel weights and fp8_e4m3 KV, with fused RMSNorm→FP8, SiLU→FP8, and QK-RoPE+KV-write. On 1.7B-class checkpoints at n ≤ 128 the main gain is weight and KV-cache footprint; at n = 512 the two precisions converge on throughput, where the step is bound by bookkeeping.
• API. Ranked beams on /v1/chat/completions. Deterministic top-k at temperature = 0; Gumbel top-k without replacement above it.
• Profiler. /start_profile / /stop_profile compatible with sglang.bench_serving --profile.

Quickstart

pip install -e .

# Public GenRec checkpoint used in the documentation.
hf download OpenOneRec/OneRec-1.7B --local-dir ./OneRec-1.7B

# Build the SID catalog from an OpenOneRec RecIF-Bench benchmark_data directory.
DATA_DIR=/path/to/OpenOneRec-RecIF/benchmark_data bash scripts/build_catalog.sh

# Serve, constrained to that catalog. Layout is inferred from the tokenizer.
flashrec --serve --model-path ./OneRec-1.7B --port 8000 \
  --beam-width 32 --max-tokens 5 \
  --sid-vocab-file data/catalogs/sid2pid_beamrec_l4.json

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"..."}],
       "n":32,"max_tokens":5,"temperature":0}'

Each beam is returned as one choices[] entry, ranked best-first, with its score under sglext.sequence_score. With --sid-vocab-file the SID layout is inferred from the checkpoint tokenizer. If the catalog is unset, the engine decodes over the full vocabulary (connectivity check only). Wide beam (n ≥ 512) also requires --cuda-graph-max-bs 4096 --batch-slots 4096.

The server binds 127.0.0.1 and has no authentication. Bind a public address only on a trusted network or behind an authenticating proxy.

Runnable offline and HTTP clients: Examples. Flags: Configuration.

Evaluation

Throughput, retrieval metrics, and HuggingFace codebook overlap were measured on an NVIDIA RTX 5090. See Evaluation. Unless noted otherwise, FlashRec runs FP8 with a SID trie; the baselines are open-vocabulary. SGLang-master (PR #31626) and SGLang 0801 (cswuyg/sglang feature/beam_search_update_0801) are separate engines; do not collapse them into one row.

Evaluation covers:

• OneRec: the OpenOneRec RecIF-Bench video task with OneRec-1.7B
• SoHuRec-1.7B / SoHuRec-0.6B: Sohu internal generative-recommendation serving traffic on the corresponding models

Speed-ups are FlashRec relative to that baseline. Do not divide OneRec QPS by SoHuRec QPS (prompt length differs by about 8×).

On OneRec, FlashRec has invalid_rate = 0; open-vocabulary engines leave about 17–27% of beams illegal. The quality gap is whether SID constraint is on, not the engine. Do not cite FlashRec concurrency-1 recall@32 (unique-beam collapse). When citing, state the device, trie vs open-vocabulary, concurrency, and sample count.

OneRec

OpenOneRec RecIF-Bench video, model OneRec-1.7B. 5,000 samples, n=50 saturation (highest completed concurrency per engine).

1.54× vs SGLang 0801, 2.02× vs SGLang-master, 2.21× vs TensorRT-LLM, 7.2× vs vLLM. At n=1000 FlashRec saturates at 4.62 QPS; SGLang-master and SGLang 0801 only hold concurrency 1 (2.5–3.0). Full matrices: Evaluation.

On SoHuRec serving prompts: at n=50, 1.7B FlashRec saturates at 220 QPS and 0.6B at 303 QPS; at n=512, 1.7B FlashRec is ~48 QPS vs 0801 / SGLang-master ~16 QPS; at n=1000 FlashRec saturates at 24.6 QPS (1.7B) and 32 QPS (0.6B), and SGLang-master and SGLang 0801 only hold concurrency 1.

Numerical match vs HuggingFace

Codebook-constrained beam search (no SID trie) against transformers, not retrieval recall. HuggingFace is a BF16 reference; FlashRec is measured in BF16 and in FP8. See Evaluation; commands in Examples.

• OneRec-1.7B (RTX 5090): BF16 matches HuggingFace's best SID at every width (n = 1–512); beam-set overlap 86–92%. FP8 swaps top-1 at n = 1 on a 0.125-nat near-tie; overlap is 80% at n = 20 and 70–76% at n ≥ 50. HuggingFace's best sequence is inside the FlashRec beam from n = 20. The public checkpoint is not FP8-trained; that drop is on-load quantization, not a framework bug.
• SoHuRec-1.7B / SoHuRec-0.6B (FP8-trained, RTX 5090, FP8 decoder GEMM on both sides): beam-set overlap 88–96% / 88–93%; prefill top-1 is 100% on SoHuRec-1.7B and 94% on SoHuRec-0.6B (noisy tail, rank corr 0.641; set overlap still about 90%). n = 512 is the mean of the first 3 prompts (HuggingFace OOMs on a longer remaining prompt). Prefer an FP8-trained checkpoint for production FP8 serving.

Supported models

Qwen3 dense: Qwen3-0.6B / 1.7B / 4B / 8B / 14B, OneRec-1.7B, SoHuRec-1.7B / SoHuRec-0.6B, and Qwen3-based GenRec checkpoints. Architectural parameters (GQA, head_dim, qk-norm, tied embeddings) are read from config.json; validated at the 1.7B scale. Broader dense and MoE coverage is on the Roadmap.

Both BF16 checkpoints (quantized on load to W8A8 per-channel under the default --quantization fp8) and pre-quantized FP8 checkpoints carrying weight_scale are supported; any other --quantization value runs in BF16.

Model size is bound by device memory — roughly up to 14B in FP8 on 32 GB. Sequence length is capped by --max-seq-len (default 4096) and stays inside the checkpoint's native positional range. The engine serves beam search over a semantic-ID catalog.

Architecture

A request is served in the same process that owns the weights and the KV pool: HTTP, scheduling, and the model share one address space.

The module layout follows mini-sglang; at runtime the engine depends on sgl-kernel, flashinfer_python, and triton. Request path, design notes, and module map: Architecture.

Documentation

Documentation · Architecture · API · Configuration · Evaluation · FAQ · Examples · Changelog

Development

Install the hooks once per clone; they then run on every commit and match what CI enforces:

pip install pre-commit && pre-commit install
pre-commit run --all-files     # check the whole tree

Unit tests run on CPU:

python -m pytest

Optional integration checks — live parity against an SGLang beam server, and an accuracy comparison against HuggingFace transformers — are documented in Configuration, along with the profiling/trace interface.

# Codebook-constrained vs HuggingFace (needs CUDA + OneRec-1.7B)
FLASHREC_DIFF_MODEL=./OneRec-1.7B \
  PYTHONPATH=python python -m unittest tests.test_beam_search_diff -v

FLASHREC_DIFF_MODEL=./OneRec-1.7B \
FLASHREC_DIFF_BEAMS=1,20,50,128,512 \
FLASHREC_DIFF_QUANT=bf16 \
  PYTHONPATH=python python -m unittest tests.test_beam_search_diff.TestBeamSearchDiff -v

Numbers: Numerical match vs HuggingFace. FLASHREC_DIFF_QUANT=fp8 (default) is the serving path.

Roadmap

• Broader GenRec model coverage: Llama / Qwen / GLM dense backbones and sparse models such as Qwen3-MoE
• Tensor and expert parallelism for 30B–200B-class MoE and larger dense recommendation models
• NVFP4 and native Blackwell storage
• Reproducible RecIF coverage across video / ad / product, and published PyPI wheels

Contributing

Contributions are welcome. See Contributing for development setup, tests, and the pull-request process. Participation is governed by the Code of Conduct. Vulnerabilities go through the private disclosure process in Security.

Acknowledgements

• SGLang
• mini-sglang — layout reference
• OpenOneRec (Kuaishou) — OneRec-1.7B and RecIF-Bench
• cswuyg/sglang feature/beam_search_update_0801 — beam semantics alignment and accuracy tests

Citation

@software{flashrec,
  title  = {FlashRec: A Wide-Beam Inference Engine for Generative Recommendation},
  author = {Wang, Chongyang and {sohu-mptc}},
  year   = {2026},
  url    = {https://github.com/sohu-mptc/FlashRec}
}

GitHub also exposes this via CITATION.cff.

License

Apache-2.0, see LICENSE.