fastjev 0.1.1

English | 简体中文

Deploy an open source Jev implementation on your own infrastructure.

FastJev is an open source implementation of Jev for deployment on infrastructure you control. It continues SemIf as an independently maintained fork. Pinned open models run the Choice, Boolean, and Score interface through Torch, vLLM, MLX, and llama.cpp. The WebGPU demo provides browser local inference.

Why FastJev?

FastJev turns Jev deployment into a standard Python workflow. The SDK loads the model, validates 2 to 16 options, performs scoring, and returns typed results with probabilities, token usage, timing, model revision, and prompt version.

Resident Torch, batched vLLM, native MLX, llama.cpp GGUF, the CLI, and the optional System One compatible HTTP API share the same result model. Torch, MLX, and llama.cpp return results in one scoring pass with zero output tokens.

Each result records the model revision and prompt version. Published evaluations add row data, raw timings, and checksums for reproducibility.

Models you can run now

The same native BF16 direct logit interface has been validated on these pinned checkpoints. Put any listed model ID and revision into the quick start below.

The table reports native BF16 checkpoint quality. Browser artifacts use separate quantized formats. Their smoke results, exact rows, and revisions are in the model ladder report.

Quick start

The default backend requires Python 3.10+, CUDA, and exactly one visible GPU. Qwen3.5-4B uses about 9 GB of disk for the source checkpoint and measured 7.891 GiB peak allocated GPU memory in the historical RTX 5090 SDK smoke run.

python -m venv .venv
. .venv/bin/activate
export HF_HOME=/path/to/large-drive/huggingface
pip install 'fastjev[torch]'

The first call automatically downloads the pinned model revision from Hugging Face and caches it under HF_HOME.

from fastjev import Choice, FastJev, Option

with FastJev.from_pretrained(
    "Qwen/Qwen3.5-4B",
    revision="851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
) as jev:
    result = jev.decide(
        state={"message": "I was charged twice and need a refund today."},
        question=Choice("Which queue should handle this request?", [
            Option("access", "Account access and authentication."),
            Option("billing", "Billing, payments, and refunds."),
            Option("sales", "Pricing and new contracts."),
        ]),
    )

print(result.value)
print(result.probabilities)
print(result.provenance)

Remote models use an immutable Hugging Face revision containing 40 characters. Local model directories use a descriptive revision label for result provenance.

Install fastjev[llama-cpp] to run GGUF files from disk or Hugging Face. The Python SDK guide covers llama.cpp setup and provenance.

Install the latest source

Use an editable checkout to run the latest code from main.

git clone https://github.com/chengyongru/fastjev.git
cd fastjev
pip install -e '.[torch]'

Measured results

RTX 5090 with Torch and vLLM

An integration run on one host used the pinned Qwen3.5-4B checkpoint, identical inputs for three questions (125, 152, and 151 tokens), one warmup, and seven measured decide_many calls.

On this RTX 5090 / WSL workload, vLLM delivered 1.96× Torch throughput and 48.9% lower median batch latency. Its additional 30.38 seconds of startup cost breaks even after roughly 383 batches of three decisions when the process stays resident. Both backends made the same three selections.

PR #7 records this historical integration measurement, including the exact environment and aggregate medians. The repository's reproducible benchmark bundle covers separate experiments with committed row data.

Decision quality

The first two rows report balanced accuracy. The third reports modal agreement with equal weighting across cases. The Jev column reproduces public records. The two open model columns come from local frozen evaluations. The results report provides the full data, perturbations, and claim boundaries.

How it compares

These projects serve adjacent deployment needs. Their published performance belongs to their own workloads.

Backends and interfaces

Use fastjev-score for JSONL jobs. Install fastjev[api,torch] and run fastjev-serve for POST /v1/systemone and GET /v1/models. The SDK guide covers batching and custom backends. The HTTP guide covers server setup, authentication, and compatibility boundaries.

Documentation

The results report covers speed, quality, perturbations, and limitations. The method guide documents frozen prompts, metrics, and timing scope. The reproduction guide provides pinned environments and verification commands. The benchmark bundle contains fixtures, runners, and source selection. The interactive replay and WebGPU browser demo provide visual ways to explore the project.

Boundaries and provenance

FastJev returns probabilities conditioned on the supplied options with calibrated=False. Deployment validation and calibration establish thresholds for consequential automation. The experimental shared prefix modes can change close BF16 argmax results.

Model weights stay on upstream hosts, and third party evaluation records stay with their original sources. Upstream models retain their licenses. Exact revisions are listed in THIRD_PARTY.md.

The third party record documents source and model provenance.

Project code is released under the MIT License.