nlp-components-library

Reusable inference, analysis and configuration components for NLP and LLM experiment pipelines, extracted from the multi-turn dice roll, UnBLEnD, decision-model psychometrics and human-disagreement projects so new experiments can be assembled from tested parts.

Installation

uv add "nlp-components-library[hf]"      # or: pip install "nlp-components-library[hf]"

The core (config, analysis, data, io) needs only numpy, polars, pydantic and scipy. Extras add the heavier parts:

Usage

Models

from nlp_components_library.config import models

for spec in models.resolve(["dice-core", "qwen3_14b"]):  # presets, keys or Hub ids
    if spec.fits(gpu_gb=48):
        print(spec.key, spec.hf_id, spec.bf16_gb)
print(models.table(models.resolve(["human-disagreement"])))

The same listing is available as nlp-components-library models dice-core and nlp-components-library presets.

Exact option probabilities

from nlp_components_library.inference import NextTokenScorer, build_chat

scorer = NextTokenScorer.from_pretrained("Qwen/Qwen3-8B", dtype="bfloat16")
chat = build_chat(
    "Roll a six-sided die. Return exactly one number from 1 to 6. Answer: "
)
(result,) = scorer.score([chat], options=("1", "2", "3", "4", "5", "6"))
result.probs  # raw probability of every option (all matching tokens summed)
result.option_mass  # share of the full distribution on the options
result.normalized()  # renormalised over the options
scorer.metadata()  # model revision, template hash, token ids, versions, ...

Every option sums the probability of all vocabulary tokens that decode to it ("4", " 4", "▁4", ...), normalised in float64 over the full vocabulary. Batches never pad (sequences are bucketed by length), so results do not depend on the batch size. APIScorer offers the same interface for OpenAI-compatible servers, where only the top-20 log-probabilities are known.

Logit lens and reward models

from nlp_components_library.config import models
from nlp_components_library.inference import (
    ChatTemplate,
    LogitLens,
    RewardScorer,
    build_chat,
    load_spec,
)

tokenizer, model = load_spec(models.BY_KEY["llama3_8b"])
lens = LogitLens(model, ChatTemplate(tokenizer), options=("A", "B", "C", "D"))
output = lens.score([build_chat(prompt) for prompt in prompts])
output.probs  # [item, layer, option]; output.predictions is the last layer
output.hidden_states

tokenizer, rm = load_spec(models.BY_KEY["skywork_qwen3_8b"])
RewardScorer(rm, ChatTemplate(tokenizer)).score_responses(prompt, ["A", "B", "C", "D"])

load_model/load_spec refuse checkpoints that would leave weights randomly initialised (e.g. a text-only class on a vision-language checkpoint).

Persona questionnaires

from nlp_components_library.config import Questionnaire
from nlp_components_library.data import sample_personas
from nlp_components_library.inference import (
    ChatTemplate,
    ContinuationScorer,
    QuestionnaireScorer,
)

segment = Questionnaire.from_json("instruments/big5.json").sole_segment
personas = sample_personas(
    "hf://datasets/nvidia/Nemotron-Personas-USA/data/train-*.parquet", 100
)
scorer = QuestionnaireScorer(
    ContinuationScorer(model, tokenizer),
    template=ChatTemplate(tokenizer),
    history_window=5,
)
responses = scorer.score(personas["persona"].to_list(), segment)

Each statement is scored by the log-likelihood of every response option; the most likely answers of the last history_window statements are shown with the next one, and personas beyond max_context_tokens are truncated.

Scoring, aggregation and statistics

from nlp_components_library.analysis import (
    aggregate,
    bootstrap_ci,
    kruskal,
    score_probabilities,
)
from nlp_components_library.config import macro_region_map
from nlp_components_library.data import load_items

items = load_items(
    "blend.json", ("A", "B", "C", "D"), annotations={"macro_region": macro_region_map()}
)
scores = score_probabilities(
    items, probs, ("A", "B", "C", "D"), lens=lens_table, model="llama3_8b"
)
aggregate(
    scores, ["model", "macro_region"]
)  # accuracy, question accuracy, certainty, ...
bootstrap_ci(scores["correct"].to_numpy())
kruskal(scores["p_gold"], scores["macro_region"])

Psychometrics

from nlp_components_library.analysis import psychometrics

choices, probabilities = psychometrics.response_tables(
    [response.probabilities for response in responses], segment
)
scales = psychometrics.scale_scores(choices, segment)
psychometrics.reliability(choices, segment)  # Cronbach's alpha per dimension
psychometrics.run_r_analysis("omega", choices, segment, "reports/omega.csv")  # needs R

Reproducibility defaults

• Thinking off. Templates receive enable_thinking=False unless ChatTemplate(..., thinking=True).
• Pinned date. Templates that stamp today's date get config.DEFAULT_TEMPLATE_DATE (2026-10-26); pass date=None for the real date. The dice-roll runs used datetime.date(2024, 7, 26).
• Provenance. io.write_results stores a CSV with a JSON sidecar (creation time, library version and your metadata, e.g. scorer.metadata()).

Development

uv sync --extra hf --extra api --extra plot
uv run pytest                 # tiny random models are built offline
uv run pytest -m "not slow"   # without the model tests
uv run ruff check && uv run ruff format --check && uv run mypy

The lock file resolves vllm in its own fork. Upgrading everything at once (uv lock --upgrade) can pull the hf fork down to vLLM's torch pin; upgrade packages selectively (uv lock --upgrade-package transformers) instead.

Releasing

Publishing uses PyPI trusted publishing from .github/workflows/publish.yml.

1. Once: on PyPI (and TestPyPI) add a trusted publisher for this repository with workflow publish.yml and environment pypi (testpypi), and create both environments in the GitHub repository settings.

2. Optional dry run: start the Publish workflow manually; it runs CI, builds and uploads to TestPyPI.

3. Release:

   uv version --bump minor            # or patch / major
   git commit -am "Release $(uv version --short)"
   git tag "v$(uv version --short)"
   git push origin main --tags
   

   The tag runs CI, checks that it matches the package version, builds with uv build and publishes with uv publish.