Mellea × Jev

A small Python adapter that brings TypeSafe Jev's semantic checks into Mellea. Use Jev to verify generated text, classify it into your labels, or rate it on a scale. Mellea continues to manage generation and repair.

• Verify requirements with Jev Noul and configurable accept/reject thresholds.
• Classify text with TypeSafe Choice and caller-defined categories.
• Score text on an ordered scale with TypeSafe Score.
• Ask several questions in one request and inspect returned token usage.
• Connect checks to Mellea Requirements so they can participate in sampling.

This is an unofficial, synchronous adapter. Jev evaluates text; it does not generate or repair it.

Quick start

Install from a checkout and set an API key from the TypeSafe console. Live requests may incur charges.

git clone https://github.com/SoundBlaster/Jev4Mellea.git
cd Jev4Mellea
make install
export TYPESAFE_API_KEY='your-key'

The Makefile defaults to Python 3.13. To use another supported interpreter, run make install PYTHON=python3.11 (or set PYTHON to your installed version).

Ask whether a candidate meets a positive requirement:

from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The answer gives the museum's opening time from the reference.",
        reference="The museum opens at 10:00 and closes at 18:00.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")
    print(verdict.outcome, verdict.p_yes)

outcome is pass, fail, or uncertain. Use verifier.as_requirement() to attach the same check to a Mellea generation flow. Once the project dependencies are installed, make demo runs without a key or network access using a mocked Jev response.

Examples

Verify a requirement with Noul

Noul returns p_yes, the probability that a positively phrased requirement is satisfied. The adapter uses two configurable thresholds; the space between them is uncertain and is never silently accepted.

from mellea_jev import JevClient, JevVerifier

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the reference.",
        reference="The museum opens at 10:00.",
        criteria={
            "true": "The candidate gives 10:00 as the opening time.",
            "false": "The candidate omits or contradicts the opening time.",
        },
        accept_at=0.90,
        reject_at=0.10,
        repair_hint="Use the opening time stated in the reference.",
    )
    verdict = verifier.evaluate("The museum opens at 10:00.")

    if verdict.outcome == "uncertain":
        print("Route for another check or human review")

The threshold values are application policy, not accuracy guarantees. Noul does not return a separate confidence field or a textual explanation.

To use the verifier as a Mellea requirement:

requirement = verifier.as_requirement()

For generation and repair with a real Mellea session, see examples/mellea_ollama.py. That example uses Ollama for generation and Jev for verification.

Classify into configured categories with Choice

Choice selects one label from the supplied criteria and returns its confidence and the probability of every label. Descriptions may be strings, JSON objects, arrays, or None.

from mellea_jev import JevClient, JevClassifier

criteria = {
    "billing": {"what": "Payments, invoices, or refunds", "examples": ["duplicate charge"]},
    "technical": "Product errors or problems using the service",
    "other": None,
}

with JevClient() as jev:
    classifier = JevClassifier(
        jev,
        "Choose the best category for this support message.",
        criteria=criteria,
    )
    result = classifier.classify("I was charged twice for my subscription.")
    print(result.choice, result.confidence, result.probabilities)

To require a Mellea candidate to be assigned to a particular category:

requirement = classifier.as_requirement(
    "billing",
    minimum_confidence=0.75,
)

Confidence thresholds are caller policy. TypeSafe supports up to 255 Choice labels.

Rate text on an ordered scale with Score

Score returns a probability-weighted position on an ordered scale. The result can fall between two levels.

from mellea_jev import JevClient, JevScorer

levels = ["Cosmetic", "Workaround exists", "Blocking"]

with JevClient() as jev:
    scorer = JevScorer(
        jev,
        "How severe is the reported issue?",
        criteria=levels,
    )
    result = scorer.evaluate("The export button crashes and there is no workaround.")
    print(result.score, result.confidence, result.probabilities)

The result can also become a Mellea requirement with inclusive score bounds:

requirement = scorer.as_requirement(
    minimum_score=1.0,
    maximum_score=2.0,
    minimum_confidence=0.6,
)

Score supports 2–10 ordered string descriptions.

Batch questions and read usage metadata

JevClient.system_one() sends named Noul, Choice, and Score questions in one request. Each answer remains a typed result. The returned usage counts are informational and do not affect validation.

from mellea_jev import ChoiceQuestion, JevClient, NoulQuestion

with JevClient() as jev:
    result = jev.system_one(
        state={"candidate": "I was charged twice and cannot log in."},
        questions={
            "urgent": NoulQuestion("Does the message convey urgency?"),
            "team": ChoiceQuestion(
                "Which team should handle this?",
                {"billing": "Payments and invoices", "technical": "Product errors"},
            ),
        },
    )

    print(result.answers["urgent"].p_yes)
    print(result.answers["team"].choice)
    if result.usage is not None:
        print(result.usage.input_tokens, result.usage.output_tokens)

For one-off checks without the Mellea helper classes, call the client methods directly. Each method sends its own request:

with JevClient() as jev:
    yes_no = jev.noul(
        state={"candidate": "The museum opens at 10:00."},
        question="Does the candidate state the opening time?",
    )
    category = jev.choice(
        state={"candidate": "I was charged twice."},
        question="Choose a category.",
        criteria={"billing": "Payments", "technical": "Product errors"},
    )
    rating = jev.score(
        state={"candidate": "The export is broken."},
        question="Rate the impact.",
        criteria=["minor", "major"],
    )

The single-question result objects also expose optional usage metadata.

Use a check during Mellea generation

For an existing Mellea session m, pass the adapter's requirement to instruct(). Keep the Jev client open until sampling finishes because the requirement calls it during validation:

from mellea.stdlib.sampling import RepairTemplateStrategy
from mellea_jev import JevClient, JevVerifier, accepted_text

with JevClient() as jev:
    verifier = JevVerifier(
        jev,
        "The candidate states the opening time supported by the source.",
        reference="The museum opens at 10:00.",
    )
    sampled = m.instruct(
        "State the museum's opening time using this source: {{source}}",
        user_variables={"source": "The museum opens at 10:00."},
        requirements=[verifier.as_requirement()],
        strategy=RepairTemplateStrategy(loop_budget=3, concurrency_budget=1),
        return_sampling_results=True,
    )
    answer = accepted_text(sampled)

accepted_text() checks the final validation state before returning text. Do not return sampled.result directly after failed or incomplete sampling.

Run checks with local Laya-MLX

On Apple Silicon macOS, install the optional backend and load a Laya checkpoint. The first load may download the model weights; inference then runs locally.

pip install -e '.[laya]'

import laya_mlx
from mellea_jev import JevClassifier
from mellea_jev.providers import LayaProvider

agent = laya_mlx.load("aac6fef/laya-mlx")
classifier = JevClassifier(
    LayaProvider(agent),
    "Which team should handle this request?",
    criteria={"billing": "Payments and refunds", "technical": "Bugs and outages"},
)
print(classifier.classify("I was charged twice.").choice)

LayaProvider accepts an already loaded agent and does not import Laya or MLX when the base package is imported. It maps Laya's Noul, Choice, and Score answers into the same response contracts used by JevClient. Review the selected model checkpoint's license separately; the Laya-MLX runtime is Apache-2.0 licensed. Laya computes Choice confidence from normalized entropy, so calibrate minimum_confidence for the selected backend rather than copying a threshold from another provider. See the Laya-MLX API and platform notes and confidence implementation.

Evaluate a provider on labeled examples

examples/evaluate.py reports false-acceptance, false-rejection, and uncertain rates for every provider, returned model, and threshold pair. The checked-in museum example is a small format demonstration, not a quality benchmark. Add representative, non-sensitive examples for your own task before drawing quality conclusions. See the initial live Jev and Laya run for a four-example smoke benchmark and its limitations.

The dataset is versioned JSONL: the first line describes the positive Noul requirement; each following line labels one candidate as accept or reject. An optional reference is sent with that candidate. For example:

{"type":"dataset","format_version":1,"name":"support-policy","version":"1.0.0","requirement":"The answer follows the refund policy."}
{"type":"example","id":"in-policy","candidate":"...","reference":"...","expected":"accept"}
{"type":"example","id":"out-of-policy","candidate":"...","reference":"...","expected":"reject"}

Inference is opt-in. This command sends each example to Jev and saves the raw predictions, so it may incur charges and transmits dataset text to TypeSafe:

python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --live --provider typesafe --model jev-latest \
  --threshold 0.10,0.90 --save-predictions /tmp/museum-predictions.jsonl

To compare multiple threshold pairs or reproduce a report, load the saved predictions offline. No provider is constructed and no request is sent:

python examples/evaluate.py examples/evaluation/museum_opening.jsonl \
  --predictions /tmp/museum-predictions.jsonl \
  --threshold 0.10,0.90 --threshold 0.20,0.80

Use --live --provider laya --model aac6fef/laya-mlx to run the same labeled examples through Laya-MLX on a supported Apple Silicon setup. Loading that checkpoint may download model weights. Prediction snapshots contain p_yes, provider, and returned model; keep them with the dataset version. The report also includes the SHA-256 of the exact dataset file, so a changed file cannot be silently paired with old predictions. A snapshot records the provider, returned model, and raw probability for each example:

{"type":"prediction_set","format_version":1,"dataset":"support-policy","dataset_version":"1.0.0","dataset_sha256":"..."}
{"type":"prediction","id":"in-policy","provider":"typesafe","model":"jev-1.13.0","p_yes":0.98}

The report defines false-acceptance rate as false accepts divided by expected rejects, false-rejection rate as false rejects divided by expected accepts, and uncertain rate as uncertain predictions divided by all examples. It also reports counts and denominators; missing classes are rejected during dataset loading. Always publish the dataset version, sample count, provider/model, thresholds, and limitations alongside any observed quality rates. Do not commit private or sensitive examples or prediction snapshots.

Requirements and compatibility

• Python 3.11 or newer.
• Mellea 0.7.0 for the Requirement integration.
• TypeSafe API access and TYPESAFE_API_KEY for live Jev requests. Mocked tests and make demo need no key.
• The TypeSafe provider uses the official TypeSafe Python SDK and its synchronous HTTPX2 transport.
• laya-mlx is optional and supported by its upstream project on Apple Silicon macOS.

The CI compatibility matrix runs the package checks with these combinations:

Python	Mellea integration
3.11	0.7.0
3.12	0.7.0
3.13	0.7.0
3.14	0.7.0

The Mellea extra is pinned to 0.7.0; other Mellea versions are not currently declared compatible.

The Mellea requirement callback is synchronous, so a Jev request can block the event loop. This package does not provide an async client. If sampling has already seen a failed candidate, Mellea may return a failed sampling result instead of propagating a later Jev error or uncertain verdict; inspect the final result with accepted_text(). See API notes for external contracts and test report for the evidence behind the current prototype status.

The Mellea helpers depend on small structural protocols: NoulProvider, ChoiceProvider, and ScoreProvider (or the combined PrimitiveProvider). A custom backend can implement only the primitive it needs; it does not need to inherit from a package class. Its response must expose the fields in NoulResponse, ChoiceResponse, or ScoreResponse. The current criteria shapes follow the TypeSafe request model, and each provider may impose its own limits. Batched requests remain a TypeSafe feature. Use JevClient as the existing compatible name, or import TypeSafeProvider explicitly from mellea_jev.providers. See provider contracts.

Development and further reading

make help    # list the repository commands
make check   # run lint, formatting, type, test, coverage, and whitespace checks
make demo    # run without API keys or network access

GitHub CI runs the same make check gate: Ruff linting and formatting, strict Mypy checks, a maximum cyclomatic complexity of 16, and at least 80% branch-aware coverage. Live service requests remain opt-in and are not part of CI.

• API notes and source references
• Test report and validation limits
• Development commands
• Release process
• Roadmap
• Offline demo
• Live Jev example

Before using the adapter with private data, account for the fact that candidate text and any supplied reference are sent to TypeSafe. The adapter does not log request bodies or API keys, and it does not follow redirects or retry requests automatically. See the API notes for details.

Licensed under MIT. This project is unofficial and is not affiliated with Mellea, IBM, or TypeSafe.