fakellm-cli
Inspect and manage fakellm-assert frozen judgment snapshots from the terminal. Part of the fakellm family.
fakellm-assert freezes a judge's verdict about a fuzzy assertion (satisfies("apologizes for the delay")) to .fakellm/judgments/judgments.json and replays it forever. Those frozen verdicts are artifacts you review in a git diff — and once you have more than a handful, you want to look at them, sanity-check them, and clean them up without hand-editing JSON. That's what this CLI is for.
pip install fakellm-cli
Requires Python 3.9+. fakellm-assert itself is not a hard dependency — the CLI reuses its types when they're importable but can also inspect a checked-in .fakellm/ store on a machine that only has the snapshots. Install them together with pip install fakellm-cli[assert] if you want both.
Commands
fakellm-cli list # every frozen verdict, with pass/fail counts
fakellm-cli list --verdict fail # just the failures
fakellm-cli show "apologizes" # one verdict in full: reasoning + response excerpt
fakellm-cli show aaaa1111 # (by fingerprint prefix or criterion substring)
fakellm-cli verify # integrity check: schema, verdict values, key match
fakellm-cli prune --verdict fail # preview removing all failing verdicts (dry run)
fakellm-cli prune --verdict fail --yes # actually remove them
fakellm-cli diff main/ feature/ # what changed between two snapshot dirs
fakellm-cli init # scaffold .fakellm/ and a conftest.py judge stub
Every read command takes --store PATH (pointing at either the judgments dir or the judgments.json file; default .fakellm/judgments) and --json for machine-readable output. Commands return a non-zero exit code on the condition you'd want to gate CI on: verify fails on integrity problems, diff fails when a verdict flipped pass↔fail.
What it does not do: re-judge
There is deliberately no fakellm-cli rejudge. Two reasons, both structural:
- The store doesn't keep enough to re-judge. A frozen record holds only a 280-character excerpt of the response, not the full text. Re-judging needs the exact response to recompute the fingerprint — and that lives in your test, not in the snapshot.
- Re-judging is a live model call that belongs in a reviewed run.
fakellm-assert's whole point is that verdicts are produced exactly once, in an explicitpytest --fakellm-update, where a human reads the diff. A CLI that judged live would route around the one safety property the library exists to provide.
So the division of labor is: pytest --fakellm-update produces verdicts; fakellm-cli manages them. When verify or diff tells you a verdict is stale or wrong, the fix is to prune it here and re-judge in pytest there.
diff matches on criterion, not fingerprint
A fingerprint includes the response text, so the "same" assertion against a regenerated response has a different fingerprint by design. diff therefore pairs verdicts across two stores by (criterion, judge_model) so it can actually catch a pass→fail flip, rather than reporting every drifted response as an unrelated add+remove.
Typical workflow
fakellm-cli init # once, to scaffold
# ... write satisfies() assertions, then:
pytest --fakellm-update # freeze verdicts (review the diff!)
fakellm-cli list # eyeball what got frozen
fakellm-cli verify # gate in CI alongside pytest
# later, when you intentionally change a prompt:
fakellm-cli prune --criterion "old wording" --yes
pytest --fakellm-update # re-freeze
License
MIT
Metadata
Release files for fakellm-cli 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| fakellm_cli-0.1.0.tar.gz | 13.5 kB | Without provenance | Details for fakellm_cli-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| fakellm_cli-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for fakellm_cli-0.1.0-py3-none-any.whl |
Total release size: 26.3 kB
Release files / fakellm_cli-0.1.0.tar.gz
| Download URL | fakellm_cli-0.1.0.tar.gz |
|---|---|
| Size | 13.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a0d65ac6ba91b4d86d4a6fe0876a3aa083c6920c9ff48bed71095729532a9e0e
|
|
BLAKE2b-256 checksum How to use checksums |
d1409a0b2466c7bc493a6a8a8df9c1df4d63b0eca68a069f82c79a7dc31a35c5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Release files / fakellm_cli-0.1.0-py3-none-any.whl
| Download URL | fakellm_cli-0.1.0-py3-none-any.whl |
|---|---|
| Size | 12.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9e27d1f1402aec6561cf904a788fa02b1c56c59e96957e815515b7fedce42dfc
|
|
BLAKE2b-256 checksum How to use checksums |
b8a5fd8fefba2f97e6fbe75c6cc24717a11387f0a2b8f3c801549a2efe628f02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.2
|
Security for fakellm-cli 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.