evalshift-sdk

In-process capture SDK for EvalShift.

Install it inside your agent process to record what the agent does — model calls, tool calls, retrievals — and write CLI-valid traces to .evalshift/captures/. The evalshift CLI reads those captures from disk; the SDK and CLI never call each other.

  • Distribution: evalshift-sdk · import name: evalshift
  • Runtime deps: none (stdlib-only)
  • Python: >= 3.10
  • Capture is off by default — set EVALSHIFT_CAPTURE=1 to record.
  • License: MIT

Install

pip install evalshift-sdk
# or
uv add evalshift-sdk

Optional LangChain integration (EvalShiftCallbackHandler):

pip install "evalshift-sdk[langchain]"   # adds langchain-core>=0.2

The adapter module is import-guarded, so the SDK stays dependency-free at runtime unless you opt in.

Co-install note: the SDK (import name evalshift) and the EvalShift CLI share the same top-level import name. Keep them in separate virtual environments.

Usage

from evalshift import capture

@capture.agent(suite="support_agent")   # no-op unless EVALSHIFT_CAPTURE=1
def handle_ticket(query): ...

Full guide: DOCS.md · dense LLM reference: https://www.evalshift.dev/sdk-llms-full.txt · locked design decisions: docs/DECISIONS.md

Keeping captures/ bounded

Capture writes one JSON file per sampled invocation, so .evalshift/captures/ is kept bounded by default (no configuration needed): identical-input re-runs are de-duplicated, and each suite directory is capped at the 200 newest captures (oldest evicted). Tune it with env vars — no code change required (precedence: an explicit configure(...) call > env var > built-in default):

Env var Default Meaning
EVALSHIFT_MAX_CAPTURES 200 Max captures kept per suite dir. 0 / none / unlimited = uncapped.
EVALSHIFT_DEDUP on Collapse identical-input captures (per-process). off to disable.
EVALSHIFT_CAPTURE_TTL off Evict captures older than N seconds.
EVALSHIFT_SAMPLE_RATE off Capture only this fraction of runs, e.g. 0.25.
EVALSHIFT_DIR .evalshift Capture root directory.

A malformed value falls back to the default (capture never crashes). To restore fully unbounded capture: EVALSHIFT_MAX_CAPTURES=0 EVALSHIFT_DEDUP=off. The same knobs are available in code via configure(max_captures=..., dedup=..., capture_ttl=..., sample_rate=...).

Project links

Data verified by PyPI on
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for babaliauskas from gravatar.com babaliauskas

Credits

Author: Lukas Babaliauskas

GitHub Statistics

Data verified by PyPI on 
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

langchain

Tags

agents capture evalshift llm observability tracing

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

evalshift_sdk-0.2.0.tar.gz (206.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

evalshift_sdk-0.2.0-py3-none-any.whl (47.4 kB view details)

Uploaded Python 3

File details

Details for the file evalshift_sdk-0.2.0.tar.gz.

File metadata

  • Download URL: evalshift_sdk-0.2.0.tar.gz
  • Upload date:
  • Size: 206.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for evalshift_sdk-0.2.0.tar.gz
Algorithm Hash digest
SHA256 48ec40cf7aebae707c3a18e8f3287de084747f83bf0919839a6adf0fde3b887e
MD5 03a745f85789e9741855f567986cf479
BLAKE2b-256 1ab72d378527146cb6837237db6f128c1759301834a7727bcb8f71f0b7b78fcf

See more details on using hashes here.

Provenance

The following attestation bundles were made for evalshift_sdk-0.2.0.tar.gz:

Publisher: release.yml on babaliauskas/evalshift-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file evalshift_sdk-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: evalshift_sdk-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 47.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for evalshift_sdk-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 904b576848bb7333c36fdc7e7517d5b19d6b6c5fd1ac9597f45a95f8ebcabeb5
MD5 aeba1714eae72e4a1ad0d32f192ba277
BLAKE2b-256 bfe3057cbdd51984ad6792723c7ee148d8977d0e1fdd3cadcb823c4ec44b990c

See more details on using hashes here.

Provenance

The following attestation bundles were made for evalshift_sdk-0.2.0-py3-none-any.whl:

Publisher: release.yml on babaliauskas/evalshift-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page