Ember-Inspect
A library for using Goodfire Ember models with Inspect AI.
Install
# Install optional dependencies for running examples
pip install ember-inspect[examples]
# Install inspect-evals separately because it's not available on PyPI
pip install "inspect-evals @ git+https://github.com/UKGovernmentBEIS/inspect_evals"
Usage
import os
import dotenv
from inspect_evals.mmlu.mmlu import mmlu_0_shot
from goodfire import Client, Variant
from ember_inspect import eval_variant
# Ensure your GoodFire API key is set
dotenv.load_dotenv(".env")
assert os.getenv("GOODFIRE_API_KEY") is not None, "GOODFIRE_API_KEY is not set"
GOODFIRE_API_KEY = os.getenv("GOODFIRE_API_KEY")
# Create Ember model variant
base_model = "meta-llama/Meta-Llama-3.1-8B-Instruct"
variant = Variant(base_model = base_model)
client = Client(api_key = GOODFIRE_API_KEY)
edits = client.features.AutoSteer(
specification = "Think like an expert in college biology",
model = variant,
)
variant.set(edits)
# Evaluate model variant with Inspect
eval_variant(
variant = variant,
tasks = [mmlu_0_shot(subjects=["college_biology"])],
limit = 5,
)
Examples
See examples/ for examples of how to use Ember-Inspect.
examples/simple: A simple demo of Ember-Inspectexamples/test_time_sae_steering/: Test-time SAE Steering
FAQ
Q: My Inspect tasks aren't completing when I use Ember models. A: This may be due to HTTP rate limits. Wait a minute and try again.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file ember_inspect-0.1.0.tar.gz.
File metadata
- Download URL: ember_inspect-0.1.0.tar.gz
- Upload date:
- Size: 4.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.0.1 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
93723a6292fdca05cfb85bb6755394553fffb8b01d26d175998763cf57d121a0
|
|
| MD5 |
1a900298235fc6f229361cb0c7d40d3b
|
|
| BLAKE2b-256 |
52d0e848a0cc3e2c1f54af30cf5d278b2a01ac29d75ac4aeabb771087c775d8a
|
Provenance
The following attestation bundles were made for ember_inspect-0.1.0.tar.gz:
Publisher:
manual_publish.yaml on dtch1997/ember-inspect
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
ember_inspect-0.1.0.tar.gz -
Subject digest:
93723a6292fdca05cfb85bb6755394553fffb8b01d26d175998763cf57d121a0 - Sigstore transparency entry: 161851923
- Sigstore integration time:
-
Permalink:
dtch1997/ember-inspect@367b8558a525c17ab5a5db62d077add3e4655d62 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/dtch1997
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
manual_publish.yaml@367b8558a525c17ab5a5db62d077add3e4655d62 -
Trigger Event:
workflow_dispatch
-
Statement type:
File details
Details for the file ember_inspect-0.1.0-py3-none-any.whl.
File metadata
- Download URL: ember_inspect-0.1.0-py3-none-any.whl
- Upload date:
- Size: 6.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.0.1 CPython/3.12.8
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7ac2572c9031ac92308d5c43a8129f028e4b48a4c693314265cf7485dcec01d9
|
|
| MD5 |
66928c6877ff7192442e0427a6606553
|
|
| BLAKE2b-256 |
5413110c8cb35ee61a16dd1110189c02973d567f29106b627afd8f5b5e8bee67
|
Provenance
The following attestation bundles were made for ember_inspect-0.1.0-py3-none-any.whl:
Publisher:
manual_publish.yaml on dtch1997/ember-inspect
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
ember_inspect-0.1.0-py3-none-any.whl -
Subject digest:
7ac2572c9031ac92308d5c43a8129f028e4b48a4c693314265cf7485dcec01d9 - Sigstore transparency entry: 161851924
- Sigstore integration time:
-
Permalink:
dtch1997/ember-inspect@367b8558a525c17ab5a5db62d077add3e4655d62 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/dtch1997
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
manual_publish.yaml@367b8558a525c17ab5a5db62d077add3e4655d62 -
Trigger Event:
workflow_dispatch
-
Statement type:
