Scorecard AI Python Library
Installation
Add this dependency to your project's build file:
pip install scorecard-ai
# or
poetry add scorecard-ai
Usage
We also export an HTTP client so that you can hit our APIs directly.
import scorecard
from scorecard.client import Scorecard
client = Scorecard(
api_key="YOUR_API_KEY" # Defaults to SCORECARD_API_KEY
)
testset = client.testset.create(
testset_id=1234,
user_query="Your prompt...",
model_params: {
"param": "value"
}
)
print(testset)
Async Client
Use our async cien to make non-blocking requests to the API.
import scorecard
import asyncio
from scorecard.client import AsyncScorecard
client = AsyncScorecard(
api_key="YOUR_API_KEY" # Defaults to SCORECARD_API_KEY
)
async def main() -> None:
testset = await client.testset.create(
testset_id=1234,
user_query="Your prompt...",
model_params: {
"param": "value"
}
)
print(testset)
asyncio.run(main())
Running Tests
The run_tests method will load your test cases,
invoke your models with saved prompts, and record success and failure. This is
available on both the sync and async client.
from scorecard.client import Scorecard
client = Scorecard(
api_key="YOUR_API_KEY"
)
client.run_tests(
# Your Testset ID
input_testset_id=123,
# Your Scoring Config ID
scoring_config_id=456,
# The model invocation that you would like to test
model_invocation=lambda prompt: call_model(prompt),
)
Timeouts
By default, the client is configured to have a timeout of 60 seconds. You can customize this value at client instantiation.
from scorecard.client import Scorecard
scorecard_client = Scorecard(
api_key="YOUR_API_KEY",
timeout=15,
)
Handling Exceptions
All exceptions thrown by the SDK will sublcass scorecard.ApiError.
from scorecard.core import ApiError
from scorecard import UnprocessableEntityError
try:
client.testset.get("testset-id")
except UnprocessableEntityError as e:
# handle unprocessable entity error
except APIError as e:
# handle any api related error
Error codes are as followed:
| Status Code | Error Type |
|---|---|
| 422 | UnprocessableEntityError |
Beta status
This SDK is in beta, and there may be breaking changes between versions without a major version update. Therefore, we recommend pinning the package version to a specific version in your pyproject.toml file. This way, you can install the same version each time without breaking changes unless you are intentionally looking for the latest version.
Contributing
While we value open-source contributions to this SDK, this library is generated programmatically. Additions made directly to this library would have to be moved over to our generation code, otherwise they would be overwritten upon the next generated release. Feel free to open a PR as a proof of concept, but know that we will not be able to merge it as-is. We suggest opening an issue first to discuss with us!
On the other hand, contributions to the README are always very welcome!
Metadata
Release files for scorecard-ai 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scorecard_ai-0.1.5.tar.gz | 14.1 kB | Without provenance | Details for scorecard_ai-0.1.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| scorecard_ai-0.1.5-py3-none-any.whl | Python 3 | none | any | Details for scorecard_ai-0.1.5-py3-none-any.whl |
Total release size: 44.4 kB
Release files / scorecard_ai-0.1.5.tar.gz
| Download URL | scorecard_ai-0.1.5.tar.gz |
|---|---|
| Size | 14.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
76beadbc02430db79cc486b3765ad7ce6b150bf8096b40735758fa00d403107a
|
|
BLAKE2b-256 checksum How to use checksums |
376bb55c79516dd11a8462e8a863f38ac8f439c1a09a7b1290322e9ebb995baf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.7.17 Linux/5.15.0-1053-azure
|
Release files / scorecard_ai-0.1.5-py3-none-any.whl
| Download URL | scorecard_ai-0.1.5-py3-none-any.whl |
|---|---|
| Size | 30.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e0526678e48f9f0fb224b8071d9b65c2d47bb66e1debebca48373b9e981c8c88
|
|
BLAKE2b-256 checksum How to use checksums |
feeddeb0ff27bb2d3b3f4154a53c86d038e0364206b302002e775463064c12ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.7.17 Linux/5.15.0-1053-azure
|
Security for scorecard-ai 0.1.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.