Skip to main content

Scorecard AI Python Library

pypi fern shield

Installation

Add this dependency to your project's build file:

pip install scorecard-ai
# or
poetry add scorecard-ai

Usage

We also export an HTTP client so that you can hit our APIs directly.

import scorecard

from scorecard.client import Scorecard

client = Scorecard(
  api_key="YOUR_API_KEY" # Defaults to SCORECARD_API_KEY
)

testset = client.testset.create(
  testset_id=1234, 
  user_query="Your prompt...", 
  model_params: {
    "param": "value"
  }
)

print(testset)

Async Client

Use our async cien to make non-blocking requests to the API.

import scorecard
import asyncio

from scorecard.client import AsyncScorecard

client = AsyncScorecard(
  api_key="YOUR_API_KEY" # Defaults to SCORECARD_API_KEY
)

async def main() -> None:
  testset = await client.testset.create(
    testset_id=1234, 
    user_query="Your prompt...", 
    model_params: {
      "param": "value"
    }
  )

  print(testset)

asyncio.run(main())

Running Tests

The run_tests method will load your test cases, invoke your models with saved prompts, and record success and failure. This is available on both the sync and async client.

from scorecard.client import Scorecard

client = Scorecard(
  api_key="YOUR_API_KEY"
)

client.run_tests(
  # Your Testset ID 
  input_testset_id=123,
  # Your Scoring Config ID
  scoring_config_id=456,
  # The model invocation that you would like to test
  model_invocation=lambda prompt: call_model(prompt),
)

Timeouts

By default, the client is configured to have a timeout of 60 seconds. You can customize this value at client instantiation.

from scorecard.client import Scorecard

scorecard_client = Scorecard(
  api_key="YOUR_API_KEY",
  timeout=15,
)

Handling Exceptions

All exceptions thrown by the SDK will sublcass scorecard.ApiError.

from scorecard.core import ApiError
from scorecard import UnprocessableEntityError

try:
  client.testset.get("testset-id")
except UnprocessableEntityError as e: 
  # handle unprocessable entity error
except APIError as e:  
  # handle any api related error

Error codes are as followed:

Status Code Error Type
422 UnprocessableEntityError

Beta status

This SDK is in beta, and there may be breaking changes between versions without a major version update. Therefore, we recommend pinning the package version to a specific version in your pyproject.toml file. This way, you can install the same version each time without breaking changes unless you are intentionally looking for the latest version.

Contributing

While we value open-source contributions to this SDK, this library is generated programmatically. Additions made directly to this library would have to be moved over to our generation code, otherwise they would be overwritten upon the next generated release. Feel free to open a PR as a proof of concept, but know that we will not be able to merge it as-is. We suggest opening an issue first to discuss with us!

On the other hand, contributions to the README are always very welcome!

Metadata

Release files for scorecard-ai 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scorecard-ai 0.1.5
File Size Uploaded Provenance Actions
scorecard_ai-0.1.5.tar.gz 14.1 kB Without provenance Details for scorecard_ai-0.1.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scorecard-ai 0.1.5
File Interpreter ABI Platform Actions
scorecard_ai-0.1.5-py3-none-any.whl Python 3 none any Details for scorecard_ai-0.1.5-py3-none-any.whl

Total release size: 44.4 kB

Release files / scorecard_ai-0.1.5.tar.gz

Download URL scorecard_ai-0.1.5.tar.gz
Size 14.1 kB
Tags Source
SHA-256 checksum
How to use checksums
76beadbc02430db79cc486b3765ad7ce6b150bf8096b40735758fa00d403107a
BLAKE2b-256 checksum
How to use checksums
376bb55c79516dd11a8462e8a863f38ac8f439c1a09a7b1290322e9ebb995baf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.5.1 CPython/3.7.17 Linux/5.15.0-1053-azure

Release files / scorecard_ai-0.1.5-py3-none-any.whl

Download URL scorecard_ai-0.1.5-py3-none-any.whl
Size 30.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e0526678e48f9f0fb224b8071d9b65c2d47bb66e1debebca48373b9e981c8c88
BLAKE2b-256 checksum
How to use checksums
feeddeb0ff27bb2d3b3f4154a53c86d038e0364206b302002e775463064c12ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.5.1 CPython/3.7.17 Linux/5.15.0-1053-azure

Security for scorecard-ai 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.5 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page