Client for Catwalk
Project description
Catwalk Client
Catwalk is case aggregator for ML solutions where model query/responses can be collected for the later evaluation.
This is client library helping to perform some common operations on the Catwalk API using python code.
Install
Run pip install catwalk-client
Sending cases
To send new open cases to the Catwalk instance you can use snippet below or run python3.10 catwalk_client/scripts/send_sample_case.py
.
from catwalk_client import CatwalkClient
# catwalk_url can be passed explicitly or can be provided in CATWALK_URL environment variable
client = CatwalkClient(submitter_name="fatman", submitter_version="1.0.0", catwalk_url="http://localhost:9100")
# direct call with dict to create new case
client.send({
"metadata": {"someint": 20},
"query": [
{"name": "lokalid", "value": "7386259234132", "type": "string"},
{"name": "test3", "value": "yup", "type": "string"},
{"name": "test2", "value": "yup", "type": "string"},
{"name": "test1", "value": "yup", "type": "string"}
],
"context": [],
"response": [
{
"name": "predictions",
"type": {
"name": "prediction",
"thresholds": [
{"from": 0, "to": 0.02, "label": "NO"},
{"from": 0.02, "to": 0.6, "label": "PERHAPS"},
{"from": 0.6, "to": 1, "label": "YES"}
]
},
"value": {
"477110": 0.1493704617023468,
"477111": 0.3493704617023468,
"477112": 0.6493704617023468
},
"evaluation": [
{
"name": "choice",
"question": "Which branchcode is correct?",
"choices": ["477110", "477111", "477112"],
"multi": True
}
]
}
]
})
# fluent API to create new cases
client.new_case().add_query(
name="some query key", value="1345243", type="str"
).add_query(
name="other query key", value="1345243", type="str"
).add_context(
name="photo", value="url", type="image"
).add_response(
name="is_valid", value=True, type="bool", evaluation=[
{"question": "Choose one", "name": "choice", "choices": ["YES", "NO"]}
]
).set_metadata(
caller="esc-1"
).send()
Exporting cases
To export cases from the Catwalk instance there is export_cases
generator function available.
# catwalk_url can be passed explicitly or can be provided in CATWALK_URL environment variable
client = CatwalkClient(catwalk_url="https://ikp-dev-c2.kubernilla.ersttest.dk/catwalk/api/")
for case in client.export_cases(
from_datetime=datetime(2022, 1, 1),
to_datetime=datetime(2022, 2, 1),
submitter_name="fatman", # submitter_name is an optional filter,
submitter_version="1.0.0", # submitter_version is an optional filter,
token="*TOKEN*", # user auth token is required, user can access his auth token from `User profile`,
max_retries=10
):
print(case)
Exporting case can be done both programmatically, by including CatwalkClient in your code, or by using export.py
script. Both approaches require to input authorization token.
Result
When a case is successfully collected client should return ID of a collected case.
In some cases host might response with an error. In this case client will inform user that it ocurred and it will display response status, error type and error message.
Exceptions
Catwalk Client might end up throwing an exception. Here are a few that user can experience:
- ConnectionError: when the connection between client and host couldn't be established. This might occur either when user enters a wrong host address or when the host is offline.
- ValidationError or TypeError: when user enters wrongly formatted case.
- Other - when any other error unhandled directly by Catwalk Client occurs it will display an exception name.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for catwalk_client-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f6c53fee893af5a52dd8b1b7a474bac025b8ef65a7cfba9709e12a42f13ab7a |
|
MD5 | bf50a2892c4c67c6595e1474fcde262a |
|
BLAKE2b-256 | efad48c4d8fdfccf00e97dc2a61648fc8f33b10e687e6aadef99b77e72390d76 |