catwalk-client 0.0.13

Client for Catwalk

Catwalk Client

Catwalk is case aggregator for ML solutions where model query/responses can be collected for the later evaluation.
This is client library helping to perform some common operations on the Catwalk API using python code.

Install

Run pip install catwalk-client

Sending cases

To send new open cases to the Catwalk instance you can use snippet below.

User can allow concurrent case collection while creating CatwalkClient by setting concurrent argument to True. The ThreadPoolExecutor is created with number of maximum workers passed by max_workers argument, by default it's 4 workers.

    from catwalk_client import CatwalkClient

    # catwalk_url can be passed explicitly or can be provided in CATWALK_URL environment variable
    client = CatwalkClient(submitter_name="fatman", submitter_version="1.0.0", catwalk_url="http://localhost:9100", concurrent=True, max_workers=2)

    # direct call with dict to create new case
    client.send({
        "metadata": {"someint": 20},
        "query": [
            {"name": "lokalid", "value": "7386259234132", "type": "string"},
            {"name": "test3", "value": "yup", "type": "string"},
            {"name": "test2", "value": "yup", "type": "string"},
            {"name": "test1", "value": "yup", "type": "string"}
        ],
        "context": [],
        "response": [
            {
                "name": "predictions",
                "type": {
                    "name": "prediction",
                    "thresholds": [
                        {"from": 0, "to": 0.02, "label": "NO"},
                        {"from": 0.02, "to": 0.6, "label": "PERHAPS"},
                        {"from": 0.6, "to": 1, "label": "YES"}
                    ]
                },
                "value": {
                    "477110": 0.1493704617023468,
                    "477111": 0.3493704617023468,
                    "477112": 0.6493704617023468
                },
                "evaluation": [
                    {
                        "name": "choice",
                        "question": "Which branchcode is correct?",
                        "choices": ["477110", "477111", "477112"],
                        "multi": True
                    }
                ]
            }
        ]
    })

    # fluent API to create new cases
    client.new_case().add_query(
        name="some query key", value="1345243", type="str"
    ).add_query(
        name="other query key", value="1345243", type="str"
    ).add_context(
        name="photo", value="url", type="image"
    ).add_response(
        name="is_valid", value=True, type="bool", evaluation=[
            {"question": "Choose one", "name": "choice", "choices": ["YES", "NO"]}
        ]
    ).set_metadata(
        caller="esc-1"
    ).send()

Exporting cases

Exporting case can be done programmatically, by including CatwalkClient in your code. It requires to input AUTHORIZATION TOKEN, you can find it by going to your User profile. Each environment (prod, preprod, dev, test) has different tokens.

To export cases from the Catwalk instance there is export_cases generator function available.

    # catwalk_url can be passed explicitly or can be provided in CATWALK_URL environment variable
    # auth_token can be passed explicitly or can be provided in CATWALK_AUTH_TOKEN environment variable
    client = CatwalkClient(
        catwalk_url="https://catwalk.ikp-test-c3.kubernilla.dk/api", auth_token="*TOKEN*", insecure=False
    )


    def get_cw_data(client: CatwalkClient, name, version):
        data = []

        for case in client.export_cases(
            from_datetime=datetime(2023, 2, 8),
            to_datetime=datetime(2023, 2, 9),
            submitter_name=name,  # submitter_name is an optional argument,
            submitter_version=version,  # submitter_version is an optional argument,
            max_retries=5,
        ):
            print(case.id)
            data.append(case)

        print("Number of exported cases:", len(data))

        return data


    data = get_cw_data(client, "test", "0.0.1")

Result

When a case is successfully collected client should return ID of a collected case.

In some cases host might response with an error. In this case client will inform user that it ocurred and it will display response status, error type and error message.

Exceptions

Catwalk Client might end up throwing an exception. Here are a few that user can experience:

• Connection error: when the connection between client and host couldn't be established. This might occur either when user enters a wrong host address or when the host is offline.
• ValidationError or TypeError: when user enters wrongly formatted case.
• Authorization Error (403): when user doesn't enter the authorization token (or enters one without appropriate permissions).
• Other - when any other error unhandled directly by Catwalk Client occurs it will display an exception name.