Skip to main content

chatsee-connector-testing

Shared offline test kit for ChatSee connectors.

A connector hooks one framework and drives chatsee.ChatseeTracker. This package lets you test all of that with no network, no API key, no framework account and no dev tenant — so capture bugs are found in seconds, not in a live 2×2 run.

pip install -e python/connectors/_shared

The idea

Don't mock ChatSee. Mock the framework.

Your connector's test supplies a small stub of the framework that replays a canned turn, instruments it with a FakeTracker, and then asserts what reached the tracker.

from chatsee_connector_testing import FakeTracker, assert_contract

tracker = FakeTracker()
chatsee = instrument_google_adk(agent_id="a", tenant_id="t", _tracker=tracker)
run_stub_agent(chatsee.plugin)

assert_contract(tracker, expected_turns=1,
                expected_tools=["get_order_status"],
                framework="google-adk")

Every connector accepts a private _tracker= keyword so tests can inject the fake. Customers never see it — they pass the normal identity arguments and the connector builds a real tracker itself.

What assert_contract checks

Rule The bug it catches
One start_turn/end_turn per turn, correctly nested Pydantic AI's run_sync delegating to run; Vercel emitting 10 for 4; CrewAI capturing only the first kickoff
bot_message non-empty OpenAI Agents streaming — final text in no span at all
Each tool once, with args and a result or an error Tool captured but its outcome lost
log_model_call once per model call Connectors that sum tokens themselves and destroy the per-call grain
Absent is never 0 or "" An unmeasured turn looking like a free one
Required metadata present, tokens/model/tools not in metadata Tokens buried in metadata are invisible to the analytics screens

Behavioural checks

These take callables rather than a tracker, because they're about how the connector behaves rather than what it recorded.

assert_fails_open(run_one_turn)                  # ChatSee must never break the agent
assert_instrument_is_idempotent(run_turns)       # instrumenting twice is a no-op
assert_concurrent_runs_isolated(run_one_turn)    # state keyed per run, not per instance

assert_concurrent_runs_isolated is the one most likely to fail first: Google ADK, LangChain and Strands all run concurrent invocations against a single instrumented object, and the ADK POC plugin holds one shared accumulator.

Running its own tests

The suite has a self-test: a correct toy connector that must satisfy every rule, and a deliberately broken one per rule that must trip it. An assertion nobody has seen fail is not evidence of anything.

pip install -e "python/connectors/_shared[dev]"
pytest python/connectors/_shared/tests -q

Metadata

Release files for chatsee-connector-testing 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chatsee-connector-testing 0.1.0
File Size Uploaded Provenance Actions
chatsee_connector_testing-0.1.0.tar.gz 14.3 kB Without provenance Details for chatsee_connector_testing-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for chatsee-connector-testing 0.1.0
File Interpreter ABI Platform Actions
chatsee_connector_testing-0.1.0-py3-none-any.whl Python 3 none any Details for chatsee_connector_testing-0.1.0-py3-none-any.whl

Total release size: 26.1 kB

Release files / chatsee_connector_testing-0.1.0.tar.gz

Download URL chatsee_connector_testing-0.1.0.tar.gz
Size 14.3 kB
Tags Source
SHA-256 checksum
How to use checksums
5a2bd1c34d7ad2de2ad73945bc0230b68fc8c0fe31ecc32fe4aadaf92142728c
BLAKE2b-256 checksum
How to use checksums
32c6e543ddca0eb29b883e856728ace0bdf27d8052ad626b114ece36589c40d3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.11

Release files / chatsee_connector_testing-0.1.0-py3-none-any.whl

Download URL chatsee_connector_testing-0.1.0-py3-none-any.whl
Size 11.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ec9a40e06d78cfdaa226f39e7efb1208a51c03fde2e85fc76188625b83846bc2
BLAKE2b-256 checksum
How to use checksums
9d893dc4456ab0ee5f0f0fac400162b0e742873c5129c24291e216e30f1fde12
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.11

Security for chatsee-connector-testing 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page