experimently (Python)

0.x beta. The API may change between 0.x minor versions; pin the exact version you test against.

Synchronous, dependency-free Python client for the Experimently public API. Experiment assignment and feature-flag evaluation are decided by the server by default (sticky per user + experiment); the SDK caches the answers per user + key. Server-side code can opt in to local evaluation (evaluation="local", beta): flags are answered in-process from the server's ruleset, with the server's answers, and anything that cannot be answered exactly goes to the server. See docs/sdk/local-evaluation.md.

Requires Python 3.9+. HTTP goes through the standard library (urllib), so there is nothing else to install.

Install

Not yet published. The experimently package is not on PyPI yet, so the first line below fails today. Install from a clone of this repository with the second line, or without a clone with pip install "experimently @ git+https://github.com/getexperimently/experimently.git#subdirectory=sdk/python".

pip install experimently
pip install -e sdk/python

• pip install experimently: once published
• pip install -e sdk/python: from this repository

Quick start

from experimentation import ExperimentationClient

client = ExperimentationClient(
    api_url="http://localhost:8000",       # backend origin; the SDK appends /api/v1/...
    api_key=os.environ["EXPERIMENTLY_API_KEY"],
)

# Experiments — sticky assignment made by the server
variant = client.get_variant("checkout_flow", user_id="user-123", user_attributes={"plan": "pro"})
if variant == "treatment":
    ...

assignment = client.get_assignment("checkout_flow", "user-123")     # raises ExperimentationError on failure
assignment.variant_name, assignment.is_control, assignment.configuration

# Feature flags — user_attributes (optional) is sent as ?context=<url-encoded JSON> for targeting rules
if client.is_feature_enabled("new_search", "user-123", user_attributes={"plan": "pro"}):
    ...
flag = client.get_feature_flag("new_search", "user-123")            # FlagEvaluation(key, enabled, config, reason)
flags = client.get_all_flags("user-123", {"plan": "pro"})            # {"new_search": True, ...}

# Tracking (never raises)
client.track("user-123", "purchase", event_value=49.99, experiment_key="checkout_flow")
client.track("user-123", "page_view", properties={"page": "/"})     # no key: fanned out, see below

Constructor: ExperimentationClient(api_url, api_key, timeout_seconds=5.0, cache_ttl_seconds=300, default_variant="control", transport=None). Instances are thread-safe.

API

ExperimentationError carries .status (HTTP status, None for network errors/timeouts) and .body (raw response text). Failures are never cached, so the next call retries. A 429 is retried once after Retry-After seconds (capped at 5 s).

user_attributes is sent as context on assignment and on flag evaluation (&context=<url-encoded JSON>, omitted when empty), so flag targeting rules evaluate against it (country also matches user.country; nested dicts flatten to dotted keys such as app.version). FlagEvaluation.reason (targeting_rule / rollout / inactive / error, or None) says why the server decided. Caches are keyed by user + key only — call clear_cache() after changing a user's attributes.

Enrolment: assigned and reason

An Assignment also carries assigned and reason. assigned is False means the server did not enrol the user (reason is "holdout", "mutual_exclusion" or "targeting") and returned the control variant so you render the default experience; no exposure was recorded. When you export exposures to a warehouse, exclude assignments with assigned is False: the user was not enrolled, and reason says why. A server that predates the fields sends neither, so assigned is None (never False). Decide deliberately how to treat those rows rather than dropping them with an assigned is True filter.

Tracking fan-out rule

• with experiment_key and/or feature_flag_key → one POST /api/v1/tracking/track;
• without a key → one POST /api/v1/tracking/batch (chunks of 100) containing one entry per experiment the user has been assigned to in this client (experiment_key) plus one per flag evaluated for the user (feature_flag_key), taken from the cache (successful, unexpired only);
• nothing cached for the user → nothing is sent and track returns False.

event_type defaults to event_name; properties is sent as metadata; timestamp may be a datetime (naive values are treated as UTC) or an ISO-8601 string.

Backend endpoints used

Every request carries X-API-Key: <key>, Content-Type: application/json and Accept: application/json.

Contract smoke

EXPERIMENTLY_API_KEY=<key> python sdk/python/examples/contract_smoke.py

It prints:

{"sdk":"python","assign":{"variant_name":"treatment","is_control":false,"sticky":true},"flag":{"enabled":true},"track":{"ok":true},"fanout":{"ok":true}}

Env: EXPERIMENTLY_API_URL (default http://localhost:8000), EXPERIMENTLY_API_KEY (required), CONTRACT_EXPERIMENT_KEY (default sdk_contract_ab), CONTRACT_FLAG_KEY (default sdk_contract_flag), CONTRACT_USER_ID (default random smoke-<uuid>).

Verified against a live backend: yes (2026-09-11) — sdk_contract_ab / sdk_contract_flag seeded with backend/scripts/seed_sdk_contract.py, run via python tests/sdk-contract/live/run_live_contract.py --sdk python --strict.

Testing your own code

experimentation.testing.FakeTransport records requests and serves canned responses:

from experimentation import ExperimentationClient
from experimentation.testing import FakeTransport

transport = FakeTransport().route("GET", "/api/v1/feature-flags/evaluate/new_search",
                                  json={"key": "new_search", "enabled": True, "config": None})
client = ExperimentationClient("http://test", "key", transport=transport)
assert client.is_feature_enabled("new_search", "user-1")
assert transport.last.query == {"user_id": "user-1"}

Development

source venv/bin/activate
python -m pytest sdk/python/tests -q -o addopts="" -p no:cacheprovider

• python -m pytest sdk/python/tests -q -o addopts="" -p no:cacheprovider: 182 tests, HTTP is faked; runs on Python 3.9+