Skip to main content

Variably Python SDK

Official Python SDK for Variably — LLM evaluation, experimentation, and prompt optimization.

Installation

pip install variably-sdk

For Docker/Kubernetes deployments, add to your requirements.txt:

variably-sdk>=2.6.1

Quick Start — Observe Mode

Add one line to your existing LLM app and get multi-dimension evaluation across 40+ metrics in 6 categories: Quality, Safety, Semantic, Grounding, Coherence, and Advanced.

No experiment setup. No prompt migration. Just log and see scores.

1. Set your environment variables

VARIABLY_API_KEY=vbl_your_key_here
VARIABLY_BASE_URL=https://api.variably.dev

2. Add one line after your LLM call

from variably import observe

# Your existing code (unchanged)
response = your_llm_call(user_query)

# Add this line:
observe(prompt=user_query, response=response)

Auto-extract tokens & model from provider response

# OpenAI
from openai import OpenAI
from variably import observe

client = OpenAI()
completion = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": user_query}],
)

observe(
    prompt=user_query,
    response=completion.choices[0].message.content,
    provider_response=completion,  # auto-extracts model, tokens
)
# Anthropic
import anthropic
from variably import observe

client = anthropic.Anthropic()
message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": user_query}],
)

observe(
    prompt=user_query,
    response=message.content[0].text,
    provider_response=message,  # auto-extracts model, tokens
)

RAG applications — grounding & hallucination scoring

observe(
    prompt=user_query,
    response=llm_answer,
    provider_response=completion,
    reference_materials=[
        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
    ],
    retrieval_query=user_query,
)

Multi-turn chat — coherence scoring

observe(
    prompt=latest_user_message,
    response=llm_answer,
    provider_response=completion,
    conversation_history=[
        {"role": "user", "content": "What is diabetes?"},
        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
        {"role": "user", "content": "What are the symptoms?"},
    ],
    session_id="conv-123",
)

RAG applications — grounding & hallucination scoring

observe(
    prompt=user_query,
    response=llm_answer,
    provider_response=completion,
    reference_materials=[
        {"id": "chunk-1", "content": "Retrieved text...", "source": "docs.pdf"},
        {"id": "chunk-2", "content": "Another chunk...", "source": "faq.md"},
    ],
    retrieval_query=user_query,
)

Multi-turn chat — coherence scoring

observe(
    prompt=latest_user_message,
    response=llm_answer,
    provider_response=completion,
    conversation_history=[
        {"role": "user", "content": "What is diabetes?"},
        {"role": "assistant", "content": "Diabetes is a chronic condition..."},
        {"role": "user", "content": "What are the symptoms?"},
    ],
    session_id="conv-123",
)
Parameter Type Required Description
prompt str Yes The user's input / question
response str Yes The LLM's generated response
provider_response object No Raw OpenAI/Anthropic/Google response — auto-extracts model, tokens
model str No Model name (auto-extracted if provider_response given)
provider str No "openai", "anthropic", etc. (auto-detected)
latency_ms int No Response generation time in milliseconds
prompt_tokens int No Input token count (auto-extracted if provider_response given)
completion_tokens int No Output token count (auto-extracted)
cost float No Cost in USD
reference_materials list[dict] No RAG chunks: [{"id", "content", "source"}] — enables grounding scoring
retrieval_query str No Query sent to retriever — enables retrieval quality scoring
conversation_history list[dict] No Prior turns: [{"role", "content"}] — enables coherence scoring
tags list[str] No Grouping labels, e.g. ["production", "rag"]
user_id str No Your user's ID
session_id str No Conversation session ID (groups multi-turn)
metadata dict No Any extra key-value data

Ending a conversation

Checks over a whole conversation (business rules with scope: session, and the other session checks) run when the conversation ends. Call end_session when it is over, for example when a call hangs up:

client.end_session("conv-123")
# {"session_id": "conv-123", "status": "ended", ...}

Calling it again is safe: it returns "status": "already_ended" and runs nothing twice. A session_id with no logged step raises NetworkError (404) with the reason. Marking the last step with is_aggregator=True keeps working.


Prompt Experimentation

Variably provides two modes for LLM prompt experimentation:

BYOR (Bring Your Own Runtime)

You call your own LLM. Variably handles variant allocation and 41-dimensional evaluation.

from variably import VariablyClient
import time

client = VariablyClient({"api_key": "your-api-key"})

user_context = {"user_id": "user-123"}
input_variables = {"query": "What are the symptoms of Type 2 diabetes?"}

# Step 1: Get the allocated variant
variant = client.get_variant("rag-prompt-experiment", user_context, input_variables)
print(f"Variant: {variant.variant_key}, Model: {variant.model}")

# Step 2: Call your LLM with the variant's prompt template
prompt = variant.prompt_template.format(**input_variables)
start = time.time()
llm_response = call_your_llm(prompt, model=variant.model)  # your LLM call
latency = int((time.time() - start) * 1000)

# Step 3: Submit the response for 41-dimensional evaluation
result = client.submit_response(
    experiment_key="rag-prompt-experiment",
    variant_key=variant.variant_key,
    executed_prompt=prompt,
    response=llm_response,
    user_context=user_context,
    input_variables=input_variables,
    provider=variant.provider,
    model=variant.model,
    latency_ms=latency,
)
print(f"Submitted: {result.status}")

Managed Execution

Variably selects the variant, calls the LLM, and evaluates — all in one call.

response = client.evaluate_prompt(
    experiment_key="rag-prompt-experiment",
    user_context={"user_id": "user-123"},
    input_variables={"query": "What are the symptoms of Type 2 diabetes?"},
    evaluation_mode="full",  # "full" | "fast"
)

print(f"Content: {response.content}")
print(f"Model: {response.model}, Latency: {response.latency_ms}ms")
print(f"Tokens: {response.token_usage}")
print(f"Quality Score: {response.quality_score}")

Managed Execution with Streaming (v2.1.0+)

Same as managed execution, but tokens stream in real-time — ideal for chatbot UIs.

from variably import VariablyClient

client = VariablyClient({"api_key": "your-api-key"})

stream = client.evaluate_prompt_stream(
    experiment_key="rag-prompt-experiment",
    user_context={"user_id": "user-123"},
    input_variables={"query": "What are the symptoms of Type 2 diabetes?"},
)

# Tokens arrive one-by-one for real-time display
for token in stream:
    print(token, end="", flush=True)

print()  # newline after stream ends

# After iteration, metadata is available (token usage, latency, quality score)
meta = stream.metadata
if meta:
    print(f"Model: {meta.model}, Latency: {meta.latency_ms}ms")
    print(f"Tokens: {meta.token_usage}")

Context-Aware Evaluation (Better RAG Quality) — v2.2.0+

For RAG chatbots, passing conversation history and retrieved chunks enables groundedness scoring, hallucination detection, and conversational coherence — dimensions that are impossible to evaluate in isolation.

The evaluation_context parameter is not sent to the LLM — it's only used by Variably's evaluator for richer scoring.

# Step 1: Collect conversation history from your session
workflow_history = [
    {"role": "user", "content": "What causes diabetes?"},
    {"role": "assistant", "content": "Key factors include genetics, diet..."},
    {"role": "user", "content": "What about potatoes?"},
]

# Step 2: Collect retrieved RAG chunks (after your retrieval step)
reference_materials = [
    {
        "id": "chunk-001",
        "content": "Unhealthy diets high in refined sugars, fats...",
        "source": "Kenya National Clinical Guidelines",
        "type": "chunk",
        "relevance_score": 0.89,
    },
    {
        "id": "chunk-002",
        "content": "Modifiable risk factors include obesity...",
        "source": "Kenya National Clinical Guidelines",
        "type": "chunk",
        "relevance_score": 0.82,
    },
]

# Step 3: Pass evaluation_context in your evaluate call
response = client.evaluate_prompt(
    experiment_key="rag-prompt-experiment",
    user_context={"user_id": "user-123"},
    input_variables={"query": "What about potatoes?", "context": context_text},
    evaluation_mode="full",
    evaluation_context={
        "reference_materials": reference_materials,
        "workflow_history": workflow_history,
        "retrieval_query": "potato consumption glycemic index diabetes risk",
    },
)

# Same works with streaming
stream = client.evaluate_prompt_stream(
    experiment_key="rag-prompt-experiment",
    user_context={"user_id": "user-123"},
    input_variables={"query": "What about potatoes?", "context": context_text},
    evaluation_context={
        "reference_materials": reference_materials,
        "workflow_history": workflow_history,
    },
)
for token in stream:
    print(token, end="", flush=True)

What this enables:

Dimension Description Requires
faithfulness % of claims grounded in retrieved chunks reference_materials
hallucination_rate % of claims with no source in context reference_materials
context_utilization % of relevant chunks actually used reference_materials
attribution_accuracy Do citations map to correct chunks? reference_materials
conversation_consistency No contradictions with prior turns workflow_history
context_retention Maintains topic awareness across turns workflow_history
transparency Discloses when going beyond source material reference_materials

BYOR mode also supports evaluation_context — pass it in submit_response():

result = client.submit_response(
    experiment_key="my-experiment",
    variant_key=variant.variant_key,
    executed_prompt=prompt,
    response=llm_response,
    user_context=user_context,
    input_variables=input_variables,
    provider=variant.provider,
    model=variant.model,
    latency_ms=latency,
    evaluation_context={
        "reference_materials": reference_materials,
        "workflow_history": workflow_history,
    },
)

evaluation_context Schema

Field Type Description
reference_materials list[dict] RAG chunks / source documents for groundedness scoring
reference_materials[].id str Unique chunk identifier
reference_materials[].content str Chunk text content
reference_materials[].source str (optional) Source document URL or name
reference_materials[].type str (optional) e.g. "chunk", "document"
reference_materials[].relevance_score float (optional) Retriever similarity score
workflow_history list[dict] Conversation turns for coherence scoring
workflow_history[].role str "user" or "assistant"
workflow_history[].content str Message content
retrieval_query str (optional) The rewritten query sent to the retriever

See Context-Aware RAG Evaluation for the full concept doc with architecture diagrams and integration examples.

LangGraph auto-instrumentation

from variably.integrations.langgraph import instrument

instrument(graph, session_id_key="session_id")   # uncompiled StateGraph, before .compile()

Every node is wrapped; a node run is sent when it has answer text, tool calls, a hand-off (sent as a router step; a tool_use step when its tool calls captured SQL, rows, documents or files) or is the aggregator. Nodes with none of these — e.g. a retriever returning {"docs": [...]} — are not sent, so they are not scored as answers. Pass log_empty_nodes=True (also on @observe_node) to send every node run.

Tool calls are read from the node's messages (each ToolMessage matched to the call that asked for it): SQL in the arguments is sent as sql, the rows it returned, retriever documents, and files / diffs a code tool read as reference materials — no context_extractor needed for those. A node that only returns tool results (ToolNode) is a kind="tool_use" step. The aggregator's answer is checked against everything the run gathered (see "Connect by app shape" below). sql_tools={"bq_execute": "statement"} names the SQL argument of an unusual query tool.

Connect by app shape: what the integrations capture (RFC 135)

The framework integrations collect the inputs for you; you write no observe call.

Built with Setup
Google ADK / Vertex AI Agent Engine Runner(..., plugins=[VariablyPlugin()]) (variably.integrations.adk)
LangGraph instrument(graph, session_id_key="thread_id") before compile()
Microsoft Agent Framework (Python) variably.integrations.agent_framework.instrument() once at start-up
Azure AI Foundry Agent Service AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED=true, variably.integrations.otel.instrument(), then Azure's tracing instrumentor
MLflow traces variably.integrations.mlflow.log_traces(...)

From each tool call (the same rules as the server uses for OpenTelemetry):

  • SQL: an argument named sql, query, statement, sql_query or query_text that starts like SQL and parses as SQL (with sqlglot from the [sql] extra; otherwise a stricter pattern) is sent as sql and checked against the project's SQL checks. "select the best plan for a family of four" is not SQL. DESCRIBE / DESC / SHOW / EXPLAIN count only in sql, statement, sql_query or query_text, never in the generic query ("describe refunds" sent to a search tool is a question). Sent also with capture_text=False, unless capture_sql=False.
  • Rows: what that call returned (a list of records, {"rows"}, {"records"}, {"data"}, {"results"}, {"columns" | "column_names", "rows"} with names or {"name": …} objects, ADK's {"result": …}, or JSON text of these), as {"id": "run_sql#1", "source": "run_sql", "rows": [...]}; n is one counter for the whole run. At most 2,000 rows per result (metadata["rows_truncated"]); a query that returned no rows is named in metadata["empty_results"].
  • sql_tools: {"bq_execute": "statement"} names the SQL argument of an unusual query tool; {"lookup_member": None} marks a tool whose records are rows although it takes no SQL. ADK (VariablyPlugin(sql_tools=...)), LangGraph (instrument(..., sql_tools=...)), MLflow (log_traces(..., sql_tools=...)), and over OpenTelemetry otel.instrument(sql_tools=...) / agent_framework.instrument(sql_tools=...): sent as the resource attribute variably.sql_tools (stamped on every span when your tracer provider already exists; with your own provider and span_exporter(), use Resource.create(otel.sql_tools_attributes({...})) or add otel.SqlToolsSpanProcessor({...})).
  • Files / diffs (code review): a result with path + content, a list of those, unified-diff text (id "diff", source the tool's name), or a path argument whose result is the file's text. A write or a post (arguments carrying content / text / body / new_content) contributes no file.
  • Documents: retriever results (documents, results, chunks, … of items with text).

Several agents: the step that writes the final answer carries its own evidence plus every earlier step's rows, documents and files of the same run (each earlier piece tagged origin_agent; its own references are not), is sent with evidence_scope="run", lists earlier steps' SQL in metadata["run_sql"] (checked once, on the step that ran it) and their zero-row results in metadata["empty_results"]. Never an earlier step's plain tool output, and never the answer of an agent called as a tool: that call is sent with "wraps_agent": true (ADK AgentTool; LangGraph when a tool returns an instrumented sub-graph's answer). The run's evidence is deduplicated by id, source and the full content, and capped at 200 references, 5,000 rows and 100,000 characters per document; a cut is said in metadata["run_evidence_truncated"].

Every step carries metadata["run_id"]: ADK the top-level invocation id (also as invocation_id, shared by AgentTool sub-runs), LangGraph a new id per invocation of the compiled graph (so a turn whose final node raised or wrote nothing never passes its evidence to the next turn on the same thread), MLflow the trace id. Hand-written code: observe(..., session_id=run, agent_name="analyst", is_aggregator=True, evidence_scope="run"); evidence_scope="run" without is_aggregator=True and a session_id is refused before sending.

MLflow traces

import mlflow
from variably.integrations.mlflow import log_traces

traces = mlflow.search_traces(experiment_ids=["1234"], return_type="list")
print(log_traces(traces, dry_run=True))   # the log() payloads; nothing sent
print(log_traces(traces, environment="prod"))

One answer per AGENT span (or the root span): question and answer from the span's inputs and outputs, RETRIEVER documents as reference materials (scope metadata such as plan_year kept), TOOL spans as tool calls (SQL text as sql, the rows that call returned as a rows reference), model and tokens from CHAT_MODEL / LLM spans, an agent's handoff object as its hand-off to the next agent. Also takes .to_dict() / .to_json() output and JSON / JSONL export paths (MLflow not required). Unrecognised shapes are skipped with the reason in the report. A trace is one run: every answer carries metadata["run_id"] (the trace id) and the last top-level agent's answer is checked against the whole trace (evidence_scope="run", as above). sql_tools= as above. Live traces: set MLFLOW_USE_DEFAULT_TRACER_PROVIDER=false and call variably.integrations.otel.instrument().

Integration with LangGraph / FastAPI streaming

from fastapi.responses import StreamingResponse

async def stream_with_variably(query: str, session_id: str):
    """Yield NDJSON events from Variably streaming evaluation."""
    stream = client.evaluate_prompt_stream(
        experiment_key="my-experiment",
        user_context={"user_id": session_id},
        input_variables={"query": query},
    )

    for token in stream:
        yield json.dumps({"type": "token", "content": token}) + "\n"

    # Send final metadata
    if stream.metadata:
        yield json.dumps({
            "type": "stream_end",
            "content": stream.metadata.content,
        }) + "\n"

@app.post("/api/chat")
async def chat(request: ChatRequest):
    return StreamingResponse(
        stream_with_variably(request.message, request.session_id),
        media_type="application/x-ndjson",
    )

Backend API: SSE Streaming Endpoint

The streaming endpoint uses Server-Sent Events (SSE). Here's the raw API:

Endpoint: POST /api/v1/internal/sdk/prompt-experiments/evaluate-stream

Headers:

X-API-Key: your-api-key
Content-Type: application/json

Request body (same as non-streaming evaluate):

{
  "experiment_key": "rag-prompt-experiment",
  "user_context": {
    "userId": "user-123",
    "sessionId": "sess-456"
  },
  "input_variables": {
    "query": "What are the symptoms of Type 2 diabetes?"
  },
  "evaluation_context": {
    "reference_materials": [{"id": "chunk-1", "content": "...", "source": "...", "type": "chunk"}],
    "workflow_history": [{"role": "user", "content": "..."}],
    "retrieval_query": "diabetes symptoms type 2"
  }
}

Response (SSE stream):

event: token
data: {"content": "Type"}

event: token
data: {"content": " 2"}

event: token
data: {"content": " diabetes"}

event: token
data: {"content": " symptoms"}

event: token
data: {"content": " include..."}

event: metadata
data: {"experiment_id": "exp-123", "variant_id": "variant-a", "execution_id": "eval-789", "provider": "anthropic", "model": "claude-3-5-haiku-20241022", "prompt_tokens": 150, "completion_tokens": 85, "total_tokens": 235, "cost_usd": 0.000425, "latency_ms": 1250}

event: done
data: {}

curl example:

curl -N -X POST http://localhost:8080/api/v1/internal/sdk/prompt-experiments/evaluate-stream \
  -H "X-API-Key: your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "experiment_key": "rag-prompt-experiment",
    "user_context": {"userId": "user-123", "sessionId": "sess-456"},
    "input_variables": {"query": "What are the symptoms of Type 2 diabetes?"}
  }'

Error handling: If an error occurs during streaming, an error event is sent:

event: error
data: {"message": "LLM generation failed: rate limit exceeded"}

Business rules

set_business_rules(experiment_id, rules, note=None) saves the rules every answer must obey, checked as "Follows business rules". rules is a dict {"facts": [...], "rules": [...]} or a .json / .yaml file (a file may keep them under a top-level business_rules: key, as the templates in the docs do). Facts are typed values read from the answer; each rule is one check (at_most, at_least, between, one_of, none_of, within, required, equals, is) with optional when / unless:

client.set_business_rules(experiment_id, {
    "facts": [{"name": "approved_visits", "type": "number", "labels": ["approve", "visits approved"]},
              {"name": "prior_auth", "type": "boolean", "labels": ["prior authorization"]}],
    "rules": [{"name": "PT visits need prior authorization after 12",
               "check": {"fact": "approved_visits", "at_most": 12},
               "unless": {"fact": "prior_auth", "is": True}}],
}, note="visit limit 12")

The server checks the rules when saved (variably policy validate checks them in the policy YAML too); {} removes them. From CI: python -m variably.settings push --business-rules settings/business_rules.yaml.

Settings as code (code lists, hand-offs, SQL checks, posting rules, business rules, chart of accounts)

set_code_lists, set_handoff_contracts, set_sql_checks, set_scope_defaults, set_posting_rules, set_business_rules, set_chart_of_accounts (a list, or a .csv / .json / .yaml file; policy projects only) and set_requirement_templates (and python -m variably.settings push in CI) change how answers are scored, so they need a project API key with the policy.write scope — "Change the evaluation policy" when you create the key in the dashboard. Without it the call is refused with 403 scope_required. Keys created before this scope existed do not have it: create a new key for the job that pushes settings, and keep event and logging keys without it.

client = VariablyClient(api_key=os.environ["VARIABLY_POLICY_KEY"])  # a key with policy.write
client.set_code_lists(experiment_id, "settings/code_lists.json", note="add NJ leave types")

Policy as code (variably policy)

The project's evaluation policy as a YAML file in your repository. The project is the API key's (VARIABLY_API_KEY, VARIABLY_BASE_URL); push and publish need the policy.write scope.

variably policy pull -o variably-policy.yaml       # the version in force, as YAML
variably policy validate variably-policy.yaml      # check it without saving anything
variably policy diff variably-policy.yaml          # what pushing it would change
variably policy push variably-policy.yaml --note "add NJ leave types"
variably policy publish --note "reviewed in PR 42"

validate checks the file exactly as push would — the YAML itself, every section, and the cross-references between them (an account a posting rule names must be in the chart of accounts, two scoring rules cannot share a name) — and saves nothing. Every problem is printed as line N: path: message, warnings as warning: line N: path: message. Exit code: 0 valid (warnings never fail), 1 invalid, 2 could not ask (no key, network, server error). Run it in CI before push:

variably-policy.yaml: invalid - 1 problem to fix before this policy can be saved (2 scoring rules, 3 posting rules, 40 accounts)
line 31: evaluation_config.posting_rules[0].debit_accounts[0]: posting rule 'Inventory purchases': debit account 1250 is not in the chart of accounts (add it, or use 1200 Inventory)

Configuration

from variably import VariablyConfig, VariablyClient

config = VariablyConfig(
    api_key="your-api-key",
    base_url="https://api.variably.com",  # default: http://localhost:8080
    environment="production",  # default: development
    timeout=5000,  # timeout in milliseconds, default: 5000
    retry_attempts=3,  # default: 3
    enable_analytics=True,  # default: True
    cache={
        "ttl": 300,  # TTL in seconds, default: 300 (5 minutes)
        "max_size": 1000,  # default: 1000
        "enabled": True  # default: True
    },
    log_level="INFO"  # DEBUG, INFO, WARNING, ERROR
)

client = VariablyClient(config)

Advanced Usage

Environment Variables

You can create a client using environment variables:

from variably import create_client_from_env

# Uses these environment variables:
# VARIABLY_API_KEY (required)
# VARIABLY_BASE_URL
# VARIABLY_ENVIRONMENT
# VARIABLY_TIMEOUT
# VARIABLY_RETRY_ATTEMPTS
# VARIABLY_ENABLE_ANALYTICS
# VARIABLY_LOG_LEVEL

client = create_client_from_env()

Different Flag Types

# Boolean flags
bool_value = client.evaluate_flag_bool("feature-enabled", False, user_context)

# String flags
string_value = client.evaluate_flag_string("theme", "light", user_context)

# Number flags
number_value = client.evaluate_flag_number("max-items", 10, user_context)

# JSON flags
json_value = client.evaluate_flag_json("config", {"timeout": 5000}, user_context)

# Get full evaluation details
result = client.evaluate_flag("feature-flag", "default", user_context)
print(f"Value: {result.value}, Reason: {result.reason}, Cache Hit: {result.cache_hit}")

Batch Evaluation

flags = client.evaluate_flags([
    "feature-a",
    "feature-b", 
    "feature-c"
], user_context)

print(flags["feature-a"].value)

Event Tracking

from datetime import datetime

# Single event
client.track({
    "name": "purchase_completed",
    "user_id": "user-123",
    "properties": {
        "amount": 99.99,
        "currency": "USD",
        "items": ["item-1", "item-2"]
    },
    "timestamp": datetime.utcnow()  # optional, auto-generated if not provided
})

# Batch events
client.track_batch([
    {"name": "page_view", "user_id": "user-123", "properties": {"page": "/home"}},
    {"name": "button_click", "user_id": "user-123", "properties": {"button": "cta"}}
])

Cache Management

# Clear cache
client.clear_cache()

# Get cache stats
stats = client.cache.get_stats()
print(stats)  # {"size": 10, "max_size": 1000, "enabled": True, "ttl": 300}

Metrics

# Get SDK metrics
metrics = client.get_metrics()
print(metrics)
# {
#     "api_calls": 25,
#     "cache_hits": 15,
#     "cache_misses": 10,
#     "errors": 1,
#     "average_latency": 45.2,
#     "cache_hit_rate": 0.6,
#     "error_rate": 0.04,
#     "flags_evaluated": 20,
#     "gates_evaluated": 5,
#     "events_tracked": 12,
#     "start_time": "2023-10-01T12:00:00Z",
#     "uptime_seconds": 3600
# }

Context Manager

# Use with context manager for automatic cleanup
with VariablyClient({"api_key": "your-api-key"}) as client:
    result = client.evaluate_flag_bool("feature", False, user_context)
    # client.close() is called automatically

Custom Logger

from variably import VariablyClient, create_logger

# Create custom logger
logger = create_logger(
    name="my-app",
    level="DEBUG",
    structured=True,  # JSON logging
    silent=False
)

# Client will use the custom logger
client = VariablyClient({
    "api_key": "your-api-key",
    "log_level": "DEBUG"
})

Error Handling

from variably import (
    VariablyError,
    NetworkError,
    AuthenticationError,
    ValidationError,
    RateLimitError,
    TimeoutError,
    ConfigurationError
)

try:
    result = client.evaluate_flag("my-flag", False, user_context)
except AuthenticationError:
    print("Invalid API key")
except NetworkError as e:
    print(f"Network error: {e.status_code}")
except ValidationError as e:
    print(f"Validation error in field: {e.field}")
except RateLimitError as e:
    print(f"Rate limited, retry after {e.retry_after} seconds")
except TimeoutError:
    print("Request timed out")
except ConfigurationError as e:
    print(f"Configuration error in parameter: {e.parameter}")
except VariablyError as e:
    print(f"Variably SDK error: {e}")

Type Hints

The SDK includes full type hints for better IDE support:

from typing import Dict, Any
from variably import VariablyClient, UserContext, FlagResult

user_context: UserContext = {
    "user_id": "user-123",
    "email": "user@example.com",
    "attributes": {
        "plan": "premium",
        "signup_date": "2023-01-01"
    }
}

result: FlagResult = client.evaluate_flag("feature", False, user_context)

Async Support

For async applications, you can wrap the synchronous client:

import asyncio
from concurrent.futures import ThreadPoolExecutor
from variably import VariablyClient

class AsyncVariablyClient:
    def __init__(self, config):
        self.client = VariablyClient(config)
        self.executor = ThreadPoolExecutor(max_workers=4)
    
    async def evaluate_flag_bool(self, flag_key, default_value, user_context):
        loop = asyncio.get_event_loop()
        return await loop.run_in_executor(
            self.executor,
            self.client.evaluate_flag_bool,
            flag_key, default_value, user_context
        )
    
    async def close(self):
        self.client.close()
        self.executor.shutdown(wait=True)

# Usage
async def main():
    client = AsyncVariablyClient({"api_key": "your-api-key"})
    
    result = await client.evaluate_flag_bool("feature", False, {
        "user_id": "user-123"
    })
    
    await client.close()

asyncio.run(main())

Development

Setup

# Install development dependencies
pip install -e ".[dev]"

Testing

pytest

Code Quality

# Format code
black src/ tests/

# Sort imports
isort src/ tests/

# Lint
flake8 src/ tests/

# Type check
mypy src/

Publishing to PyPI

Prerequisites

  1. Create a PyPI account at https://pypi.org/account/register/
  2. Generate an API token at https://pypi.org/manage/account/token/
    • Scope: select "Entire account" for first upload, or project-specific after that
  3. Install build tools:
    pip3 install build twine
    

Note: build and twine install to user site-packages and may not be on your PATH. Always use python3 -m build and python3 -m twine instead of bare build/twine.

Configure PyPI credentials

Create ~/.pypirc:

[distutils]
index-servers = pypi

[pypi]
username = __token__
password = pypi-YOUR_API_TOKEN_HERE

Secure the file:

chmod 600 ~/.pypirc

Build and publish

The version in the build output (e.g., variably_sdk-2.0.0-py3-none-any.whl) comes directly from pyproject.toml's version field. PyPI rejects re-uploads of the same version — you must bump the version to publish again.

# 1. Clean previous builds
rm -rf dist/ build/ src/*.egg-info

# 2. Build sdist and wheel
python3 -m build

# 3. Verify the package (optional but recommended)
python3 -m twine check dist/*

# 4. Upload to TestPyPI first (optional, for dry-run)
python3 -m twine upload --repository testpypi dist/*

# 5. Upload to PyPI
python3 -m twine upload dist/*

Verify the published package

pip3 install variably-sdk==2.1.0
python3 -c "from variably import VariablyClient, PromptVariant; print('OK')"

Version bumping checklist

When releasing a new version, update these files then clean-build-publish:

  1. src/variably/version.py — __version__
  2. pyproject.toml — version
  3. src/variably/http_client.py — User-Agent header string
  4. CHANGELOG.md — the release notes (new features and behaviour changes)
# Example: bumping from 2.0.0 to 2.0.1
# After updating the 3 files above:
rm -rf dist/ build/ src/*.egg-info
python3 -m build
python3 -m twine upload dist/*

Requirements

  • Python 3.7+
  • requests >= 2.25.0

License

MIT License - see LICENSE file for details.

Metadata

Release files for variably-sdk 2.19.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for variably-sdk 2.19.0
File Size Uploaded Provenance Actions
variably_sdk-2.19.0.tar.gz 203.3 kB Without provenance Details for variably_sdk-2.19.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for variably-sdk 2.19.0
File Interpreter ABI Platform Actions
variably_sdk-2.19.0-py3-none-any.whl Python 3 none any Details for variably_sdk-2.19.0-py3-none-any.whl

Total release size: 337.8 kB

Release files / variably_sdk-2.19.0.tar.gz

Download URL variably_sdk-2.19.0.tar.gz
Size 203.3 kB
Tags Source
SHA-256 checksum
How to use checksums
bc3670edd99d38365008884e48487f8cc2854175ba265fe62a8ed25b9241bfb3
BLAKE2b-256 checksum
How to use checksums
c9f76da21da952471f6373220919f91cbb124ff34672c0e751444717d5a13220
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / variably_sdk-2.19.0-py3-none-any.whl

Download URL variably_sdk-2.19.0-py3-none-any.whl
Size 134.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4a16a0defcc73781a3a219d5dbcc4539c0c14bbced5f8588920325ee725d8fdc
BLAKE2b-256 checksum
How to use checksums
538475c97d9a90f7a2b40ab33a2961e6d2d2e9bdb0036454b9be1cfc57a4e741
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for variably-sdk 2.19.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page