foam-otel 2.0.0

Foam's OpenTelemetry instrumentation package — the Python implementation of the fleet base-package spec.

foam-otel

Foam's Python base package: a thin, safe wrapper around the official OpenTelemetry libraries that turns on automatic instrumentation, ships traces, metrics, and logs to foam, and hands you a small set of bulletproof helper functions. One init() at process start is the whole integration — everything else is optional. Built on OpenTelemetry (the Python API/SDK, the OTLP-over-HTTP exporters, and the official opentelemetry-instrumentation-* contrib packages); see THIRD-PARTY-NOTICES for full credits.

This README is the FDE manual — everything you need to integrate, verify, and troubleshoot lives here, not in foam's source.

Install

pip install foam-otel

foam-otel depends only on the OpenTelemetry API/SDK, the OTLP-HTTP exporter, and the instrumentation base. Framework and client instrumentations arrive as extras — installing one pulls the official contrib package, and init() auto-activates it (the entry-point sweep, below). No adapter code ships in foam.

pip install "foam-otel[fastapi]"   # + opentelemetry-instrumentation-fastapi
pip install "foam-otel[httpx]"     # + opentelemetry-instrumentation-httpx

Any other official instrumentation you install (pip install opentelemetry-instrumentation-requests, -botocore, -psycopg, …) is activated the same way at init() — presence-checked, never blind-registered.

The scenario matrix

One copy-paste-ready snippet per supported situation (spec rule 33). If your situation is missing here, that is a docs bug — file it. All calls use the real init(...) keyword arguments. Load foam's init() before your app imports the libraries you want instrumented — instrumentation that registers after a library is imported never applies to already-bound references (GOTCHAS G1).

# 1 — CLEAN SERVICE (door 1, the common case) — run at the top of your entrypoint
import os
from foam_otel import init

env = os.environ.get("APP_ENV", "development")
init(
    name="checkout-api",                 # -> service.name
    environment=env,                     # -> deployment.environment.name (verbatim)
    enabled=env not in ("test", "ci"),   # the one export switch (idiomatic recipe)
    token=os.environ["FOAM_OTEL_TOKEN"],      # Bearer auth; required only when enabled
    version=os.environ.get("GIT_SHA"),   # -> service.version (recommended)
)

# 2 — BESIDE A PROPRIETARY AGENT (situation A): disjoint pipelines.
# Foam runs its own pipeline next to the agent's; you wire the agent's intake
# host into the loop guard so foam never traces the agent's own egress.
init(
    name="checkout-api", environment=env, enabled=True, token=FOAM_OTEL_TOKEN,
    ignored_outbound_hosts=["agent-intake.vendor.example"],
)

# 3 — TENANT RIDES FOAM (situation C): a scoped SDK (LLM eval / AI-obs) attaches
# a processor to foam's pipeline. Pass a CONSTRUCTED instance + the REQUIRED
# ignore entry for its exporter's host (or its export traffic becomes a loop).
from eval_tool_sdk import EvalToolSpanProcessor

init(
    name="checkout-api", environment=env, enabled=True, token=FOAM_OTEL_TOKEN,
    additional_span_processors=[EvalToolSpanProcessor(project="prod")],
    ignored_outbound_hosts=["ingest.eval-tool.example"],
)

# 4 — FOREIGN OTEL SDK OWNS A SIGNAL (situation B): door 2 — the ingest entries.
# ONE LINE per claimed signal, added to THEIR OTel setup. The tap is an ADDITIVE
# READER: their pipeline keeps working exactly as before, and now ALSO sends
# foam a masked, identity-stamped COPY. Full contract + the REQUIRED loop step:
# see "Door 2 — the ingest entries" below.
from foam_otel import (
    create_foam_ingest_span_processor,
    create_foam_ingest_log_record_processor,
    create_foam_ingest_metric_reader,
)

# THEIR setup module — one added line per claimed signal:
their_tracer_provider.add_span_processor(create_foam_ingest_span_processor(
    token=os.environ["FOAM_OTEL_TOKEN"], environment=APP_ENV))
their_logger_provider.add_log_record_processor(create_foam_ingest_log_record_processor(
    token=os.environ["FOAM_OTEL_TOKEN"], environment=APP_ENV))
their_meter_provider = MeterProvider(          # THEIR constructor line — THE recipe
    resource=their_resource,
    metric_readers=[their_reader, create_foam_ingest_metric_reader(
        token=os.environ["FOAM_OTEL_TOKEN"], environment=APP_ENV)],
)
# If any signal is free, ALSO run init() — the doors compose (scenario 5).

# 5 — PER-SIGNAL COMPOSITION (traces claimed shown; symmetric over any subset).
# A foreign SDK owns traces only; metrics + logs are free.
init(name="checkout-api", environment=env, enabled=True, token=FOAM_OTEL_TOKEN)
# -> warns ONCE: "traces is owned by <owner> — foam is dark for traces", and the
#    warning names the one-line fix. metrics + logs register to foam on the FULL
#    contract. Trace ids still correlate through the shared W3C context.
their_tracer_provider.add_span_processor(create_foam_ingest_span_processor(
    token=FOAM_OTEL_TOKEN, environment=env))   # the tap rides THEIR tracer
# -> foam now ALSO receives their spans (tier-marked foam.ingest.tier=external,
#    carrying THEIR service identity); door-1 metrics/logs carry foam's identity.
# (Python has per-signal globals, so the doors compose — unlike Java's single
#  composite global.)

# 6 — TESTS / CI: fully inert. No token needed — token validates only when enabled.
init(name="checkout-api", environment="test", enabled=False)
# No SDK loads, no providers registered, no network, no warning. Helpers no-op.

# 7 — SERVERLESS (door 1 + the flush recipe). The platform freezes the process
# the instant the handler returns; flush() before it does, or buffered telemetry
# is lost. flush() never raises and is safe even if init() never ran.
from foam_otel import init, flush

init(name="orders-fn", environment="production", enabled=True, token=FOAM_OTEL_TOKEN)

def handler(event, context):
    try:
        return run(event)
    finally:
        flush()

Scenario 8 (browser) does NOT apply to Python. It exists only in the JS core's matrix — the browser package is a separate companion, build-time gated with a browser-scoped token, and this Python core has no browser surface.

init options

init() is keyword-only. Required arguments raise at boot (rule 10) so a misconfiguration fails on your machine, never silently in production. The unhappy path, shown (every other foam function no-ops instead of raising — init() is the ONE deliberate exception):

# MISCONFIGURED BOOT: a missing/blank required argument raises at init —
# the failure is a red CI run, never a dark production service.
from foam_otel import init

init(name="", environment="production", enabled=True, token="t0k")
# -> ValueError: [foam] init: name is required and must be a non-empty string
# Same shape for a blank/non-string `environment`, a non-bool `enabled`
# (TypeError), and a missing/blank `token` while enabled=True.

Option	Type	Required	Default	What it changes on the wire / why
name	str	YES	—	service.name on every export. Blank/non-string raises ValueError at init.
environment	str	YES	—	deployment.environment.name, exported VERBATIM. Values outside {production, staging, development, test} warn (typo guard) but export unchanged. Blank/non-string raises.
enabled	bool	YES	—	The ONE export switch. False = fully inert: no SDK import, no providers, no network, no warning, token untouched. True = export in that environment. Non-bool raises TypeError. No default because "on by accident in CI" and "off by accident in prod" are both incidents.
token	str	when enabled=True	None	Authorization: Bearer <token> on every OTLP export. Validated (and stripped) ONLY when enabled=True and the kill switch is unset; blank/missing there raises ValueError. Never read from the environment BY THE PACKAGE — you wire it from the service's real secret source. Fleet convention: read it from the FOAM_OTEL_TOKEN env var.
version	str	no	None	service.version, verbatim (git SHA recommended). Omitting warns — foam never detects a deploy id at runtime.
redact_keys	Sequence[str]	no	None	EXTENDS the always-on secrets floor (never replaces it). Matched keys get the tail mask (********cafe); a matched key holding a dict/list masks in full. Normalized-token match: ["apiKey"] also catches api_key, x-api-key.
redact_pii_keys	Sequence[str]	no	None	YOUR OWN PII field names. Always FULL [REDACTED], never a tail (last-4 of an email is the domain). Foam ships no PII preset and infers nothing — PII posture is your call.
additional_instrumentations	Sequence[object]	no	None	Constructed instrumentor instances registered on foam's pipeline, fault-isolated: one throwing instance is skipped with a loud [foam] warning naming it, boot proceeds.
additional_span_processors	Sequence[object]	no	None	Tenant seam (traces). Your SpanProcessor joins foam's tracer provider AFTER the redaction stage — you see the masked view, byte-identical to what foam exports. A throwing instance is DISABLED and warned; foam's export is unaffected.
additional_log_record_processors	Sequence[object]	no	None	Same seam, logs (LogRecordProcessor).
additional_metric_readers	Sequence[object]	no	None	Same seam, metrics. NOTE: a MetricReader TRIGGERS collection on its own schedule/thread — that is your code on your thread; foam's never-throw guard covers only foam-invoked calls (flush/shutdown), not your reader's collect loop.
ignored_outbound_hosts	Sequence[str]	no	None	EXTENDS the outbound loop guard (rule 24), never replaces it. Explicit hostnames whose outbound calls produce no client spans. REQUIRED for any tenant/agent exporting over HTTP from inside your process (scenarios 2/3).
diagnostics	bool	no	False	Turns on [foam] self-narration at INFO (the init-ok line with per-signal ownership + instrumentation count, skipped-instrumentation notes). Off = the same messages log at DEBUG. Telemetry data never appears here.

Unknown keyword arguments are ignored with a single [foam] warning naming them (forward/backward-compat safety) — they never raise.

Deliberately absent knobs (documented AS absent)

• endpoint — the fleet endpoint (https://otel.api.foam.ai) is pinned in code. The ONE override is the operator-level OTEL_EXPORTER_OTLP_ENDPOINT env var (warns loudly). Never an init option: a code change must not silently re-route telemetry.
• sampling / any sampler knob — none. Every span ships (ParentBased(ALWAYS_ON)); an unsampled inbound flag cannot drop foam's spans, and OTEL_TRACES_SAMPLER is inert. Cost control is server-side (rule 4).
• cadence / batch-delay knobs — not init surface. Batch cadence is delegated to the standard OTEL_BSP_SCHEDULE_DELAY / OTEL_BLRP_SCHEDULE_DELAY / OTEL_METRIC_EXPORT_INTERVAL env vars (operator tuning), read by the SDK's own processors.
• disabled_environments — gone. enabled is an explicit boolean; compute it from your environment in one line (scenario 1).
• exporter / processor injection — the only pipeline seam is the three additive additional_* options. They can never redirect, drop, or displace foam's export.

Environment variables

Variable	Posture
OTEL_SDK_DISABLED=true	HONORED — the emergency kill switch (config-only, no redeploy). SUPERSEDES enabled=True: when set, foam is fully off and warns loudly. Parsed exactly like the SDK: literal true, case-insensitive, trimmed.
OTEL_EXPORTER_OTLP_ENDPOINT (+ OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT)	HONORED — the one operator-level override of the pinned fleet endpoint. Per-signal variant wins over the base (OTLP-spec precedence). Active override = one loud [foam] warning per signal naming the destination.
OTEL_BSP_SCHEDULE_DELAY, OTEL_BLRP_SCHEDULE_DELAY, OTEL_METRIC_EXPORT_INTERVAL	HONORED — batch/export cadence, delegated to the SDK's own batch processors and periodic reader. Positive integers only; anything else is ignored. Never customer (init) surface.
OTEL_PROPAGATORS=none	HONORED — the propagation kill lever: injection stops AND inbound extraction stops (the service becomes its own trace root), telemetry keeps flowing. Foam implements none itself and warns loudly. Any OTHER value warns "set but ignored" — foam's propagator set is fixed (W3C tracecontext + baggage).
OTEL_PYTHON_DISABLED_INSTRUMENTATIONS	INERT — set values warn loudly "set but ignored" and disable NOTHING: every installed instrumentation and both foam gap-fillers activate regardless. Deliberate: the honored table above is CLOSED, and an env lever that silently drops whole instrumentations' telemetry is exactly the class it bans. (Versions ≤ 1.1.x HONORED this var — see the migration notes below.)
OTEL_PYTHON_TRACER_PROVIDER / OTEL_PYTHON_METER_PROVIDER / OTEL_PYTHON_LOGGER_PROVIDER	INERT — foam registers by ATTEMPT (set-once, first wins), never by pre-reading a slot, so the API's entry-point provider loaders never fire on foam's account. Proven inert by a dedicated test per var.
TRACEPARENT / TRACESTATE	HONORED (inbound) — a launcher that sets these parents the whole process to its trace (spawn is an inbound call whose carrier is the environment). No-op when unset.
Every other OTEL_* (OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, OTEL_EXPORTER_OTLP_HEADERS/_TIMEOUT/_COMPRESSION, OTEL_SEMCONV_STABILITY_OPT_IN, OTEL_*_EXPORTER, attribute limits, …)	INERT — and inert BY CONSTRUCTION. Foam reads every honored var itself, then builds the whole pipeline inside a synchronous window that strips every OTEL_* var from os.environ (restored before init returns), so no un-consumed var reaches an exporter/provider/instrumentation constructor. Identity and auth come from init options only.

API — every public function

Every function below never throws and no-ops silently before init(), when enabled=False, or when the kill switch is active — the one exception is init() itself, which raises on developer error (missing/blank required options) so a misconfigured boot fails in CI, not dark in production. Import everything from the top-level foam_otel.

init(*, name, environment, enabled, token=None, version=None, redact_keys=None, redact_pii_keys=None, additional_instrumentations=None, additional_span_processors=None, additional_log_record_processors=None, additional_metric_readers=None, ignored_outbound_hosts=None, diagnostics=False) -> bool

Wire up foam telemetry (door 1). Validates arguments, then per signal ATTEMPTS to register foam's provider into OTel's set-once global and reads the outcome (attempt-then-read — foam never pre-reads a slot to predict, and never displaces an owner); a lost attempt warns once naming the owner and foam stays DARK for that signal only. Activates installed instrumentations (traces only), wires the loop guard, and installs the stdlib-logging root export. Idempotent: a second call warns and returns. Returns True when foam is exporting at least one signal, False when disabled / killed / every signal is foreign-owned or failed. See the options table for every parameter.

from foam_otel import init

owned = init(
    name="checkout-api",
    environment="production",
    enabled=True,
    token=os.environ["FOAM_OTEL_TOKEN"],
    version=os.environ.get("GIT_SHA"),
)
# owned is True when >= 1 signal reaches foam

flush() -> None

Force-flush every foam-owned signal without tearing anything down. Never raises; safe before init() (silent no-op). The serverless and pre-crash lifeline — call it before the process can freeze (scenario 7).

from foam_otel import flush

flush()  # batched spans/logs/metrics are on the wire when this returns

shutdown() -> None

Flush and stop export for every foam-owned signal, unregister the globals foam set (providers, the TRACEPARENT context token, the stdlib root handler), and reset to a pre-init state — so a later init() (e.g. a post-fork re-init) can run. Fail-open; safe without init; never raises. Helpers no-op afterward.

import signal, sys
from foam_otel import shutdown

def _graceful(*_):
    shutdown()
    sys.exit(0)

signal.signal(signal.SIGTERM, _graceful)

span(name, attributes=None)

A context manager: run a block inside a new ACTIVE span. On exception it records the exception, sets status ERROR, re-raises the IDENTICAL error, and ALWAYS ends the span. When foam is not active it yields None and runs the block uninstrumented — your code never behaves differently because telemetry is off. attributes, if given, are set on the span (masked).

from foam_otel import span

with span("price-cart", {"cart.items": len(cart)}) as s:
    total = price_cart(cart)   # if this raises, the error is recorded and re-raised

set_attribute(key, value) -> None

Set one attribute on the currently active span; no-op when none is active (never fabricates a span). The value rides the redaction engine, so a secret-named key is masked before it lands on the span.

from foam_otel import set_attribute

set_attribute("order.id", order.id)
set_attribute("api_key", key)   # -> "********" + last 4, masked before it exists on the span

set_attributes(attributes) -> None

Set many attributes on the active span at once; each value is masked by key. No-op without an active span.

from foam_otel import set_attributes

set_attributes({"customer.tier": "gold", "password": "sup3rs3cr3tvalue"})
# -> password: "********alue"

add_event(name, attributes=None) -> None

Add a timestamped event to the active span; attributes masked. No-op without an active span.

from foam_otel import add_event

add_event("cache.miss", {"key": cache_key})

record_exception(error, attributes=None) -> None

Record a standard OTel exception event (type/message/stacktrace) on the active span and set status ERROR. Accepts any BaseException, including your own error classes. NEVER notifies a vendor tracker — the bridge direction is tracker -> foam only. No-op without an active span. Attributes masked.

from foam_otel import record_exception

try:
    charge(card)
except PaymentError as err:
    record_exception(err)
    raise

get_trace_context() -> dict

The active span's ids for correlating foam traces with your OWN logs and outbound payloads: {"trace_id": <32 hex>, "span_id": <16 hex>}. Returns an EMPTY dict when no valid span is active — never fabricates ids.

from foam_otel import get_trace_context

ctx = get_trace_context()
if ctx:
    my_logger.info("charging card", extra={"trace_id": ctx["trace_id"]})

increment_counter(name, n=1, attributes=None) -> None

Add to a monotonic counter (create_counter). Instruments are lazily created and cached per name (rebound once if the provider swaps). Names pass through verbatim.

from foam_otel import increment_counter

increment_counter("orders_placed", 1, {"plan": "pro"})

record_histogram(name, value, attributes=None) -> None

Record a distribution observation (durations, sizes) via create_histogram.

import time
from foam_otel import record_histogram

record_histogram("checkout_duration_ms", (time.monotonic() - started) * 1000)

add_up_down_counter(name, n, attributes=None) -> None

Add to a counter that can decrease (in-flight counts, pool sizes), via create_up_down_counter. Pass a negative n to subtract.

from foam_otel import add_up_down_counter

add_up_down_counter("jobs_in_flight", +1)
# ... work ...
add_up_down_counter("jobs_in_flight", -1)

set_metric(name, value, attributes=None) -> None

Set a synchronous gauge — last value wins (queue depth, temperature-style readings), via create_gauge.

from foam_otel import set_metric

set_metric("queue_depth", queue.qsize())

Metrics — attribute discipline. Every distinct attribute combination is a live metric stream. increment_counter("requests", 1, {"user_id": uid}) is a memory leak and a flat dashboard (the SDK caps streams protectively). Keep attribute VALUES low-cardinality: plans, regions, status classes — never ids, raw paths, or emails.

log(severity, body, attributes=None) -> None

Emit a log record through foam's pipeline, trace-correlated to the active span. severity is one of trace|debug|info|warn|warning|error|fatal|critical (case-insensitive; anything unknown lands as INFO — never throws). The body and attributes ride the redaction engine (string bodies get the value-pattern pass; dict/list bodies deep-redact).

from foam_otel import log

log("warn", "payment retry scheduled", {"attempt": 2, "gateway": "stripe"})

Most apps don't call log() directly — they use stdlib logging, which foam bridges automatically (see install_root_export below).

redact(value) -> str

Foam's masking on demand, for your OWN sinks (your logger, an audit trail). Same fail-closed engine foam uses internally, with no key context: a scalar string/number gets the tail mask, everything else masks in full. Empty string on hard failure — never the unredacted payload. Never raises.

from foam_otel import redact

my_logger.info(f"token used: {redact(api_token)}")
# redact("sk-live-0123456789abcdef") -> "********cdef"   (len >= 12, tail kept)
# redact("short")                    -> "********"        (len < 12, full mask)
# redact({"nested": "secret"})       -> "********"        (non-scalar, full mask)

DEFAULT_REDACTED_KEYS is also exported (read-only tuple) — the always-on secrets floor redact_keys/redact_pii_keys extend. It is informational; you cannot disable it.

get_tracer(name=None) / get_meter(name=None) / get_logger(name=None)

The raw OTel Tracer / Meter / Logger on whatever pipeline owns the process globals (foam's when foam owns the signal; a foreign SDK's in inert mode). Everything the helper set doesn't wrap — span links, explicit span kinds, observable/async instruments, batch log emission — is reachable here; this is what keeps the helper set small and CLOSED. Default instrumentation scope is foam-otel (the package identity, matching the JS core's @foam-ai/otel; versions ≤ 1.1.x used app.custom — see the migration notes below). Before init they return safe no-op objects. NOTE: a span you start_span() here is yours to end() — a never-ended span leaks; the span() helper cannot leak by construction.

from foam_otel import get_tracer
from opentelemetry.trace import SpanKind

tracer = get_tracer("worker")
with tracer.start_as_current_span("drain-batch", kind=SpanKind.CLIENT, links=links) as s:
    drain()   # explicit kind + links — not wrapped by span(), reached via the passthrough

from foam_otel import get_meter
from opentelemetry.metrics import Observation

meter = get_meter("inventory")
def observe(options):
    return [Observation(current_stock())]
meter.create_observable_gauge("stock_level", callbacks=[observe])  # async instrument

install_root_export() -> bool

Attach a plain OTel LoggingHandler to the ROOT logger so every stdlib logger that propagates (the normal case) exports to foam, trace-correlated. init() calls this automatically when foam owns the logs signal — you rarely call it yourself. Idempotent: it stands down if any OTel export handler is already on root, and is RE-callable after logging.basicConfig(force=True) wipes root handlers. Returns True when a foam (or existing OTel) handler covers root, False when foam isn't active or owning logs. Fail-open.

import logging
from foam_otel import install_root_export

logging.basicConfig(force=True)   # your app resets root handlers, wiping foam's
install_root_export()             # re-attach foam's root export

export_handler(record_filter=None, transform=None)

Build a policy-drivable OTel export handler for loggers you attach it to YOURSELF — the path for libraries that set propagate=False and never reach the root handler. record_filter(record) -> bool gates each record; transform(copy) may scrub a COPY of the record (return None to drop it, so other handlers on that logger still see the original). Returns None when foam isn't active or doesn't own logs.

import logging
from foam_otel import export_handler

def only_warnings(record):
    return record.levelno >= logging.WARNING

handler = export_handler(record_filter=only_warnings)

attach_to_non_propagating(handler, *, skip_prefixes=("foam_otel", "opentelemetry")) -> list

Attach handler to every ALREADY-REGISTERED logger with propagate=False (libraries that detach from root, so the root export never sees them). Idempotent per logger by a marker attribute (not class identity), so a --reload re-import can't double-attach. Propagating loggers are never touched. Returns the list of logger names it attached to. Re-call after importing libraries that configure their loggers late.

from foam_otel import export_handler, attach_to_non_propagating

handler = export_handler()
if handler is not None:
    attached = attach_to_non_propagating(handler)   # e.g. ["browser_use", "some.detached.logger"]

Failure modes — what happens, and where the warning appears

All foam warnings are single-line, prefixed [foam], emitted on the foam_otel stdlib logger (visible on the root handler / your logging config).

Situation	Behavior
Missing/blank token while enabled=True (kill switch unset)	init() raises ValueError at boot — fail in CI, not dark in prod.
Blank/non-string name or environment	init() raises ValueError, naming the option.
Non-bool enabled	init() raises TypeError.
enabled=False	Fully inert and SILENT. No SDK import, no providers, no network, token never touched. Helpers no-op.
OTEL_SDK_DISABLED=true	Same inert posture, but LOUD: one [foam] warning that the kill switch superseded enabled=True.
environment outside {production, staging, development, test}	Exports verbatim, plus one [foam] warning (typo guard).
version omitted	Exports without service.version, plus one [foam] warning.
A foreign OTel SDK owns a signal's global	init() registers nothing there and warns ONCE per claimed signal naming the owner (see the coexistence guide); free signals still register to foam.
Export endpoint overridden via OTEL_EXPORTER_OTLP_ENDPOINT	One [foam] warning per signal naming the destination.
A broken exporter / instrumentation at runtime	That signal is disabled with a [foam] warning; the host app never crashes (fail-open per signal). Runtime detail is visible under diagnostics=True.
init() called twice	Second call warns [foam] init called twice … and returns the current ownership; the first init stands.

Coexistence guide — when foam is not alone

init() attempts to register foam's provider into each of the three global provider slots (traces, metrics, logs) — OTel's set-once, first-wins globals — and reads the outcome back: every FREE slot becomes foam's, and every CLAIMED slot warns once naming the owner (foam never pre-reads a slot to predict, and never displaces an owner). Three situations (rule 18):

• A — a proprietary APM agent beside foam (a vendor agent running its own private pipeline, no OTel globals). Disjoint pipelines; both run. Foam never touches the agent. Your job: add the agent's intake host to ignored_outbound_hosts (scenario 2) so foam never traces the agent's own export traffic. (If a modern agent ALSO registers an OTel global, that signal becomes situation B — the classifier decides per signal.)

• B — a foreign OTel SDK owns a signal's global. Foam's init() is dark for that signal — not degraded: NOTHING for that signal reaches foam through door 1. For each claimed signal, init() emits exactly one warning:

  [foam] traces is owned by <Owner> — foam is dark for traces; helper telemetry for it rides the foreign SDK to ITS backend. To also send that pipeline to foam, add one line in THEIR setup: create_foam_ingest_span_processor. See README "Door 2".

  What it means: door 1 registered nothing for that signal, and the fix is door 2 — the shipped create_foam_ingest_* entries ("Door 2 — the ingest entries" below): one line in THEIR setup taps their pipeline additively, so their export keeps working AND foam receives a masked, tier-marked copy. Your other options stay valid: remove the foreign SDK so foam owns the signal (scenario 1), or feed foam server-side from their collector. Your helpers keep working either way, but helper telemetry for the claimed signal rides the foreign provider and carries ITS resource identity, not foam's. (The owner name is read from the provider's class, MODULE-QUALIFIED — e.g. opentelemetry.sdk.trace.TracerProvider vs a vendor's ddtrace.opentelemetry.TracerProvider — because the bare class name is generic; a registered provider doesn't announce its vendor, so "an unknown OTel SDK" is the honest fallback.)

• C — a scoped tenant rides foam's pipeline (an LLM-eval / AI-observability SDK that attaches a processor/reader). Pass its CONSTRUCTED instance via additional_span_processors / additional_log_record_processors / additional_metric_readers (scenario 3). Guarantees: strictly ADDITIVE (foam's export is byte-identical with or without it), FAULT-ISOLATED (a throwing span/log instance is DISABLED loudly, foam unaffected), and ALREADY-MASKED (the redaction stage runs first, so the tenant only ever sees the masked view). REQUIRED: the tenant exporter's host in ignored_outbound_hosts — otherwise foam traces the tenant's export calls, which the tenant re-exports (a loop in the tenant's pipeline). A metric READER is different in kind: it collects on its own schedule/thread, so its collect loop is your code, not covered by foam's guard.

Claim severity is not uniform. A claimed MeterProvider costs foam little — RED metrics derive server-side from foam-owned spans; only custom metric-helper data diverts. A claimed TracerProvider costs the spans AND everything derived from them. Prioritize freeing traces first.

Late arrival. A foreign SDK can register AFTER foam boots. The same once-per-signal warning fires on the next helper use when the cached provider identity no longer matches the global — a coexistence warning is not only a boot-time event.

Door 2 — the ingest entries

When a foreign OTel SDK owns a signal (situation B), init() cannot serve it — door 2 does: one constructed instance per claimed signal, added by you — ONE LINE — to the CUSTOMER'S OWN OTel setup. Their pipeline keeps working exactly as before, and now ALSO sends foam a copy. The package provides the tap; you install the tap (consent is the line in their config — that is also the off-switch: there is no enabled param; to stop the tap, delete the line).

# signatures — identical shape across all three entries (keyword-only)
from foam_otel import (
    create_foam_ingest_span_processor,        # -> real SpanProcessor
    create_foam_ingest_log_record_processor,  # -> real LogRecordProcessor
    create_foam_ingest_metric_reader,         # -> real PeriodicExportingMetricReader
)

tap = create_foam_ingest_span_processor(
    token=os.environ["FOAM_OTEL_TOKEN"],  # REQUIRED — Bearer auth, same as init
    environment=APP_ENV,                  # REQUIRED — -> deployment.environment.name,
                                          #   stamped AT EXPORT TIME on foam's copy
    redact_keys=["internal_ref"],         # optional — EXTENDS the secrets floor
    redact_pii_keys=["customer_email"],   # optional — YOUR PII fields, full [REDACTED]
    diagnostics=False,                    # optional — tap-scoped [foam] narration
)

Validation is loud and at construction (the same bar as init()): a blank/missing token or environment raises a [foam]-prefixed ValueError at boot — never a dark tap. The ONE exception is the operator kill switch: under OTEL_SDK_DISABLED=true each entry returns a truly inert instance of the correct type (zero threads, zero exporters, zero network — your setup file keeps one shape) with one loud warning, and the token is NOT validated — the operator's off wins and must never crash your boot. The switch is read ONCE at construction; unset it and restart to re-enable.

# unhappy paths, shown:
create_foam_ingest_span_processor(token="", environment="production")
# -> ValueError: [foam] create_foam_ingest_span_processor: token is required
#    and must be a non-empty string
#
# OTEL_SDK_DISABLED=true ->
# [foam] OTEL_SDK_DISABLED=true — this foam ingest entry is disabled by the
# operator kill switch and will send nothing. Unset the variable and restart
# to re-enable.

Per-signal recipes

Traces and logs — post-construction attach (Python's providers support it):

their_tracer_provider.add_span_processor(create_foam_ingest_span_processor(
    token=os.environ["FOAM_OTEL_TOKEN"], environment=APP_ENV))
their_logger_provider.add_log_record_processor(create_foam_ingest_log_record_processor(
    token=os.environ["FOAM_OTEL_TOKEN"], environment=APP_ENV))

Metrics — constructor placement is THE recipe. Foam's reader collects on its own schedule beside theirs (multiple readers per MeterProvider is spec-legal); their Views shape what foam receives — an external-tier caveat, never fought:

their_meter_provider = MeterProvider(     # THEIR constructor line
    resource=their_resource,
    metric_readers=[their_reader, create_foam_ingest_metric_reader(
        token=os.environ["FOAM_OTEL_TOKEN"], environment=APP_ENV)],
)

MeterProvider.add_metric_reader() exists at the pinned floor and works as a FALLBACK when their MeterProvider(...) line is out of reach — but it is lifecycle-incomplete upstream: the provider's force_flush() and shutdown() walk only constructor-time readers, so a fallback-attached reader never receives the provider's final flush (GOTCHAS P16). Foam mitigates with an idempotent shutdown() plus its own atexit hook, which delivers the final collection at normal interpreter exit. Serverless caveat: constructor placement is REQUIRED on serverless — the platform freezes the process, so atexit never fires and a fallback-attached reader's last window is lost. One more invariant: every create_foam_ingest_metric_reader() call returns a fresh instance and each instance goes to exactly ONE provider — the SDK itself raises in THEIR constructor if one reader is registered twice. Never share one.

Door 2 — the required loop step

The tap's exports to foam are outbound HTTP calls THEIR instrumentation will span, and their pipeline hands those spans back to the tap — a feedback loop in THEIR pipeline that foam's own rule-24 guard cannot reach. Three layers stop it:

1. Foam wraps every tap export in the SDK's suppression context — official contrib instrumentations (requests/urllib/urllib3/httpx/aiohttp) bail out.
2. REQUIRED — your step: instrumentation that ignores the suppression key (vendor SDKs, mesh/eBPF layers) still spans foam's endpoint. Add the foam export host — otel.api.foam.ai, or the active OTEL_EXPORTER_OTLP_ENDPOINT override host — to THEIR client-side exclusions. In contrib-Python that is the PER-CLIENT vars (OTEL_PYTHON_REQUESTS_EXCLUDED_URLS, OTEL_PYTHON_URLLIB_EXCLUDED_URLS, OTEL_PYTHON_URLLIB3_EXCLUDED_URLS, OTEL_PYTHON_HTTPX_EXCLUDED_URLS, OTEL_PYTHON_AIOHTTP_CLIENT_EXCLUDED_URLS) — never the global OTEL_PYTHON_EXCLUDED_URLS, which their SERVER instrumentation also honors and which would drop their inbound spans when the collector is on localhost.
3. The echo filter, foam's structural backstop: if both layers are defeated, a client-kind span describing foam's own export call is withheld from FOAM'S copy only (their pipeline keeps it), with one loud warning naming this missing step. The loop is bounded even with the README step forgotten — but the step stays REQUIRED: the filter withholds foam's echo, it cannot stop their pipeline from paying for those spans.

What the tap does — and never does

• Additive reader only: their data, resource, and export are NEVER mutated. Foam's labels merge at serialization onto foam's copy; your pipeline sees your data byte-identical. The tap masks foam's copy only — your pipeline still carries what it captured.
• Identity: foam's copy carries THEIR resource identity (service.name, service.version, service.instance.id untouched) plus exactly four stamp keys: deployment.environment.name (the argument), foam.ingest.tier: external, telemetry.distro.name: foam, telemetry.distro.version.
• Masking: the always-on secrets floor plus your two extend-only lists run on everything before it leaves — attributes, event/link attributes, metric datapoint attributes, and (logs) the value-pattern pass over the free-text body. Fail-closed on foam's side: an item whose masked copy cannot be built is dropped from foam's batch with a loud counter, never exported raw.
• Never-throw: after construction, nothing in the tap can break your pipeline or your app — failures warn [foam] and foam's copy alone is lost.
• Self-ingest stand-down: wiring a tap onto foam's OWN door-1 pipeline is a wiring bug — the tap warns once and stands down (door 1 already exports that data; nothing is lost, nothing is doubled), and tears its own machinery down.
• Same-signal double-claim: if init() already exports a signal AND a tap sits on a separate foreign provider, both keep flowing (different pipelines; the tier marker distinguishes them) with one loud warning — if unintended, remove one.
• Transport: OTLP/HTTP protobuf to foam's fleet /v1/* endpoints with Authorization: Bearer <token> — same pins, same operator-override env var (honored + loudly warned) as door 1.
• Forbidden surface: no endpoint, cadence, sampling, temporality, or Views options — deliberately absent, same as init().
• Fork safety: taps built pre-fork self-heal in workers via the SDK's at-fork hooks (regression-pinned). Door-1's post-fork re-init does not govern the tap — door-2 identity is theirs, there is no foam instance id to re-mint.

Recipes

• Per-environment enable (the one-liner): enabled=os.environ.get("APP_ENV") not in ("test", "ci") — scenario 1. Other valid shapes: enabled=os.environ.get("APP_ENV") in ("production", "staging") (explicit allowlist), enabled=os.environ.get("APP_TELEMETRY") == "on" (your own app-named flag, compared EXPLICITLY — never bool(os.environ.get(...)), which is True for "false" and "0"), enabled=True (export everywhere, honestly labeled). Foam itself defines no on/off env flag — the only foam env key that exists is the token you wire yourself.

• Serverless flush: scenario 7 — try: … finally: flush(). The platform freezes the process before any batch timer fires; flush() is the fix. Import cost is lazy: importing foam_otel costs only the OTel API; the SDK and exporters load inside init() (and never with enabled=False).

• Uvicorn / ASGI graceful shutdown — wire shutdown() at lifespan shutdown. Uvicorn re-raises SIGTERM after its graceful stop, so the process dies BY SIGNAL and the SDK's atexit backstop never runs — telemetry buffered since the last batch timer is silently lost. The signal.signal(SIGTERM, ...) recipe above CANNOT be used under uvicorn (uvicorn owns the SIGTERM handler; replacing it disables graceful HTTP shutdown). The reliable path is the ASGI lifespan hook — proven by the conformance app under 60s batch delays:

# uvicorn/ASGI graceful shutdown: flush buffered telemetry before the process dies
from contextlib import asynccontextmanager
from fastapi import FastAPI
import foam_otel

@asynccontextmanager
async def lifespan(app):
    yield
    foam_otel.shutdown()   # flushes every buffered span/log/metric

app = FastAPI(lifespan=lifespan)

• Gunicorn / uvicorn / any pre-fork server — foam re-inits automatically. Foam registers a post-fork hook (os.register_at_fork), so under gunicorn --preload or a uvicorn worker fork each child re-inits automatically and re-mints a fresh service.instance.id — N workers report as N processes, not one. You do NOT need a manual post_fork hook or a shutdown()+init() dance. (A bare init() inside a hand-written post_fork would be a no-op — one init per process — but the automatic hook makes that unnecessary. On platforms without os.fork, the hook is skipped.) See GOTCHAS "Fork / pre-fork servers".

• gevent / eventlet workers: call monkey.patch_all() as line 1 of your entrypoint, BEFORE import foam_otel — patching after threading/ssl/socket are imported breaks context propagation and the export threads. See GOTCHAS.

• What foam deliberately does NOT automate: reading your git SHA (pass version yourself), choosing your PII fields (pass redact_pii_keys yourself), sampling (none exists), and named logger bridges beyond the generic stdlib bridge (structlog/loguru arrive via the update path — log() and the root-export bridge cover the mechanism today). LLM instrumentation: see the dedicated section below — since 1.1.0 foam ships ONE narrow gap-filler (the async OpenAI Responses API); every other LLM library is covered by installing its official instrumentor (auto-activated) or by tier-3 spans through the helpers/passthroughs.

LLM instrumentation — the OpenAI Responses gap-filler (since 1.1.0)

Foam ships ONE foam-authored LLM surface. When openai is importable and foam owns traces, init() wraps openai.resources.responses.responses. AsyncResponses.create (the async Responses API — what reasoning-model services call via await client.responses.create(...)), because the official opentelemetry-instrumentation-openai-v2 covers chat/embeddings but not this path. Know exactly what it does in your process:

• What it patches: AsyncResponses.create, via a wrapt wrapper. It stands DOWN (no wrap) when openai is absent, or when upstream (openai-v2 or anyone) already wrapped the method — foam never double-instruments.
• What it emits: ONE CLIENT span per logical call, named chat <model>, carrying gen_ai METADATA only — gen_ai.operation.name, gen_ai.provider.name: openai, gen_ai.request.model, gen_ai.response.model, gen_ai.response.id, and gen_ai.usage.input_tokens / output_tokens (current gen-ai semconv). On a failed call the span additionally carries the observed error.type and — when the provider error exposes one — http.response.status_code. A successful call carries NO status attribute (nothing is observed, so nothing is stated; versions ≤ 1.1.x emitted gen_ai.response.status and a fabricated http.status_code: 200 — see the migration notes below). Prompt and response content is NEVER captured — not even behind the upstream OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT gate, which is deliberately not honored. The HTTP transport span underneath is suppressed so one call yields one span.
• Turning it off: there is no env lever — OTEL_PYTHON_DISABLED_INSTRUMENTATIONS is inert (env table above; versions ≤ 1.1.x honored the name openai_responses there). Errors inside the wrapper never break your call (fail-open).
• Migration note (1.1.0 → 1.1.1): 1.1.0 briefly emitted gen_ai.system and — behind the upstream capture gate — prompt/instruction content. 1.1.1 aligned to the fleet contract: the provider attribute is gen_ai.provider.name (dashboards keyed on gen_ai.system from 1.1.0 must re-key), content capture was removed permanently (a privacy/contract fix), and http.status_code was added (itself corrected away in the next major — see the migration notes below). The attribute-key set above is frozen additively by the golden fixture — keys may be added within a major, never removed or renamed again.

Thrown-then-mapped exceptions (FastAPI/Starlette). Also since this line: an exception your @app.exception_handler maps to a clean 4xx is handled BELOW the OTel middleware, so the official instrumentation alone would export the request span with no exception event. Foam records the exception on the server span at handler-lookup time — a throw is a genuine fault signal even when mapped (contract §6). Control-flow exceptions (HTTPException — every 404 — and request-validation errors) are never recorded, and unmapped exceptions are left to the official instrumentation (no double recording). Like the Responses gap-filler, it has no env disable lever (versions ≤ 1.1.x honored the name mapped_exceptions in the now-inert OTEL_PYTHON_DISABLED_INSTRUMENTATIONS).

Supported versions

Surface	Supported	Reason
Python	>= 3.10	init() uses importlib.metadata.entry_points(group=...), whose keyword form landed in 3.10.
opentelemetry-api / opentelemetry-sdk	>= 1.44, < 2	The logs SDK's processor contract is on_emit(ReadWriteLogRecord) at this line (the older emit(LogData) shape is a different, incompatible signature), and foam's redaction rides that hook. API and SDK move in lockstep. The < 2 ceiling is deliberate (rule 40): a future OTel 2.0 is NOT supported until the matrix re-proves it — an open floor would let a breaking major resolve into your install as a silent no-op.
opentelemetry-exporter-otlp-proto-http	>= 1.44, < 2	The OTLP-over-HTTP transport foam exports on. Same deliberate ceiling as the API/SDK.
opentelemetry-instrumentation	>= 0.65b0, < 0.66b0	The entry-point activation base foam sweeps — pre-1.0, so capped to the PROVEN minor (rule 40: 0.x lines break between minors); widening is a deliberate, re-proven bump.
wrapt	>= 1.0, < 3	Direct dependency of the gap-fillers' wrap_function_wrapper (declared, never assumed transitively).
Instrumentations (opentelemetry-instrumentation-fastapi, -httpx, …)	via extras / installed	Auto-activated at init(), presence-checked. FastAPI is proven end-to-end by the conformance app.

The EXACT versions everything above was proven against (the rule-44 known-good set) are committed in constraints.txt; install with pip install -c constraints.txt foam-otel[...] to reproduce the proven environment byte-for-byte.

Foam builds on the OpenTelemetry project — the Python API/SDK, the OTLP exporters, and the contrib instrumentations — all Apache-2.0. Full credits in THIRD-PARTY-NOTICES.

MIGRATION — deliberate wire/surface corrections after 1.1.x (breaking; ships in the next MAJOR)

Three 1.1.x behaviors were corrected because they violated the wire contract (rule 26: the package never invents, renames, or fabricates a wire-visible name) or the closed env-var posture table. Each is a breaking change, shipped deliberately and gated by the golden fixture; check the following before upgrading.

1. OpenAI Responses gap-filler wire names (finding python-otelapi-4). The chat <model> gen_ai CLIENT span no longer carries gen_ai.response.status (an invented name — it exists nowhere in the gen-ai semantic conventions) or http.status_code (deprecated semconv, and on success it was a fabricated constant 200, never read from any response). Now, per the current GenAI span conventions (open-telemetry/semantic-conventions-genai, docs/gen-ai/gen-ai-spans.md): a successful call carries NO status attribute; a failed call carries error.type (the exception's qualified name, as the official openai-v2 instrumentation emits) plus the observed http.response.status_code when the provider error exposes one. Check your side: dashboards, alerts, or queries keyed on gen_ai.response.status or http.status_code on gen_ai spans will go empty — re-key error monitoring to error.type / span status ERROR (+ http.response.status_code where present); "success" is a span with status unset/OK, not http.status_code == 200.
2. OTEL_PYTHON_DISABLED_INSTRUMENTATIONS is now INERT (finding python-surface-6). 1.1.x honored it (skipping listed instrumentations, including the gap-filler names openai_responses / mapped_exceptions). The spec's honored env table is closed, and this lever silently dropped whole instrumentations' data — so it now warns loudly "set but ignored" and disables nothing. Check your side: any deployment setting this var will, after upgrade, emit telemetry from the instrumentations it used to suppress (more spans, possibly new span names/attributes on dashboards and in billing) and log one [foam] warning at boot. Remove the var, or uninstall the instrumentation package you need gone.
3. Helper telemetry scope renamed app.custom → foam-otel (finding python-surface-10). The instrumentation scope on every helper-emitted span, metric, and log is now the package identity (matching the JS core's @foam-ai/otel); app.custom was producer-classification stamp vocabulary the package must not ship (rule 4) — classification is foam's collector's job, server-side. Check your side: queries, dashboards, or collector/backend rules that filter on instrumentation scope app.custom (e.g. otel.library.name/otel.scope.name == "app.custom") must be re-pointed at foam-otel; scope-based routing in a customer-side collector should be updated the same way.

License

Proprietary — see LICENSE. Use is permitted only by Foam and customers with an active Foam agreement. Third-party components are credited in THIRD-PARTY-NOTICES (OpenTelemetry is Apache-2.0).