Dataoad

Open-source execution contracts and failure testing for side-effecting AI agent actions.

A timeout does not prove an action failed.

A successful tool call does not necessarily prove the intended external effect occurred.

Naive retry
calls=2
charges=2

Dataoad
execution=UNKNOWN
verification=CONFIRMED
final=CONFIRMED
calls=1
charges=1

Dataoad provides safe execution primitives for agents and other software that take real-world actions. If a charge, ticket, email, refund, or infrastructure change may have happened before its response was lost, blindly retrying can repeat the side effect.

Dataoad makes that uncertainty explicit:

CONFIRMED != FAILED != UNKNOWN

UNKNOWN is allowed to remain UNKNOWN. The default policy never automatically retries an ambiguous side-effecting action.

Dataoad is an early, local-first library and fault harness. It is not production-ready, does not make arbitrary third-party APIs idempotent, and does not guarantee exactly-once execution.

Install

Dataoad requires Python 3.11 or newer. From a repository checkout:

python -m venv .venv

Activate the environment with . .venv/bin/activate on POSIX or .venv\Scripts\Activate.ps1 in Windows PowerShell, then install the package:

python -m pip install -e .

For tests and contributor tools, install the development extra instead:

python -m pip install -e ".[dev]"

30-second example

This fake provider commits a charge and then raises TimeoutError, reproducing a lost response after mutation. Dataoad records UNKNOWN, makes no second charge, and then uses a read-only verifier to reconcile the receipt.

import asyncio

from dataoad import ActionRunner, ActionStatus, InMemoryLedger
from dataoad.testing import ChargeFault, FakePaymentProvider, PaymentVerifier


async def main() -> None:
    provider = FakePaymentProvider(charge_faults=[ChargeFault.TIMEOUT_AFTER_COMMIT])
    runner = ActionRunner(InMemoryLedger())
    request = {"customer_id": "customer_123", "amount_cents": 2_500}

    result = await runner.execute(
        "charge_customer",
        lambda: provider.charge(
            customer_id="customer_123",
            amount_cents=2_500,
            client_reference="order-123",
        ),
        request=request,
        idempotency_key="order-123",
        provider_reference="order-123",
        risk="financial",
    )

    assert result.status is ActionStatus.UNKNOWN
    assert provider.charge_calls == 1

    result = await runner.reconcile(result.receipt.action_id, PaymentVerifier(provider))
    assert result.status is ActionStatus.CONFIRMED
    assert provider.charge_calls == 1  # verification did not repeat the mutation


asyncio.run(main())

See examples/payment_timeout_after_commit.py for the complete naive-retry comparison and safety suite. The example is included in repository checkouts and source distributions; it is not installed as package data by the wheel.

For durable local receipts, give ActionRunner a file-backed SQLiteLedger instead. Reopening the same database preserves completed receipts and their idempotency claims across process restarts:

from pathlib import Path

from dataoad import ActionRunner, SQLiteLedger

runner = ActionRunner(SQLiteLedger(Path("dataoad.sqlite3")))

State semantics

The public outcome is an evidence statement, not merely a translation of an exception:

State	Exact meaning	Default mutation behavior
CONFIRMED	There is sufficient evidence that the intended side effect happened.	Do not retry.
FAILED	There is sufficient evidence that the intended side effect did not happen.	Retry only when policy and adapter evidence explicitly allow it.
UNKNOWN	There is not enough trustworthy evidence to decide whether it happened.	Never automatically retry.

An ordinary exception after the operation is invoked becomes UNKNOWN. This includes timeouts, lost connections, and malformed responses whenever mutation may already have occurred. DefinitiveFailure is different: raising it is a strong assertion by the provider adapter that the intended side effect did not happen. Use it only when that fact is known, not as a wrapper for a generic provider error.

A normal return is treated according to the evidence contract chosen by the caller:

• Without a verifier, Dataoad preserves the simple v0.1 behavior and trusts the operation response as authoritative evidence of occurrence. The receipt is CONFIRMED even if the returned Python value is falsey.
• With a verifier, a normal return establishes only that execution completed. Dataoad records direct external-effect evidence as UNKNOWN, always runs the verifier, and lets its bound evidence determine CONFIRMED, FAILED, or UNKNOWN.

Receipts preserve three related fields:

• execution_status describes what direct execution established.
• verification_status describes what independent read-back established.
• final_status is the current public confidence statement returned as ActionResult.status.

A timeout-after-commit can therefore have execution_status=UNKNOWN, verification_status=CONFIRMED, and final_status=CONFIRMED. PENDING and COMPLETED are separate internal claim-lifecycle states; PENDING is not a fourth public outcome.

CONFIRMED means that sufficient evidence shows the intended effect happened at least once. It does not prove that the effect happened exactly once or that no duplicate exists.

Evidence is also aggregated across attempts. Once any attempt may have produced an unverified side effect, a later independent FAILED attempt cannot erase that uncertainty: the action remains UNKNOWN. A later authoritative confirmation may establish CONFIRMED; an action-wide final-state verifier may also resolve all relevant attempts explicitly.

ActionResult.value cannot be read for FAILED or UNKNOWN results. It raises UnresolvedActionError, so ambiguous work cannot look like normal success.

Retry policy

The default RetryPolicy performs one attempt and retries neither FAILED nor UNKNOWN. A retry of FAILED requires remaining capacity under max_attempts, retry_on_failed=True, and trustworthy evidence that no side effect occurred. That evidence is either:

• a DefinitiveFailure(..., retryable=True) from the adapter; or
• VerificationResult.absent(..., retry_safe=True) from a verifier that establishes healthy, conclusive absence and proves the original request is terminal or fenced so it cannot commit later.

VerificationResult.absent(...) defaults to retry_safe=False. A healthy read-back that shows nothing now remains UNKNOWN while the original request could still be in flight. Set retry_safe=True only when the adapter can prove terminal or fenced non-occurrence; that stronger result becomes FAILED and may authorize retry when policy also allows it.

retry_on_unknown=True is available only as an explicit escape hatch and emits a runtime warning. It can duplicate real-world side effects unless the downstream operation is independently safe.

Direct calls to RetryPolicy.should_retry(...) validate their runtime inputs before making any decision: status must be an exact ActionStatus, attempts an exact positive integer, and failure_is_retryable an exact boolean. Invalid values raise instead of falling through to a retryable branch.

The execution protocol owns this decision. An LLM or agent should not be asked to guess whether an ambiguous mutation is safe to repeat.

Idempotency semantics

An idempotency claim is scoped by (action_name, idempotency_key). request must include every input that can affect the intended side effect. Dataoad stores its canonical SHA-256 hash rather than the raw request.

The request must be an exact native Python dict representing a strict JSON object. Values may contain only None, exact bool, exact int, finite exact float, exact str, lists, and dictionaries with string keys. Python-specific representations such as tuples, enums, dataclasses, dates, datetimes, UUIDs, custom containers, and scalar subclasses are rejected instead of being silently coerced. Convert them explicitly at the call site when that conversion matches the application's intended semantics.

For one shared ledger:

1. Same action, key, and request hash while the first claim is PENDING raises ActionInProgressError; the duplicate operation is not invoked.
2. Same action, key, and request hash after completion replays the existing receipt with result.replayed=True; the operation is not invoked again. If that receipt is UNKNOWN and a verifier is supplied, Dataoad attempts read-only reconciliation instead.
3. Same action and key with a different request hash raises IdempotencyConflictError before the new operation can run.
4. Concurrent contenders are reduced to one winning claim by the ledger's atomic conditional write.
5. The same key under a different action name is a different claim.

provider_reference is a separate, caller-supplied identity used to correlate the mutation with later provider lookup. It is not inferred from idempotency_key, and it is not the provider-generated provider_request_id returned after a successful request. Supply the exact external lookup reference before mutation whenever the receipt may be verified. Reusing a completed claim with a different provider reference is an idempotency conflict.

Provider return values are intentionally not persisted. An immediate confirmed result can expose its value, but a replay from a durable receipt may raise ResultValueUnavailableError when .value is accessed. Persist or read back business data in the system of record instead of relying on the ledger as a response store.

Caller metadata must be JSON-compatible. The top-level keys attempt_history, attempt_records, provider, provider_request_id_conflict, provider_request_ids, recovery, verification, verification_conflict, and verification_observations are reserved for Dataoad's receipt protocol and are rejected before a claim is created.

Local claim deduplication is not downstream idempotency

Dataoad's local claim deduplication prevents cooperating callers that share the same ledger and key contract from deliberately starting a second operation. Downstream idempotency is a property of the external provider, such as a provider-enforced idempotency key or conditional write.

There is still a distributed-systems gap between committing an external side effect and durably recording its outcome. If a ledger transition fails after invocation, Dataoad raises ActionPersistenceError with the action receipt and the in-process status/evidence it observed. The idempotency claim remains a safety barrier, but its exact durable state may be PENDING or already COMPLETED if the write committed before its response was lost. Inspect the ledger; never treat the error as permission to repeat the mutation. Use read-only recovery as described below when the receipt remains PENDING. Dataoad cannot turn a non-idempotent remote API into an exactly-once transaction. Use provider idempotency when available, in addition to Dataoad's local claim.

Before invoking the operation, Dataoad durably increments the attempt count. If that start_attempt() write succeeds but its acknowledgement is lost, the runner reads the claim back and continues only when the receipt proves exactly the single conditional transition it requested. It does not increment again. If that exact transition cannot be demonstrated, the operation is not invoked and ActionPersistenceError preserves the claim as a safety barrier.

Verification semantics

A verifier is a provider-specific object or callable that receives an ActionReceipt and performs read-only observation. It returns a VerificationResult:

• VerificationResult.confirmed(...) reports positive evidence of occurrence.
• VerificationResult.absent(...) records a healthy absence observation but remains UNKNOWN by default. Passing retry_safe=True asserts the stronger terminal/fenced non-occurrence needed for FAILED.
• VerificationResult.unknown(...) reports missing or untrustworthy evidence.

A negative result is accepted as FAILED only when its health is HEALTHY, conclusive_absence=True, and retry_safe=True. Other negative claims are normalized to UNKNOWN. Verifier exceptions, invalid verifier responses, stale or partial reads, authentication failures, broken pagination, and failed positive controls must also remain UNKNOWN; "could not verify" is not "did not happen."

Mutation and verification must share the receipt's durable provider_reference. A decisive verifier result must echo that exact identity as verified_provider_reference; missing or mismatched binding is downgraded to UNKNOWN. Passing a verifier directly to execute(...) therefore requires provider_reference. If verification will happen only through a later reconcile(...) or recover_pending(...) call, still set the reference on the original execute(...) so it is present in the durable receipt.

Pass a verifier to execute(...) to check immediately after ambiguous execution, or call runner.reconcile(action_id, verifier) later for a completed UNKNOWN receipt. Verifier correctness is application-specific: Dataoad enforces the result contract, but cannot prove that custom evidence or a remote system of record is truthful.

The bundled fake payment verifier sets retry_safe=True only after checking a fake-provider in-flight fence as well as lookup health and absence. A production adapter needs an equivalent provider guarantee; a healthy empty snapshot alone is not enough.

Concurrent verifier observations are merged instead of using first-writer-wins. Contradictory CONFIRMED/FAILED evidence makes the durable result UNKNOWN and preserves both observations. Two confirmations with different provider request IDs remain CONFIRMED (occurrence is still known), but the receipt records provider_request_id_conflict=True and all observed IDs because they may indicate duplicate effects. Dataoad withholds a recovered or direct value when it cannot correlate that value to the retained provider ID. For verifier-recovered values, two missing IDs are never considered a match: a non-null provider_request_id must be present in both the verification and the retained receipt. This keeps occurrence CONFIRMED while refusing an uncorrelated optional value.

Stranded claim recovery

Recovery is explicit and read-only; Dataoad never leases or steals an in-flight mutation automatically.

• If an ActionPersistenceError leaves a PENDING receipt with attempts >= 1, first ensure the original worker will not issue another mutation, then call await runner.recover_pending(action_id, verifier). A negative recovery can become FAILED only when the verifier returns absent(..., retry_safe=True) and therefore proves no late commit is possible. Other absence remains UNKNOWN.
• If a crash occurred after claim creation but before the durable attempt count was incremented, the receipt has attempts == 0. An operator may call runner.release_unstarted(action_id). This conditionally removes only that uninvoked claim so the same idempotency key can be claimed again. It races atomically with attempt start: if invocation has begun, release is refused.

An ActionPersistenceError.receipt is the last safely observed receipt, not a promise that a failed write did not commit. Re-read the ledger before choosing between these recovery paths.

Concurrent recovery observations use the same conservative conflict merge as normal reconciliation. ActionPersistenceError also exposes observed_status, observed_metadata, and, when an immediate confirmed value was returned, value_available/observed_value. These are diagnostic evidence, not proof that the corresponding ledger transition committed.

Fault harness

From a repository checkout or an unpacked source distribution, run the bundled deterministic payment suite:

dataoad test examples/payment_timeout_after_commit.py

The CLI loads a Python file whose zero-argument run_safety_suite() function returns a dataoad.testing.SafetyReport. The bundled suite covers success, explicit pre-commit failure, replay, concurrency, key/body conflicts, faults on both sides of commit, verifier outage, an unhealthy false-negative, a verifier using the wrong external identity, and preservation of earlier UNKNOWN evidence across later attempts.

Representative output:

Timeout-after-commit comparison
  Naive retry: calls=2, charges=2, total=5000 cents
  Dataoad:    execution=UNKNOWN, verification=CONFIRMED, final=CONFIRMED, calls=1, charges=1

Action: charge_customer

Normal success                     PASS
Explicit pre-side-effect failure   PASS
Replay safety                      PASS
Concurrent duplicate safety        PASS
Same-key/body conflict             PASS
Timeout-before-commit recovery     PASS
Timeout-after-commit recovery      PASS
Unknown auto-retry protection      PASS
Verifier unavailable handling      PASS
Unhealthy false-negative handling  PASS
wrong-verification-identity        PASS
unknown-evidence-erasure           PASS

Final classification:
SAFE UNDER TESTED CONDITIONS

"Safe under tested conditions" describes only these deterministic scenarios. It is not formal verification or a universal safety claim. Each harness SafetyCheck.passed value must be an exact boolean; truthy values such as the string "false" are rejected rather than rendered as a pass. SafetyReport also rejects duck-typed or otherwise invalid check entries, so the constructor cannot bypass that invariant.

Architecture

caller / agent
      |
      v
 ActionRunner ---- atomic claim and receipt transitions ----> Ledger
      |                                                   InMemory / SQLite
      |
      +---- mutation operation ----> external system
      |
      +---- read-only Verifier ----> system of record

• ActionRunner claims before invocation, classifies evidence, applies retry policy, and reconciles ambiguity.
• ActionReceipt is an immutable snapshot with an optimistic version. It keeps hashes, identifiers, bounded errors, status, and explicit JSON metadata, not raw request bodies or arbitrary provider responses. Structured per-attempt records preserve execution, verification, and retry decisions.
• InMemoryLedger provides thread-safe, process-local atomic claims for tests and ephemeral runs.
• SQLiteLedger provides durable local receipts and cross-process claim uniqueness using SQLite transactions and constraints.
• Verifier separates mutation execution from provider-specific read-back.
• dataoad.testing supplies deterministic fault injection and qualified safety reports; it is not a simulator of every provider failure mode.

The core has no dependency on an agent framework or LLM provider.

Limitations and honest guarantees

Dataoad v0.1 is alpha software intended for inspection, testing, and local integration work. In particular:

• It provides no exactly-once or production-ready guarantee.
• Correct deduplication requires all contenders to share a ledger and use a stable action name, idempotency key, and complete semantic request.
• InMemoryLedger loses state on exit. SQLiteLedger is local persistence, not a hosted, replicated, highly available coordination service.
• A hard process stop can leave a PENDING claim requiring explicit application or operator recovery. v0.1 provides conditional recovery primitives but has no leases, ownership service, fencing tokens, or stale-claim worker.
• The external mutation and ledger update cannot generally be one atomic transaction. Provider-side idempotency remains strongly recommended.
• Reconciliation is only as sound and fresh as the custom verifier and provider read path.
• The request hash avoids retaining a raw payload but is not encryption or an anonymization guarantee. Idempotency keys and explicit metadata are stored; do not put secrets in them. Bounded exception messages are also retained for diagnosis, so provider adapters should not include secrets in exception text.
• Arbitrary provider values are not durable, and there are no built-in provider integrations, queues, dashboards, authentication, or multi-tenancy.

Roadmap

Likely next steps, subject to evidence from real integrations:

1. Extend explicit PENDING recovery with durable ownership, leases, and provider-aware fencing semantics.
2. Add adapter guidance and contract tests for provider idempotency and robust verification.
3. Add machine-readable safety reports and broader crash/concurrency fault cases.
4. Evaluate another transactional ledger backend when multi-host coordination is justified.
5. Add thin framework adapters only after the core protocol is stable.

Cloud services, a dashboard, policy languages, and broad "agent platform" features are deliberately outside the v0.1 scope.

Contributing

See CONTRIBUTING.md. Contributions should preserve the central invariant: ambiguous side effects remain UNKNOWN unless trustworthy evidence proves otherwise.

License

Licensed under the Apache License 2.0. See LICENSE.