ContextDB Memory CI

Run a hosted Memory CI suite from any CI shell and fail the job when memory behavior regresses.

Memory CI is Hosted Alpha. This package is 0.1.0a1, the reusable GitHub Action tag is 0.1.0, and there is no availability SLA.

GitHub Actions (recommended)

- name: Run ContextDB Memory CI
  uses: atomsai/contextdb-clients@memory-ci-action-v0.1.0
  with:
    token: ${{ secrets.CONTEXTDB_EVAL_TOKEN }}
    project-id: ${{ vars.CONTEXTDB_PROJECT_ID }}
    suite-id: ${{ vars.CONTEXTDB_EVAL_SUITE_ID }}
    upload-artifacts: "true"

The Action pins this package version, writes a safe GitHub summary, and uploads content-free JSON and JUnit exports after every CLI result. See the Action guide and example workflow.

Raw CLI

Use the CLI directly in other CI systems or when you need a custom shell step:

pip install contextdb-memory-ci==0.1.0a1

contextdb-memory-ci run \
  --project-id "$CONTEXTDB_PROJECT_ID" \
  --suite-id "$CONTEXTDB_EVAL_SUITE_ID" \
  --json-output memory-ci.json \
  --junit-output memory-ci.xml

Set CONTEXTDB_EVAL_TOKEN to a cbe_ evaluation token before running the command. Keep the token in a CI secret store. The client sends it only in the Authorization header and never prints it.

The command starts one run, polls to a terminal status, then writes requested safe exports atomically. It exits:

• 0 for passed plus unchanged
• 1 for failed behavior or a regression
• 2 for operational errors, cancellation, timeout, or no baseline

A passed run with no baseline exits 2 by default. Use --allow-no-baseline only when that policy is intentional.

The public client accepts and displays aggregate status, opaque IDs, bounded machine failure codes, and content hashes. It rejects undeclared status fields. It does not receive suite names, queries, assertions, memory content, raw failures, or case details.

When to run Memory CI

• Before changing an embedding model, retrieval policy, or memory-formation prompt, run the pinned suite and block evidence or action-outcome changes.
• After adding a source backfill, verify that expected customer context appears without reviving forbidden stale evidence.
• Before releasing a voice or support agent, confirm the same memory cases still produce the expected act, ask, or abstain result.

Run and export retention can be bounded, and service failures exit 2 rather than being reported as behavioral regressions.

Frequently asked questions

Does the CLI receive customer memory or test queries?

No. Polling returns aggregate status and opaque IDs. JSON and JUnit exports are content-free. The CLI rejects response fields outside the published contract.

Which exit code should block a pull request?

Exit 1 means observed memory behavior failed or regressed. Exit 2 means the check could not make a behavioral determination, so fail the job and inspect the operational error separately.

What happens when a suite has no baseline?

A passing run exits 2 by default. Pin a successful baseline in ContextDB, or use --allow-no-baseline only for an intentional first-run workflow.

Is the evaluation token the same as a project API key?

No. cbe_ tokens are project-bound automation credentials limited to Memory CI run and export routes. Keep them in CI secret storage.

• Memory CI
• ContextDB documentation
• Public API contract

Apache-2.0. ContextDB trademarks are not licensed by Apache-2.0.