Shards-Kernel — Deterministic Control Planes for AI Systems

A kernel is a deterministic state machine that governs AI agent execution via jurisdiction, fail-closed arbitration, and an append-only hash-chained audit ledger.

Hard constraint: no tool execution without an explicit ALLOW; no ALLOW without a committed audit decision.

Problem (Operational, Not Philosophical)

Ungoverned agent systems fail in predictable ways:

Shards-Kernel exists to make those failure modes structurally expensive and forensically obvious.

Model

OPERATOR (authority) ── KernelRequest ──▶ KERNEL (policy+state+audit) ──▶ TOOL REGISTRY
       ▲                                           │
       └───────────── KernelReceipt ◀──────────────┘

AGENT has no direct tool access; it submits requests and receives results.

Separation is the point:

Core Invariants (Normative)

A compliant kernel MUST satisfy:

State Machine (Canonical)

BOOTING → IDLE → VALIDATING → ARBITRATING → AUDITING (intent)
        → EXECUTING → AUDITING (outcome) → COMPLETED → IDLE
Any unhandled exception ────────────────────────────────────────▶ HALTED

Public API (Supported Surface)

Treat everything else as internal. Your stable integration points:

from shards_kernel import (
    # Core types
    Kernel,              # Protocol / interface
    KernelRequest,       # Request structure
    KernelReceipt,       # Response structure
    Decision,            # ALLOW | DENY | HALT
    KernelConfig,        # Configuration
    
    # Variants
    StrictKernel,        # Maximum enforcement
    PermissiveKernel,    # Relaxed thresholds
    EvidenceFirstKernel, # Requires evidence field
    DualChannelKernel,   # Requires constraints dict
    
    # Tooling
    ToolRegistry,        # Tool registration + dispatch
    
    # Evidence
    replay_and_verify,   # External verification
    verify_evidence_bundle,
)

If a module is not reachable via shards_kernel.api, it is not a supported API surface.

Kernel Variants

Variants tune posture while preserving invariants:

Install

The PyPI distribution name is shards-kernel, the import namespace is shards_kernel, and the CLI command is shards-kernel:

python -m pip install shards-kernel  # after the approved PyPI release
python -c "import shards_kernel"

A verified PyPI release is pending. The different PyPI project kernels belongs to Hugging Face. Do not install kernels when intending to install Shards-Kernel. Until the approved release, install this repository from GitHub:

python -m pip install "git+https://github.com/Shards-foundation/Shards-kernel.git"

Or clone it and install the checkout in editable mode:

git clone https://github.com/Shards-foundation/Shards-kernel.git
cd Shards-kernel
python -m pip install -e .

Both approaches provide import shards_kernel and the shards-kernel command.

For existing users migrating from the old kernels Python import path, see the package migration guide.

Agent-ready local MCP integration (first release scope)

An agent can use this kernel today without a separate Shards MCP Gateway. The installable shards-kernel mcp-call command is a one-shot official MCP SDK v2 client for an enrolled agent; the existing mcp-relay command also supports stdio MCP hosts directly. A distinct signed v2 local operator, agent-scoped bearer and independently issued one-use permit are required.

# Python 3.11+; uses an existing signed local operator and issued permit.
python -m pip install -e ".[crypto,mcp]"
shards-kernel demo --json
shards-kernel mcp-call --help
# Following trusted operator/issuer provisioning, same-host agent:
shards-kernel mcp-call --port 8765 --tool echo \
  --arguments-json '{"text":"Hello from my agent"}' \
  --permit-file /absolute/private/path/to/issued-echo-permit.json

See the agent-first getting started guide for the operator trust prerequisites, secret handling, MCP-host integration, expected responses and reproducible acceptance. This is a usable local reference product, not a deployed remote MCP Gateway, a trading executor or production certification. The latter are separately gated.

Reproducible offline permit demonstration

Verify the complete local deny → approved execution → replay/expiry/revocation denial → verified audit restoration sequence using an ephemeral Ed25519 issuer:

python -m pip install -e ".[crypto]"
shards-kernel demo
shards-kernel demo --json

This demonstration executes only the built-in echo tool. It uses an isolated test clock and an in-memory evidence bundle; it is not a live gateway, trusted issuer deployment, or evidence of production readiness. See offline acceptance demo.

Separate offline permit issuer and human approver

shards-kernel issuer approve and shards-kernel issuer sign provide two offline-only, distinct Ed25519 authorities: a human-reviewed signature of an exact unsigned permit draft, followed by a separately trusted issuer signature. Private keys are provided only via inherited file descriptors, never the Shards-Kernel HTTP/MCP service.

This is a local reference workflow, not a KMS-backed production issuer. It limits signed capabilities to single-execution, short-lived built-in echo and add tools. See the offline issuer guide for exact commands, approval steps, threats and limitations.

Certificate-pinned HTTPS ingress for remote agent transport

The optional shards-kernel mtls-ingress --config <signed-ingress.json> provides a separate TLS 1.3 mutual-authentication boundary. It validates an offline Ed25519-signed ingress policy and two or more pinned client certificate fingerprints before forwarding only governed submissions to the existing loopback operator. Each request still requires its own enrolled agent bearer and separate signed execution permit. Administrative halt/evidence/status/policy routes are not forwarded.

This is a restricted HTTPS reference, NOT a native network MCP server, deployed gateway or production approval. See the mTLS agent ingress runbook.

Official MCP SDK v2 multi-agent local relay

shards-kernel mcp-relay --port <local-operator-port> exposes a single governed_execute tool over the official MCP v2 stdio transport. It bridges a separately enrolled agent's bearer into the signed-policy HTTP operator on loopback, which enforces actor identity, tool scope and a separate Ed25519 permit. Two independent MCP stdio clients can use distinct credentials without holding operator privileges.

Requires shards-kernel[mcp,crypto] on Python 3.11+ and one SHARDS_KERNEL_AGENT_TOKEN securely injected per local MCP client. No remote MCP/OIDC/mTLS or financial tools have been implemented. See two-agent MCP operator guide.

Two-agent identity boundary (local reference)

A signed operator config v2 can enroll 2–16 separate agents, each with a unique high-entropy bearer token and its own tool allowlist. The signed file stores only memory-hard scrypt token verifiers, not raw bearer secrets. A separate operator-held SHARDS_KERNEL_AGENT_AUTH_PEPPER supplies the MAC key outside the signed policy. POST /submit binds each authenticated bearer to the exact actor and its permitted tool; a separately signed Ed25519 permit is still mandatory. The distinct operator bearer alone can access halt, evidence, status and policy endpoints; it cannot execute as an agent.

This is loopback only and does not establish remote workload identity, OAuth/mTLS or distributed replay prevention. See the two-agent local identity guide.

Restricted local operator service

A standalone, loopback-only HTTP reference service is available with pinned public-only Ed25519 issuer keys, real UTC clock, strictly allowlisted built-in tools, one locally authenticated actor and SQLite audit/replay recovery:

python -m pip install -e ".[crypto]"
shards-kernel serve --config /absolute/path/to/signed-operator-policy.json

It will refuse startup without a valid Ed25519-signed operator policy, a separately pinned SHARDS_KERNEL_POLICY_PUBLIC_KEY_HEX, and SHARDS_KERNEL_API_KEY set to a protected strong bearer credential. It cannot run financial tools or bind remote interfaces. Unsigned policies require an explicit development-only --allow-unsigned-local flag. See the operator service guide for config fields, trust boundaries and unverified production requirements.

Operator readiness checks

The shards-kernel doctor CLI is a preflight inspection, not a production deployment certificate. By default it checks service prerequisites, reports every unverified runtime guarantee, and exits 2 when operational readiness is not demonstrated:

shards-kernel doctor
shards-kernel doctor --json
shards-kernel doctor --profile library --json

The library profile checks local package prerequisites and may pass with exit code 0; it does not imply the control plane is safely deployed. The service profile intentionally reports NOT_VERIFIED for issuer trust, durable audit/replay recovery and remote agent identity until independent runtime validation exists. API credentials are never printed.

See operator preflight guide.

HTTP server authentication

The HTTP server and Flask/FastAPI adapters require a configured bearer key. Without one, they refuse to start; audit exports, policy inspection, submit, status and halt are authenticated routes. The SDK's api_key argument now has a corresponding server-side check.

export SHARDS_KERNEL_API_KEY="$(python -c 'import secrets; print(secrets.token_urlsafe(48))')"
python -m shards_kernel.sdk.server
curl --fail http://127.0.0.1:8080/health
curl --fail -H "Authorization: Bearer ${SHARDS_KERNEL_API_KEY}" http://127.0.0.1:8080/status

This is a local reference HTTP interface, not a production-ready public multi-agent gateway. Do not publicly expose the built-in server. See HTTP authentication and deployment limitations.

Quickstart

git clone https://github.com/Shards-foundation/Shards-kernel.git
cd Shards-kernel
python -m pip install -e .
python -m unittest discover -s tests -v
bash scripts/smoke.sh
python examples/01_minimal_request.py
python -m shards_kernel --help

Runnable Demos

The repository includes deterministic, offline examples for request handling, fail-closed decisions, evidence verification, and durable SQLite audit recovery. Start with the demo guide.

python -m pip install -e ".[dev]"
python examples/01_minimal_request.py
python examples/02_verify_evidence.py
python examples/03_fail_closed_ambiguity.py
python examples/04_sqlite_persistent_ledger.py

The SQLite demo injects a SQLite-backed audit ledger into the kernel and checks that verified evidence is restored after restart. See the demo guide for the scope and operational limitations.

Organisation-wide repository review automation

Use the helper below to clone every repository in a GitHub organisation, run standard quality checks, and export machine-readable + markdown summaries.

export GITHUB_TOKEN=ghp_xxx   # token with access to your organisation repos
python scripts/org_repo_review.py --org Shards-foundation

Outputs:

• reports/org-review.json with full command output and exit codes.
• reports/org-review.md with per-repo pass/fail status.

Autonomous Codex contribution scaffold

Local scripts under scripts/agents/ support a closed-loop contribution workflow. .github/workflows/codex-pr-generator.yml and .github/workflows/codex-self-heal.yml are not in this repository. The workflow files that are present are:

• .github/workflows/ci.yml
• .github/workflows/dependency-canary.yml
• .github/workflows/dependency-health.yml
• .github/workflows/docs.yml
• .github/workflows/jekyll-gh-pages.yml
• .github/workflows/release.yml
• .github/workflows/security.yml
• .github/workflows/smoke.yml
• .github/workflows/static.yml

The local scripts are:

• scripts/agents/codex_generate_patch.py writes auditable run payloads to .codex/runs/.
• scripts/agents/build_codex_prompt.py injects kernel skill constraints from skills/ into an agent-ready prompt payload with failure-aware routing.
• scripts/agents/append_metrics.py attaches benchmark deltas to each run.
• scripts/agents/evaluate_run.py scores runs and updates .codex/meta/leaderboard.json and .codex/memory/*.json for active memory.
• scripts/agents/run_end_to_end.py executes the full loop in one command (run generation → metrics attach → prompt build → evaluation).

Quick local smoke test:

python scripts/agents/codex_generate_patch.py .github/ISSUE_TEMPLATE/feature_request.md
latest_run="$(ls -td .codex/runs/* | head -n1)"
python scripts/agents/append_metrics.py benchmarks/results.sample.json "$latest_run"
python scripts/agents/build_codex_prompt.py "$latest_run/request.json" --run-dir "$latest_run" --out "$latest_run/prompt.json"
python scripts/agents/evaluate_run.py "$latest_run"

# or run all stages:
python scripts/agents/run_end_to_end.py .github/ISSUE_TEMPLATE/feature_request.md --metrics benchmarks/results.sample.json

Specs + Docs

Specifications (normative):

Architecture + Threat Model:

What This Refuses To Be

It does one job: deterministic arbitration with immutable audit.

Versioning Policy

SemVer with strict interpretation:

Breaking change = any change to invariants, transitions, audit schema, or public types.

Contributing / Security

See CONTRIBUTING.md and SECURITY.md.

License

MIT