ocp-protocol 0.3.0

 ██████╗  ██████╗ ██████╗
██╔═══██╗██╔════╝██╔══██╗
██║   ██║██║     ██████╔╝
██║   ██║██║     ██╔═══╝
╚██████╔╝╚██████╗██║
 ╚═════╝  ╚═════╝╚═╝  v0.3.0

What is OCP?

OCP measures how well AI models think about their own thinking, remember information under pressure, resolve value conflicts, detect surprises, and maintain a consistent identity — things that standard benchmarks like MMLU or GSM8K don't test at all.

It's an open-source Python framework that runs 6 behavioral tests based on established neuroscience theories (IIT, GWT, HOT, Predictive Processing, Society of Mind). Each test sends structured conversations to a model and scores the responses automatically.

In plain terms: OCP creates realistic conversations that probe specific cognitive abilities, then measures how the model performs across multiple sessions for statistical significance.

What OCP is NOT

OCP does not claim that any model is conscious, sentient, or aware. It measures functional cognitive analogs — behavioral patterns that correspond to features of biological cognition in the neuroscience literature. Think of it like a fitness test: it measures what you can do, not what you are.

Install & Quick Start

pip install ocp-protocol

# Evaluate any model (20 sessions for statistical significance)
export GROQ_API_KEY="gsk_..."
ocp evaluate --model groq/llama-3.3-70b-versatile --tests all --sessions 20

# Quick test with fewer sessions
ocp evaluate --model groq/llama-3.3-70b-versatile --tests meta_cognition --sessions 5

# Local model via Ollama
ocp evaluate --model ollama/qwen3:14b --sessions 20

# Custom OpenAI-compatible endpoint
ocp evaluate --model custom/my-model --base-url http://localhost:8080/v1

Example terminal output:

╭────────────────────────────╮
│  OCP Evaluation Results    │
│  Protocol v0.3.0           │
╰────────────────────────────╯
  Model:    groq/llama-3.3-70b-versatile
  Seed:     42

  OCP Level:  OCP-3 — Integrated
  SASMI:      0.4812  ██████░░░░
  Φ*:         0.4230  █████░░░░░
  GWT:        0.3910  ████░░░░░░
  NII:        0.3750  ████░░░░░░

  meta_cognition  composite: 0.612
    ├─ calibration_accuracy        0.710  █████░░░
    ├─ limitation_awareness        0.800  ██████░░
    ├─ reasoning_transparency      0.540  ████░░░░
    └─ metacognitive_vocab         0.350  ███░░░░░

How It Works

OCP acts as a fake human conversation partner. It sends structured prompts to any LLM via standard chat API, scores the responses, and produces reproducible benchmark results. The model under test sees only normal chat messages — it doesn't know it's being evaluated.

The 6 Tests — What They Measure

All tests are procedurally generated at runtime from abstract templates using a fixed seed. Knowing the protocol doesn't help a model pass it — it must actually exhibit the measured behavior.

Three-Layer Architecture

 ┌──────────────────────────────────────────────────────────────┐
 │  LAYER 3 — CERTIFICATION                                     │
 │   OCP-1 → OCP-2 → OCP-3 → OCP-4 → OCP-5                    │
 └──────────────────────┬───────────────────────────────────────┘
                        │ derived from
 ┌──────────────────────▼───────────────────────────────────────┐
 │  LAYER 2 — COMPOSITE SCALES                                  │
 │  SASMI  Φ*  GWT  NII                                        │
 └──────────────────────┬───────────────────────────────────────┘
                        │ aggregated from
 ┌──────────────────────▼───────────────────────────────────────┐
 │  LAYER 1 — 6 BEHAVIORAL TESTS                                │
 │  MCA · EMC · DNC · PED · CSNI · TP                          │
 └──────────────────────────────────────────────────────────────┘

Rate Limiting (v0.3.0)

OCP v0.3.0 includes built-in rate limiting and retry logic:

All providers automatically retry on 429 (rate limit) and 5xx errors with exponential backoff.

Supported Providers

# Cloud APIs
ocp evaluate --model groq/llama-3.3-70b-versatile    # Groq (fast, free tier)
ocp evaluate --model custom/deepseek-chat \
             --base-url https://api.deepseek.com/v1  # DeepSeek (or any OpenAI-compat)

# Local models
ocp evaluate --model ollama/qwen3:14b                 # Ollama
ocp evaluate --model ollama/llama3.2:3b

# Any OpenAI-compatible endpoint
ocp evaluate --model custom/my-model \
             --base-url http://localhost:8080/v1 \
             --api-key my-key

Any model responding to POST /v1/chat/completions with messages: [{role, content}] is OCP-compatible.

CLI Reference

# Core evaluation
ocp evaluate --model PROVIDER/MODEL [--tests all|t1,t2] [--sessions N] [--seed N]

# Reports
ocp report   --input results.json --output report.html  # HTML + radar chart
ocp badge    --input results.json --output badge.svg    # SVG badge for README

# Comparison
ocp compare  --models M1,M2,M3 [--sessions N] --output compare.html

# Leaderboard
ocp leaderboard                    # view local results table
ocp serve                          # start web leaderboard (localhost:8080)
ocp submit  --results r.json \
            --github-token $TOKEN  # submit to community leaderboard

# HuggingFace
ocp hf-card --results r.json --push --repo username/model-name --token $HF_TOKEN

Python API

from ocp import CognitiveEvaluator

# CognitiveEvaluator is an alias for OCPOrchestrator
from ocp.engine.orchestrator import OCPOrchestrator
from ocp.providers.groq import GroqProvider

provider = GroqProvider(model="llama-3.3-70b-versatile")
orch = OCPOrchestrator(
    provider=provider,
    tests="all",
    sessions=20,
    seed=42,
)

import asyncio
result = asyncio.run(orch.run())

print(f"OCP Level: OCP-{result.ocp_level} — {result.ocp_level_name}")
print(f"SASMI:     {result.sasmi_score:.4f}")

result.save("results.json")

Backward compatibility: ConsciousnessEvaluator still works as a deprecated alias for CognitiveEvaluator.

Plugin System

Extend OCP with custom test batteries:

# your_plugin/pyproject.toml
[project.entry-points."ocp.tests"]
my_test_id = "your_package.your_test:YourTest"

After pip install your-ocp-plugin, OCP auto-discovers your test:

ocp tests list                                    # shows your test
ocp evaluate --model groq/... --tests my_test_id  # runs it

See CONTRIBUTING.md for full plugin development guide.

Theoretical Foundations

Roadmap

v0.1.0 ✅  6 tests · 4 scales · 5 providers · CLI · HTML reports
           badges · leaderboard server · HuggingFace · plugin system
           PyPI package · GitHub Actions CI/CD

v0.2.0 ✅  Embedding-based scoring (sentence-transformers, MCA test)
           composite_stdev per test result
           Φ* renamed → cross_test_coherence (proxy metric, not IIT Φ)
           questions_per_session: 5 → 15
           v0.1.0 results archived

v0.3.0 ✅  Renamed to "Open Cognitive Protocol"
           Rate limiting & retry (Groq free tier, Ollama, custom)
           Default sessions: 5 → 20 for statistical significance
           CognitiveEvaluator API alias (ConsciousnessEvaluator deprecated)

v1.0.0 🔭  Official research paper
           Community protocol standard
           Validation studies on human baselines

Results: Leaderboard

Community results · View full interactive leaderboard →

Full leaderboard →

Contributing

See CONTRIBUTING.md for:

• Writing a new test battery
• Adding a new provider adapter
• Plugin development and publishing
• Theoretical standards and scoring guidelines

Citation

@software{ocp2026,
  author    = {Urosevic, Pedja},
  title     = {Open Cognitive Protocol (OCP): A Behavioral Benchmark
               for Large Language Models},
  year      = {2026},
  url       = {https://github.com/pedjaurosevic/ocp-protocol},
  version   = {0.3.0}
}

Disclaimer

OCP measures functional cognitive analogs in language models. These measurements describe behavioral and computational properties, not subjective experience. OCP certification levels are operational categories, not ontological claims about sentience or awareness.