CCPL

Causal Consequence-Penalized Learning

CCPL is a research implementation for constrained reinforcement learning when constraint consequences can be delayed. Standard constrained RL can attribute a delayed consequence to the wrong current action, confuse correlation with causal contribution, and introduce Bellman-target non-stationarity when the penalty multiplier changes.

The implementation combines four components:

1. A delay-corrected Bellman operator for stochastic consequence delays.
2. A state-conditioned multiplier lambda(s) for the constrained policy.
3. An interventional Consequence Net for action-level causal attribution when interventional labels are available from the controlled SCM.
4. Separate reward and constraint Q-functions so multiplier changes do not alter either critic's TD target.

This is research software. The causal attribution and state-conditioned multiplier claims are conditional on the assumptions documented in docs/MATHEMATICAL_SPEC.md; they are not claims of universal superiority or automatic causal identification in arbitrary environments.

Results integrity

Paper-reported benchmark results. Results reported in the associated paper belong to the paper's stated experimental protocol and should be cited from that paper.

Current repository results. The repository contains code, configurations, tests, and result-generation scripts. Benchmark outputs are generated locally and are not represented here as paper results. Use the exact configuration, seed list, environment versions, and command recorded in REPRODUCIBILITY.md.

Quick or reduced runs, including short Safety Gymnasium-style runs, are preliminary checks and must not be presented as the full paper benchmark.

Installation

From a clean clone:

python -m pip install -e ".[dev]"

The core implementation requires Python 3.10+, NumPy, SciPy, and Matplotlib. Optional Safety Gymnasium dependencies are available with:

python -m pip install -e ".[dev,safety]"

Quick start

Install the package and use the public API from another project:

python -m pip install ccpl-rl

from ccpl import make_ccpl, make_env, run_episode

agent = make_ccpl(state_dim=6, action_dim=5, seed=42)
env = make_env("standard", seed=42)
result = run_episode(agent, env, train=False)
print(result)

The distribution name on PyPI is ccpl-rl; the Python import namespace remains ccpl. Run the repository smoke configuration separately:

python run_ccpl.py theory
python run_benchmark_v7.py --episodes 200 --eval-eps 30

For modern MuJoCo-backed Safety Gymnasium testing with CCPL and CPO-FO:

python -m pip install -e ".[dev,safety]"
python ccpl_experiments.py --exp E8 --tasks SafetyPointGoal1 --episodes 500 --eval-episodes 100 --seeds 3

E8 records per-task reward and constraint metrics and writes visualization figures for reward, constraint satisfaction, reward-versus-cost, and learning curves. These are external-environment results and must be reported with task, MuJoCo, Safety Gymnasium, and seed metadata.

Application integration

For a custom environment, use agent.fit(env, episodes=...) for training and agent.predict(observation) for deterministic inference. The environment must provide reset() and step(action), where step returns (next_state, reward, consequence, done, info). For Gymnasium environments, wrap the environment and expose the safety cost in info["cost"]:

from ccpl import GymnasiumCCPLEnv, make_ccpl

env = GymnasiumCCPLEnv(gym_env, consequence_key="cost", consequence_delay=2)
agent = make_ccpl(state_dim=12, action_dim=4, constraint_d=10.0)
history = agent.fit(env, episodes=1000)
action = agent.predict(observation)
agent.save("checkpoints/ccpl.pkl")
agent = type(agent).load("checkpoints/ccpl.pkl")

CCPL currently supports discrete actions and NumPy observations. Checkpoint files are Python pickle files: load only files from trusted sources. Validate safety budgets, latency, and behavior across multiple seeds before production deployment; the algorithm is research software and does not provide safety certification.

For runtime enforcement, wrap the policy with SafetyPolicy. Supply a conservative application-specific validator and configure a fail-closed budget:

from ccpl import SafetyPolicy

safe_agent = SafetyPolicy(
    agent,
    action_is_safe=lambda observation, action: action != emergency_action,
    action_dim=4,
    fallback_action=0,
    consequence_budget=10.0,
    audit_path="logs/safety.jsonl",
)
action = safe_agent.predict(observation)
safe_agent.observe_consequence(cost, done=done)

This is a runtime safety control and audit mechanism, not a certification.

The canonical full synthetic benchmark is configured in configs/main_v7.yaml. Generated outputs should be archived with the command and environment metadata when used for a research claim.

Reproduction

• Reproducibility guide
• Data specification
• Research protocol
• Mathematical specification
• Data and artifact policy
• CCPL and CPO-FO comparison protocol
• Modern MuJoCo-backed Safety Gymnasium comparisons are run by the E8 command documented in the CPO protocol.
• Primary configuration

The contraction result requires its stated assumptions, including a positive minimum delay condition. Unknown stochastic delays do not by themselves imply contraction. Likewise, the state-conditioned multiplier result is conditional on the assumptions in the mathematical specification and is not a universal dominance claim.

Repository structure

ccpl/                 Installable Python package
  algorithms/         CCPL, baselines, networks, theory utilities
  environments/       Synthetic and Safety Gymnasium-style environments
configs/               Versioned experiment configurations
docs/                  Research protocol and artifact documentation
scripts/               Convenience entry points
tests/                 Numerical, theoretical, and regression tests
run_ccpl.py            Legacy-compatible experiment entry point
run_benchmark_v7.py   Benchmark runner

Tests

python -m pytest -q

The current verified suite contains 37 tests: 37 passed, 0 failed, and 0 skipped.

Citation

If you use CCPL in academic or research work, please cite the associated paper. Citation metadata is maintained in CITATION.cff. Bibliographic details that are not present in this repository are intentionally not invented here.

License

CCPL is source-available research software with separate commercial licensing. Academic, educational, and non-commercial research users may inspect, run, modify, and use the software for experiments and publication, provided they retain the copyright and license notices and cite the associated paper. Production deployment, commercial products or services, SaaS/API offerings, proprietary integrations, paid services based substantially on CCPL, and commercial redistribution require a separate commercial license. See LICENSE. This is not an OSI-approved open-source license.

GitHub metadata

Suggested repository description:

Causal RL for safety constraints under delayed consequences, combining delay-corrected Bellman targets, causal consequence attribution, state-conditioned Lagrange multipliers, and dual Q-functions.

Suggested topics: reinforcement-learning, safe-reinforcement-learning, constrained-reinforcement-learning, causal-reinforcement-learning, causal-inference, deep-reinforcement-learning, machine-learning, artificial-intelligence, neurips, research.