Skip to main content

mdplib

PyPI Python

A Python library of algorithms for solving infinite-horizon, discounted Markov Decision Processes (MDPs), with a performance-critical core written in C++ (via pybind11).

Implements:

  • Value Iteration (Jacobi and Gauss-Seidel variants)
  • Policy Iteration
  • Q-Learning and SARSA (model-free reinforcement learning)

Installation

pip install mdplib

Prebuilt wheels are published for Linux, Windows and macOS (Python 3.10-3.14), so this does not require a C++ compiler on your machine. If your platform/Python version has no matching wheel, pip falls back to building from source, which does require a C++17 compiler (the extension is compiled at install time):

git clone https://github.com/LuccaGiannelli/mdplib.git
cd mdplib
pip install .

Basic usage

from mdp import MDP

S = [0, 1, 2]   # states
A = [0, 1, 2]   # actions

def Pt(sn, a, sn1):
    # returns P(sn1 | sn, a)
    ...

def rt(sn, a, sn1):
    # returns the reward
    ...

problem = MDP(S, A, Pt, rt)

# Value Iteration
problem.VI(gamma=0.9, tol=1e-6)
problem.print_results()

# Policy Iteration
problem.PI(gamma=0.9, tol=1e-6)
problem.print_results()

See examples/basic_example.py for a complete example (a machine maintenance problem).

Available algorithms

Method Description
VI Value Iteration (Jacobi)
VIGS Value Iteration with Gauss-Seidel
PI Policy Iteration
QLearning Q-Learning (off-policy TD control)
SARSA SARSA (on-policy TD control)

Common parameters

Parameter Description
gamma Discount factor (0 ≤ γ < 1)
tol Convergence tolerance
max_iter Maximum number of iterations
v0 Initial value vector (optional)
verbose Print progress on completion
norm_type 0 = relative sup norm, 1 = sup norm, 2 = euclidean norm
epsilon Final epsilon for the epsilon-greedy policy (Q-Learning / SARSA)
alpha Learning rate (Q-Learning / SARSA)

How it works

You define the state space S, action space A, a transition probability function Pt(sn, a, sn1), and a reward function rt(sn, a, sn1). The constructor enumerates all feasible transitions into a CSV (sn,a,sn1,prob,reward) and loads it into a NumPy array; the algorithms then hand that array to the C++ core, which does the inner loops with direct memory access for performance.

After solving, results are available as:

Attribute Description
value optimal value for each state
policy optimal action for each state
time execution time (seconds)
n_iterations number of iterations to convergence

Adding a new algorithm

  1. Create mdp/algorithms/new_algo.py with a function def NewAlgo(self, ...):
  2. Import it in mdp/algorithms/__init__.py
  3. Attach it to the class in mdp/core.py: NewAlgo = NewAlgo

Tests

pip install pytest
pytest tests/

License

MIT

Metadata

Release files for mdplib 1.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mdplib 1.0.2
File Size Uploaded
mdplib-1.0.2.tar.gz 13.0 kB Details

Built distributions (wheels)

Table of built distributions (wheels) for mdplib 1.0.2
File
mdplib-1.0.2-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details
mdplib-1.0.2-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.14 CPython 3.14 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.2-cp314-cp314-macosx_14_0_arm64.whl CPython 3.14 CPython 3.14 macOS 14.0+ ARM64 Details
mdplib-1.0.2-cp313-cp313-win_amd64.whl CPython 3.13 CPython 3.13 Windows x86-64 Details
mdplib-1.0.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.2-cp313-cp313-macosx_14_0_arm64.whl CPython 3.13 CPython 3.13 macOS 14.0+ ARM64 Details
mdplib-1.0.2-cp312-cp312-win_amd64.whl CPython 3.12 CPython 3.12 Windows x86-64 Details
mdplib-1.0.2-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 Details
mdplib-1.0.2-cp312-cp312-macosx_14_0_arm64.whl CPython 3.12 CPython 3.12 macOS 14.0+ ARM64 Details
mdplib-1.0.2-cp311-cp311-win_amd64.whl CPython 3.11 CPython 3.11 Windows x86-64 Details
mdplib-1.0.2-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.2-cp311-cp311-macosx_14_0_arm64.whl CPython 3.11 CPython 3.11 macOS 14.0+ ARM64 Details
mdplib-1.0.2-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details
mdplib-1.0.2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.2-cp310-cp310-macosx_14_0_arm64.whl CPython 3.10 CPython 3.10 macOS 14.0+ ARM64 Details

Total release size: 9.4 MB

Release files / mdplib-1.0.2.tar.gz

Download URL mdplib-1.0.2.tar.gz
Size 13.0 kB
Tags Source
SHA-256 checksum
How to use checksums
5f28b1aeff6b315be1b66002c284c34b586399d555525103a6151b8844c3227f
BLAKE2b-256 checksum
How to use checksums
b1148160b9c2868b5e9f9e3fd545c22ce84b33231f4f90828eb06e76647e9995
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp314-cp314-win_amd64.whl

Download URL mdplib-1.0.2-cp314-cp314-win_amd64.whl
Size 308.1 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
a1a9b19dec463f337230d6d52b253897fda40a13490bbc8a397a5a65a5d7d4b2
BLAKE2b-256 checksum
How to use checksums
355759c8ab1264b620a713c624faa5bac79a47ec5c805e5e044841d1798754a0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.2-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.14 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
393901b927ddb4ecb6fe30b32fdc78b487319b63057edf7c0fd3ba98e14e4f5c
BLAKE2b-256 checksum
How to use checksums
4cebe1fa20eae697a8b1d3b309dd5bdb0702a61ebfde4901cff52d72d6ffdc65
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp314-cp314-macosx_14_0_arm64.whl

Download URL mdplib-1.0.2-cp314-cp314-macosx_14_0_arm64.whl
Size 112.6 kB
Tags CPython 3.14 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
e235429a61b8dd36160d116fcfd6ca8a5eeaaa99629fefcde7768176c3184d9f
BLAKE2b-256 checksum
How to use checksums
63300f5507d170e42855ce365d6d73d816f225a37ca07b32159b23b4f9ea1b5a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp313-cp313-win_amd64.whl

Download URL mdplib-1.0.2-cp313-cp313-win_amd64.whl
Size 298.4 kB
Tags CPython 3.13 Windows x86-64
SHA-256 checksum
How to use checksums
42291c41c9a8e5a4edfd8816e3a5e1b306296003af1cf08264d8de912bd7e68b
BLAKE2b-256 checksum
How to use checksums
c90d109a490de77eb515241948d786d375ea26b6a249cb483487c1d68a59a93c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.13 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
745c96214b86f5b413d32c7333ac3f9465317402f15b66ae05bc7b7500fdc507
BLAKE2b-256 checksum
How to use checksums
7de072e43ae45bcddd4f0dc01a6fdbad89f63020c9a057ede6c75cd73c3f5977
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp313-cp313-macosx_14_0_arm64.whl

Download URL mdplib-1.0.2-cp313-cp313-macosx_14_0_arm64.whl
Size 112.8 kB
Tags CPython 3.13 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
4f3b25c8a4bd455397db69462033ede1e7e525db019036a9b0fbc9560ec22fc2
BLAKE2b-256 checksum
How to use checksums
0207e6de208116f62d24ef152de576ff7d455153988b13d53baceacb8d609332
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp312-cp312-win_amd64.whl

Download URL mdplib-1.0.2-cp312-cp312-win_amd64.whl
Size 298.3 kB
Tags CPython 3.12 Windows x86-64
SHA-256 checksum
How to use checksums
d2697d1d727d11a8b6697f70924acd986ad20be42e4d8a92438f29e6c148e849
BLAKE2b-256 checksum
How to use checksums
b39bd20e3d99576281224898cedf04cbed8b83a1ede8d2835f5fb885e2cc7d56
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.2-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.12 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
4510d144773d3593f56f4503310ed0a98aa086ee04afe8a6dffe83a4ebf3bb14
BLAKE2b-256 checksum
How to use checksums
8dc04045dd2433fb88d2497d4acff63f01162b6f05e9f2d9929991d9109a3809
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp312-cp312-macosx_14_0_arm64.whl

Download URL mdplib-1.0.2-cp312-cp312-macosx_14_0_arm64.whl
Size 112.8 kB
Tags CPython 3.12 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
00e84659e79c5d07dea27d5bb20a7f8bcf9e0c80f3ca505b428b7d5517b05837
BLAKE2b-256 checksum
How to use checksums
d22266d6c9b4f1168d08b3a9c0549e3eb00100d27b3acd3a4503a2b899c952ee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp311-cp311-win_amd64.whl

Download URL mdplib-1.0.2-cp311-cp311-win_amd64.whl
Size 295.5 kB
Tags CPython 3.11 Windows x86-64
SHA-256 checksum
How to use checksums
0efeeba75f69e17ff01b40030c6841817e110228a2d86db6f248f035a7571760
BLAKE2b-256 checksum
How to use checksums
9818060e4b8770a7160b7d5c9ea9e6747276eceb9c13ec84a15d70231125b829
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.2-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.4 MB
Tags CPython 3.11 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
5c3729fc3de7e061b475ccca529118d5fee5082de9fc73a54bee0b03a0a8b3f9
BLAKE2b-256 checksum
How to use checksums
d62a36b407956e765722732b6a4a36db0423feef8e9fbedf73a7a7957a4f1a0a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp311-cp311-macosx_14_0_arm64.whl

Download URL mdplib-1.0.2-cp311-cp311-macosx_14_0_arm64.whl
Size 109.7 kB
Tags CPython 3.11 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
475c9007df6cee1a2cd2dababb643344d3fdfd608a4e3e68a0f93daf03301e83
BLAKE2b-256 checksum
How to use checksums
5c6d2c179958b564b91cb6eabee6f41a2a24a361eef6287e9ffd3fa2349e664b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp310-cp310-win_amd64.whl

Download URL mdplib-1.0.2-cp310-cp310-win_amd64.whl
Size 295.0 kB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
0efc03e6eeaa4b430416d0508f73fd23c52f6d9a8ad0f15f3ef411f304979883
BLAKE2b-256 checksum
How to use checksums
13143fac3e4dbf16220daeb4c8e408c36387694151de351adbd318c500c30289
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.4 MB
Tags CPython 3.10 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
e834db1b34be94097f2c973d5e510072de25c65e398aff3228f6a17a5c9ac47c
BLAKE2b-256 checksum
How to use checksums
d7ff76b1b927b5859e8429c59c18fc4c9d6fce8094f81323745c1290e356a1b0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release files / mdplib-1.0.2-cp310-cp310-macosx_14_0_arm64.whl

Download URL mdplib-1.0.2-cp310-cp310-macosx_14_0_arm64.whl
Size 108.6 kB
Tags CPython 3.10 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
fa52d55e85845db804ea591da22581738588bb846f98b0ed3c7e4b873a1bbb96
BLAKE2b-256 checksum
How to use checksums
3a900fce7ecd42a37732e5c21a523677a2840259b38dc53695c12cefd1c9f2ee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 16, 2026.

Transparency log

Release history Release notifications | RSS feed

This release

1.0.2 This release

16 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page