mdplib
A Python library of algorithms for solving infinite-horizon, discounted Markov Decision Processes (MDPs), with a performance-critical core written in C++ (via pybind11).
Implements:
- Value Iteration (Jacobi and Gauss-Seidel variants)
- Policy Iteration
- Q-Learning and SARSA (model-free reinforcement learning)
Installation
pip install mdplib
Prebuilt wheels are published for Linux, Windows and macOS (Python 3.10-3.14), so this does not require a C++ compiler on your machine. If your platform/Python version has no matching wheel, pip falls back to building from source, which does require a C++17 compiler (the extension is compiled at install time):
git clone https://github.com/LuccaGiannelli/mdplib.git
cd mdplib
pip install .
Basic usage
from mdp import MDP
S = [0, 1, 2] # states
A = [0, 1, 2] # actions
def Pt(sn, a, sn1):
# returns P(sn1 | sn, a)
...
def rt(sn, a, sn1):
# returns the reward
...
problem = MDP(S, A, Pt, rt)
# Value Iteration
problem.VI(gamma=0.9, tol=1e-6)
problem.print_results()
# Policy Iteration
problem.PI(gamma=0.9, tol=1e-6)
problem.print_results()
See examples/basic_example.py for a complete example (a machine maintenance problem).
Available algorithms
| Method | Description |
|---|---|
VI |
Value Iteration (Jacobi) |
VIGS |
Value Iteration with Gauss-Seidel |
PI |
Policy Iteration |
QLearning |
Q-Learning (off-policy TD control) |
SARSA |
SARSA (on-policy TD control) |
Common parameters
| Parameter | Description |
|---|---|
gamma |
Discount factor (0 ≤ γ < 1) |
tol |
Convergence tolerance |
max_iter |
Maximum number of iterations |
v0 |
Initial value vector (optional) |
verbose |
Print progress on completion |
norm_type |
0 = relative sup norm, 1 = sup norm, 2 = euclidean norm |
epsilon |
Final epsilon for the epsilon-greedy policy (Q-Learning / SARSA) |
alpha |
Learning rate (Q-Learning / SARSA) |
How it works
You define the state space S, action space A, a transition probability function Pt(sn, a, sn1), and a reward function rt(sn, a, sn1). The constructor enumerates all feasible transitions into a CSV (sn,a,sn1,prob,reward) and loads it into a NumPy array; the algorithms then hand that array to the C++ core, which does the inner loops with direct memory access for performance.
After solving, results are available as:
| Attribute | Description |
|---|---|
value |
optimal value for each state |
policy |
optimal action for each state |
time |
execution time (seconds) |
n_iterations |
number of iterations to convergence |
Adding a new algorithm
- Create
mdp/algorithms/new_algo.pywith a functiondef NewAlgo(self, ...): - Import it in
mdp/algorithms/__init__.py - Attach it to the class in
mdp/core.py:NewAlgo = NewAlgo
Tests
pip install pytest
pytest tests/
License
Metadata
Release files for mdplib 1.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mdplib-1.0.4.tar.gz | 13.1 kB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| mdplib-1.0.4-cp314-cp314-win_amd64.whl | CPython 3.14 | CPython 3.14 | Windows x86-64 | Details |
| mdplib-1.0.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl | CPython 3.14 | CPython 3.14 | Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 | Details |
| mdplib-1.0.4-cp314-cp314-macosx_14_0_arm64.whl | CPython 3.14 | CPython 3.14 | macOS 14.0+ ARM64 | Details |
| mdplib-1.0.4-cp313-cp313-win_amd64.whl | CPython 3.13 | CPython 3.13 | Windows x86-64 | Details |
| mdplib-1.0.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl | CPython 3.13 | CPython 3.13 | Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 | Details |
| mdplib-1.0.4-cp313-cp313-macosx_14_0_arm64.whl | CPython 3.13 | CPython 3.13 | macOS 14.0+ ARM64 | Details |
| mdplib-1.0.4-cp312-cp312-win_amd64.whl | CPython 3.12 | CPython 3.12 | Windows x86-64 | Details |
| mdplib-1.0.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl | CPython 3.12 | CPython 3.12 | Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 | Details |
| mdplib-1.0.4-cp312-cp312-macosx_14_0_arm64.whl | CPython 3.12 | CPython 3.12 | macOS 14.0+ ARM64 | Details |
| mdplib-1.0.4-cp311-cp311-win_amd64.whl | CPython 3.11 | CPython 3.11 | Windows x86-64 | Details |
| mdplib-1.0.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl | CPython 3.11 | CPython 3.11 | Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 | Details |
| mdplib-1.0.4-cp311-cp311-macosx_14_0_arm64.whl | CPython 3.11 | CPython 3.11 | macOS 14.0+ ARM64 | Details |
| mdplib-1.0.4-cp310-cp310-win_amd64.whl | CPython 3.10 | CPython 3.10 | Windows x86-64 | Details |
| mdplib-1.0.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl | CPython 3.10 | CPython 3.10 | Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 | Details |
| mdplib-1.0.4-cp310-cp310-macosx_14_0_arm64.whl | CPython 3.10 | CPython 3.10 | macOS 14.0+ ARM64 | Details |
Total release size: 9.4 MB
Release files / mdplib-1.0.4.tar.gz
| Download URL | mdplib-1.0.4.tar.gz |
|---|---|
| Size | 13.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6b16a404e49e4b5f0062ba292bcfb9fe7f36aa43755e7fb0355037253fea3bc6
|
|
BLAKE2b-256 checksum How to use checksums |
a87dd285a1c9aa4df47d768d91ac5489961f0d9e1cafd04a961d50ce2a1985c4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp314-cp314-win_amd64.whl
| Download URL | mdplib-1.0.4-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 308.2 kB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
b28b4ddfa800fda832abe86229f6f44b4049c8cc474ba6a9e56b9ebbada2cabc
|
|
BLAKE2b-256 checksum How to use checksums |
ba93628291b70eee28c2058eab87d2cb47e5f89ebe47a5133861e61076ee7f11
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdplib-1.0.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.14 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
eaf7d042a8328f1c8517ab6730fa717dd50971b2188fc782c13390f174f8aa96
|
|
BLAKE2b-256 checksum How to use checksums |
98a359018d1c3353c0b93ed8256fc03f352046a6f6cc91ea3c35b9034d15c787
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp314-cp314-macosx_14_0_arm64.whl
| Download URL | mdplib-1.0.4-cp314-cp314-macosx_14_0_arm64.whl |
|---|---|
| Size | 112.6 kB |
| Tags | CPython 3.14 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
a2e622b5aeece1559b9d0a2930bb1a78896f8f1a3871d397cc095990a789771a
|
|
BLAKE2b-256 checksum How to use checksums |
b8c4e766f6bdfc85fcb0d3970d97b4bcd9160741816a1baffdd95c4699199f00
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp313-cp313-win_amd64.whl
| Download URL | mdplib-1.0.4-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 298.4 kB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
8429d578bfd6ca4f2a1b1ee23d08a9af11559c0557762a55d534732dbd963489
|
|
BLAKE2b-256 checksum How to use checksums |
7aee50cc09011f52d47ce5b1626be9d0c6b93425f89bf9a547650309e4f3cc66
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdplib-1.0.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.13 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
e854dfa7c89d8eb823ed38be4b00c14a402f644207cc39b5a360d3c9488430da
|
|
BLAKE2b-256 checksum How to use checksums |
93de0737f400f1212009828f64d4a7feb30c32e5ae0e85266e693df97df66bcf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp313-cp313-macosx_14_0_arm64.whl
| Download URL | mdplib-1.0.4-cp313-cp313-macosx_14_0_arm64.whl |
|---|---|
| Size | 112.9 kB |
| Tags | CPython 3.13 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
3c01c38e4df626f4a29b2841fd2b7c3080da52862f9e88cd4c8b0cdf541fa4d3
|
|
BLAKE2b-256 checksum How to use checksums |
29a2ba44703746ec3ee46a14354383c98a958e4f6b37a127672de4983b5cdb3e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp312-cp312-win_amd64.whl
| Download URL | mdplib-1.0.4-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 298.3 kB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
44ef613e4ebbefcf953403a281d51dfe84a1f7a91a8107db9e8c860d49c68527
|
|
BLAKE2b-256 checksum How to use checksums |
1afdd44ea7d71ad91c0c0f158846c6dadb1df0838c3edd7a4fb83b0255615a70
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdplib-1.0.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.5 MB |
| Tags | CPython 3.12 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
8efa26f3c4a7e39a9cbede7c70aa12b586402c4be3a95516e635734e209a86a6
|
|
BLAKE2b-256 checksum How to use checksums |
f10647d487c4c8e6d73cf22a3a79558635091eb263d3f51218b58639c67ffdbd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp312-cp312-macosx_14_0_arm64.whl
| Download URL | mdplib-1.0.4-cp312-cp312-macosx_14_0_arm64.whl |
|---|---|
| Size | 112.8 kB |
| Tags | CPython 3.12 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
168f143c7558c6eb3b0d536dd3a6a164aed99fb31ec28028e336545cb96cdf48
|
|
BLAKE2b-256 checksum How to use checksums |
2767654db63fdef9e98fdcba552a22ef341bf13319cd8e13095e5d3df95fdf9a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp311-cp311-win_amd64.whl
| Download URL | mdplib-1.0.4-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 295.5 kB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
a50ac321359940b2b7f19ce656370c1c8f3441c65a376712454c141512d77eb6
|
|
BLAKE2b-256 checksum How to use checksums |
9ae850012246171d816e84454c5ea99cb8313571b1f88153d4f84bb246295b5e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdplib-1.0.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.4 MB |
| Tags | CPython 3.11 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
e3a9780d4f839e38a5eb40e8b6cc233d852def475440a1735c2a8c450e048312
|
|
BLAKE2b-256 checksum How to use checksums |
0f24bae9b52979f948cc849f3b8448602489e1b4dbee2f9d1be2e713d1116f27
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp311-cp311-macosx_14_0_arm64.whl
| Download URL | mdplib-1.0.4-cp311-cp311-macosx_14_0_arm64.whl |
|---|---|
| Size | 109.8 kB |
| Tags | CPython 3.11 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
ec1e2113d76c31a9d7cb70e000bc1957be54910c00db6c2e252d002dcd7a73f4
|
|
BLAKE2b-256 checksum How to use checksums |
516ff26b7ada4c73b93a15962051dd12796664311553c23d2b100af63d3cec57
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp310-cp310-win_amd64.whl
| Download URL | mdplib-1.0.4-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 295.0 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
1c31144ec962713b0f8e27bf1ed3175cc028baac9b0cbb2ee412d98978760a2c
|
|
BLAKE2b-256 checksum How to use checksums |
a5b7b06162934ae87934f3712e48e9177ca67fcf78c0d1935a44608ae45003cc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdplib-1.0.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 1.4 MB |
| Tags | CPython 3.10 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
b64a14596834325c5f572199981b1713ca8655ca294518f07765b90b8cf56b6f
|
|
BLAKE2b-256 checksum How to use checksums |
166c2047b3671476c41e81b5f0d903e9837f38bf6242e79a5ff4f82c7bffe1de
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency logRelease files / mdplib-1.0.4-cp310-cp310-macosx_14_0_arm64.whl
| Download URL | mdplib-1.0.4-cp310-cp310-macosx_14_0_arm64.whl |
|---|---|
| Size | 108.6 kB |
| Tags | CPython 3.10 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
debe0efc3294b244c016b1c3d75a1014b8f4e7b1c6bd4cffd48bca2da26de4da
|
|
BLAKE2b-256 checksum How to use checksums |
1f04119d84fab942a4b4dd36c5ac4290f708e7a9df6198aea0485621d788cd51
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.
Transparency log