Skip to main content

mdplib

PyPI Python

A Python library of algorithms for solving infinite-horizon, discounted Markov Decision Processes (MDPs), with a performance-critical core written in C++ (via pybind11).

Implements:

  • Value Iteration (Jacobi and Gauss-Seidel variants)
  • Policy Iteration
  • Q-Learning and SARSA (model-free reinforcement learning)

Installation

pip install mdplib

Prebuilt wheels are published for Linux, Windows and macOS (Python 3.10-3.14), so this does not require a C++ compiler on your machine. If your platform/Python version has no matching wheel, pip falls back to building from source, which does require a C++17 compiler (the extension is compiled at install time):

git clone https://github.com/LuccaGiannelli/mdp-solver.git
cd mdp-solver
pip install .

Basic usage

from mdp import MDP

S = [0, 1, 2]   # states
A = [0, 1, 2]   # actions

def Pt(sn, a, sn1):
    # returns P(sn1 | sn, a)
    ...

def rt(sn, a, sn1):
    # returns the reward
    ...

problem = MDP(S, A, Pt, rt)

# Value Iteration
problem.VI(gamma=0.9, tol=1e-6)
problem.print_results()

# Policy Iteration
problem.PI(gamma=0.9, tol=1e-6)
problem.print_results()

See examples/basic_example.py for a complete example (a machine maintenance problem).

Available algorithms

Method Description
VI Value Iteration (Jacobi)
VIGS Value Iteration with Gauss-Seidel
PI Policy Iteration
QLearning Q-Learning (off-policy TD control)
SARSA SARSA (on-policy TD control)

Common parameters

Parameter Description
gamma Discount factor (0 ≤ γ < 1)
tol Convergence tolerance
max_iter Maximum number of iterations
v0 Initial value vector (optional)
verbose Print progress on completion
norm_type 0 = relative sup norm, 1 = sup norm, 2 = euclidean norm
epsilon Final epsilon for the epsilon-greedy policy (Q-Learning / SARSA)
alpha Learning rate (Q-Learning / SARSA)

How it works

You define the state space S, action space A, a transition probability function Pt(sn, a, sn1), and a reward function rt(sn, a, sn1). The constructor enumerates all feasible transitions into a CSV (sn,a,sn1,prob,reward) and loads it into a NumPy array; the algorithms then hand that array to the C++ core, which does the inner loops with direct memory access for performance.

After solving, results are available as:

Attribute Description
value optimal value for each state
policy optimal action for each state
time execution time (seconds)
n_iterations number of iterations to convergence

Adding a new algorithm

  1. Create mdp/algorithms/new_algo.py with a function def NewAlgo(self, ...):
  2. Import it in mdp/algorithms/__init__.py
  3. Attach it to the class in mdp/core.py: NewAlgo = NewAlgo

Tests

pip install pytest
pytest tests/

License

TBD.

Metadata

Release files for mdplib 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mdplib 1.0.0
File Size Uploaded
mdplib-1.0.0.tar.gz 12.2 kB Details

Built distributions (wheels)

Table of built distributions (wheels) for mdplib 1.0.0
File
mdplib-1.0.0-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details
mdplib-1.0.0-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.14 CPython 3.14 Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 Details
mdplib-1.0.0-cp314-cp314-macosx_14_0_arm64.whl CPython 3.14 CPython 3.14 macOS 14.0+ ARM64 Details
mdplib-1.0.0-cp313-cp313-win_amd64.whl CPython 3.13 CPython 3.13 Windows x86-64 Details
mdplib-1.0.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.0-cp313-cp313-macosx_14_0_arm64.whl CPython 3.13 CPython 3.13 macOS 14.0+ ARM64 Details
mdplib-1.0.0-cp312-cp312-win_amd64.whl CPython 3.12 CPython 3.12 Windows x86-64 Details
mdplib-1.0.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.0-cp312-cp312-macosx_14_0_arm64.whl CPython 3.12 CPython 3.12 macOS 14.0+ ARM64 Details
mdplib-1.0.0-cp311-cp311-win_amd64.whl CPython 3.11 CPython 3.11 Windows x86-64 Details
mdplib-1.0.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.24+ x86-64, Linux glibc 2.28+ x86-64 Details
mdplib-1.0.0-cp311-cp311-macosx_14_0_arm64.whl CPython 3.11 CPython 3.11 macOS 14.0+ ARM64 Details
mdplib-1.0.0-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details
mdplib-1.0.0-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.0-cp310-cp310-macosx_14_0_arm64.whl CPython 3.10 CPython 3.10 macOS 14.0+ ARM64 Details

Total release size: 9.4 MB

Release files / mdplib-1.0.0.tar.gz

Download URL mdplib-1.0.0.tar.gz
Size 12.2 kB
Tags Source
SHA-256 checksum
How to use checksums
55accdc5b5e31ae512113668969fda846369a7468b148b44589032b10d7e05c6
BLAKE2b-256 checksum
How to use checksums
e9be18d8d26dfddd0edc9649cfc106a244daf7346b9b716c93dabb53e5e2ca88
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp314-cp314-win_amd64.whl

Download URL mdplib-1.0.0-cp314-cp314-win_amd64.whl
Size 307.1 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
993b4fcde825bfa6f53d3ee10230268d862a6011eb54a15f5c10a20c64742a58
BLAKE2b-256 checksum
How to use checksums
b8c83ba2480e502a13f5443852ea6eada39252f35108d9bb33c5f551ee7fc3b6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.0-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.14 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
b2cad5491a5dfe09b50328a174eaaef0daa3eff9a821d94e040acc0e45ca8faa
BLAKE2b-256 checksum
How to use checksums
0bddd8fe9b602729b7c4791459aa2a1d545cf0febaae42f10357f7db303a6bbe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp314-cp314-macosx_14_0_arm64.whl

Download URL mdplib-1.0.0-cp314-cp314-macosx_14_0_arm64.whl
Size 111.6 kB
Tags CPython 3.14 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
6e5842b812be1814f1c90aff15aab0f5b03980741755689b590a1bb88c4ea319
BLAKE2b-256 checksum
How to use checksums
81d396e32ad2cdc77a614889411b4ef453022801607c5fdc961f35c9f0b49243
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp313-cp313-win_amd64.whl

Download URL mdplib-1.0.0-cp313-cp313-win_amd64.whl
Size 297.3 kB
Tags CPython 3.13 Windows x86-64
SHA-256 checksum
How to use checksums
3ae4caeb7b20b83b898cbc8af0926505144c033edf087a32e8a21d2be9ea171a
BLAKE2b-256 checksum
How to use checksums
45c873385bb0c1d859e330dfa78f9949581e5c28bb8b494be5148e9c923c5bfa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.13 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
7987e94a5b7916d0cabee5fd8481041312dcf354bf053cedafbbf7246df7c898
BLAKE2b-256 checksum
How to use checksums
07827ea79a3f7c56d094e60a9902f020c59d5b63f109773c800ea4489edd7f3f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp313-cp313-macosx_14_0_arm64.whl

Download URL mdplib-1.0.0-cp313-cp313-macosx_14_0_arm64.whl
Size 111.8 kB
Tags CPython 3.13 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
c104c1f86a55c07b7f16bcbd352ea50fc76195c09201784a5828719b12d903d4
BLAKE2b-256 checksum
How to use checksums
6c10f1b5d29dc9e9c87fe0159416a48140ab489d73609207aab704201529a6ff
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp312-cp312-win_amd64.whl

Download URL mdplib-1.0.0-cp312-cp312-win_amd64.whl
Size 297.3 kB
Tags CPython 3.12 Windows x86-64
SHA-256 checksum
How to use checksums
98b3a6e54bb8ad54ef71a75bf284458dd94e2e8316cbf20728784a6004ae96bb
BLAKE2b-256 checksum
How to use checksums
9c870cf08a4f383be450a9e23a9984a081f4bb8badb0bec2e98588eecb4fd562
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.12 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
2181a0ba986c2db649cf07d0c582099e84adf9521ced45066cd3158dee5fb6a8
BLAKE2b-256 checksum
How to use checksums
02faa3fc44a9092927d4b993a377d3dffd7cac5f930018f31e420cb8eab6519c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp312-cp312-macosx_14_0_arm64.whl

Download URL mdplib-1.0.0-cp312-cp312-macosx_14_0_arm64.whl
Size 111.7 kB
Tags CPython 3.12 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
0efbf7a2a4a671ee85f6998e4d416ebcaeec3007c1581bc5c1d8681c91e4f075
BLAKE2b-256 checksum
How to use checksums
81890664d9dafbf8f5c8ea9f0423253ccd70a7ae0f2fecdadfb9912a74318ec9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp311-cp311-win_amd64.whl

Download URL mdplib-1.0.0-cp311-cp311-win_amd64.whl
Size 294.5 kB
Tags CPython 3.11 Windows x86-64
SHA-256 checksum
How to use checksums
a913e148669c688c49e845aefb02f08e5d7137023f7b21370b5ec217bde060ae
BLAKE2b-256 checksum
How to use checksums
c36253917eb7b3af9c732f55336eec914f4639dab510a96d98452fcd96ebd2fc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.0-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.4 MB
Tags CPython 3.11 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
a1d6ce10370168c144ad714442887b05bf168fde48dde4e098e31cf45618c301
BLAKE2b-256 checksum
How to use checksums
b64b1d3d7e2072ff39a9d59d23ac8b62ffdbc4e69eaffe62bec784feb6b692a0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp311-cp311-macosx_14_0_arm64.whl

Download URL mdplib-1.0.0-cp311-cp311-macosx_14_0_arm64.whl
Size 108.7 kB
Tags CPython 3.11 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
49cd70b48e24f6b0912568e62bd58f46bc44870095734f23c10749ef44fa6842
BLAKE2b-256 checksum
How to use checksums
2b34241a8c296b09e8751261c5fc14dfe082a7571d3b7408be6278142cfff331
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp310-cp310-win_amd64.whl

Download URL mdplib-1.0.0-cp310-cp310-win_amd64.whl
Size 294.0 kB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
e3be6deb7c28a20f22a899fdddc9fa39ab35cc7fc4ebe3ebe92146da8654a08f
BLAKE2b-256 checksum
How to use checksums
e007ce853f7b7894c579ceefe9d505be60bd2c0f72a07da3ab361693a1202e23
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.0-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.4 MB
Tags CPython 3.10 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
02f81bba12fac96845cbdda8c1a1cb858912a29779d7c7197f6d9abb5f0aa00f
BLAKE2b-256 checksum
How to use checksums
903867f32eacbeaa6d449b04661d59a599b8dd40793819a70fe4ba10d5186bb6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release files / mdplib-1.0.0-cp310-cp310-macosx_14_0_arm64.whl

Download URL mdplib-1.0.0-cp310-cp310-macosx_14_0_arm64.whl
Size 107.6 kB
Tags CPython 3.10 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
f919bc86cb32be3d1ab2cf4e5e377b160fc3659059eb1037d00b3a1a4c08898c
BLAKE2b-256 checksum
How to use checksums
ba3e9ab90f7dc9437ff608639f76413d841619e5a35c7db27b0f1751f3adf167
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 1, 2026.

Transparency log

Release history Release notifications | RSS feed

1.0.2

16 release files

This release

1.0.0 This release

16 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page