Skip to main content

mdplib

PyPI Python DOI

A Python library of algorithms for solving infinite-horizon, discounted Markov Decision Processes (MDPs), with a performance-critical core written in C++ (via pybind11).

Implements:

  • Value Iteration (Jacobi and Gauss-Seidel variants)
  • Policy Iteration
  • Q-Learning and SARSA (model-free reinforcement learning)

Installation

pip install mdplib

Prebuilt wheels are published for Linux, Windows and macOS (Python 3.10-3.14), so this does not require a C++ compiler on your machine. If your platform/Python version has no matching wheel, pip falls back to building from source, which does require a C++17 compiler (the extension is compiled at install time):

git clone https://github.com/LuccaGiannelli/mdplib.git
cd mdplib
pip install .

Basic usage

from mdp import MDP

S = [0, 1, 2]   # states
A = [0, 1, 2]   # actions

def Pt(sn, a, sn1):
    # returns P(sn1 | sn, a)
    ...

def rt(sn, a, sn1):
    # returns the reward
    ...

problem = MDP(S, A, Pt, rt)

# Value Iteration
problem.VI(gamma=0.9, tol=1e-6)
problem.print_results()

# Policy Iteration
problem.PI(gamma=0.9, tol=1e-6)
problem.print_results()

See examples/basic_example.py for a complete example (a machine maintenance problem).

Available algorithms

Method Description
VI Value Iteration (Jacobi)
VIGS Value Iteration with Gauss-Seidel
PI Policy Iteration
QLearning Q-Learning (off-policy TD control)
SARSA SARSA (on-policy TD control)

Common parameters

Parameter Description
gamma Discount factor (0 ≤ γ < 1)
tol Convergence tolerance
max_iter Maximum number of iterations
v0 Initial value vector (optional)
verbose Print progress on completion
norm_type 0 = relative sup norm, 1 = sup norm, 2 = euclidean norm
epsilon Final epsilon for the epsilon-greedy policy (Q-Learning / SARSA)
alpha Learning rate (Q-Learning / SARSA)

How it works

You define the state space S, action space A, a transition probability function Pt(sn, a, sn1), and a reward function rt(sn, a, sn1). The constructor enumerates all feasible transitions into a CSV (sn,a,sn1,prob,reward) and loads it into a NumPy array; the algorithms then hand that array to the C++ core, which does the inner loops with direct memory access for performance.

After solving, results are available as:

Attribute Description
value optimal value for each state
policy optimal action for each state
time execution time (seconds)
n_iterations number of iterations to convergence

Adding a new algorithm

  1. Create mdp/algorithms/new_algo.py with a function def NewAlgo(self, ...):
  2. Import it in mdp/algorithms/__init__.py
  3. Attach it to the class in mdp/core.py: NewAlgo = NewAlgo

Tests

pip install pytest
pytest tests/

License

MIT

Metadata

Release files for mdplib 1.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mdplib 1.0.4
File Size Uploaded
mdplib-1.0.4.tar.gz 13.1 kB Details

Built distributions (wheels)

Table of built distributions (wheels) for mdplib 1.0.4
File
mdplib-1.0.4-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details
mdplib-1.0.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.14 CPython 3.14 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.4-cp314-cp314-macosx_14_0_arm64.whl CPython 3.14 CPython 3.14 macOS 14.0+ ARM64 Details
mdplib-1.0.4-cp313-cp313-win_amd64.whl CPython 3.13 CPython 3.13 Windows x86-64 Details
mdplib-1.0.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.4-cp313-cp313-macosx_14_0_arm64.whl CPython 3.13 CPython 3.13 macOS 14.0+ ARM64 Details
mdplib-1.0.4-cp312-cp312-win_amd64.whl CPython 3.12 CPython 3.12 Windows x86-64 Details
mdplib-1.0.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.4-cp312-cp312-macosx_14_0_arm64.whl CPython 3.12 CPython 3.12 macOS 14.0+ ARM64 Details
mdplib-1.0.4-cp311-cp311-win_amd64.whl CPython 3.11 CPython 3.11 Windows x86-64 Details
mdplib-1.0.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.4-cp311-cp311-macosx_14_0_arm64.whl CPython 3.11 CPython 3.11 macOS 14.0+ ARM64 Details
mdplib-1.0.4-cp310-cp310-win_amd64.whl CPython 3.10 CPython 3.10 Windows x86-64 Details
mdplib-1.0.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.28+ x86-64, Linux glibc 2.24+ x86-64 Details
mdplib-1.0.4-cp310-cp310-macosx_14_0_arm64.whl CPython 3.10 CPython 3.10 macOS 14.0+ ARM64 Details

Total release size: 9.4 MB

Release files / mdplib-1.0.4.tar.gz

Download URL mdplib-1.0.4.tar.gz
Size 13.1 kB
Tags Source
SHA-256 checksum
How to use checksums
6b16a404e49e4b5f0062ba292bcfb9fe7f36aa43755e7fb0355037253fea3bc6
BLAKE2b-256 checksum
How to use checksums
a87dd285a1c9aa4df47d768d91ac5489961f0d9e1cafd04a961d50ce2a1985c4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp314-cp314-win_amd64.whl

Download URL mdplib-1.0.4-cp314-cp314-win_amd64.whl
Size 308.2 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
b28b4ddfa800fda832abe86229f6f44b4049c8cc474ba6a9e56b9ebbada2cabc
BLAKE2b-256 checksum
How to use checksums
ba93628291b70eee28c2058eab87d2cb47e5f89ebe47a5133861e61076ee7f11
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.14 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
eaf7d042a8328f1c8517ab6730fa717dd50971b2188fc782c13390f174f8aa96
BLAKE2b-256 checksum
How to use checksums
98a359018d1c3353c0b93ed8256fc03f352046a6f6cc91ea3c35b9034d15c787
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp314-cp314-macosx_14_0_arm64.whl

Download URL mdplib-1.0.4-cp314-cp314-macosx_14_0_arm64.whl
Size 112.6 kB
Tags CPython 3.14 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
a2e622b5aeece1559b9d0a2930bb1a78896f8f1a3871d397cc095990a789771a
BLAKE2b-256 checksum
How to use checksums
b8c4e766f6bdfc85fcb0d3970d97b4bcd9160741816a1baffdd95c4699199f00
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp313-cp313-win_amd64.whl

Download URL mdplib-1.0.4-cp313-cp313-win_amd64.whl
Size 298.4 kB
Tags CPython 3.13 Windows x86-64
SHA-256 checksum
How to use checksums
8429d578bfd6ca4f2a1b1ee23d08a9af11559c0557762a55d534732dbd963489
BLAKE2b-256 checksum
How to use checksums
7aee50cc09011f52d47ce5b1626be9d0c6b93425f89bf9a547650309e4f3cc66
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.13 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
e854dfa7c89d8eb823ed38be4b00c14a402f644207cc39b5a360d3c9488430da
BLAKE2b-256 checksum
How to use checksums
93de0737f400f1212009828f64d4a7feb30c32e5ae0e85266e693df97df66bcf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp313-cp313-macosx_14_0_arm64.whl

Download URL mdplib-1.0.4-cp313-cp313-macosx_14_0_arm64.whl
Size 112.9 kB
Tags CPython 3.13 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
3c01c38e4df626f4a29b2841fd2b7c3080da52862f9e88cd4c8b0cdf541fa4d3
BLAKE2b-256 checksum
How to use checksums
29a2ba44703746ec3ee46a14354383c98a958e4f6b37a127672de4983b5cdb3e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp312-cp312-win_amd64.whl

Download URL mdplib-1.0.4-cp312-cp312-win_amd64.whl
Size 298.3 kB
Tags CPython 3.12 Windows x86-64
SHA-256 checksum
How to use checksums
44ef613e4ebbefcf953403a281d51dfe84a1f7a91a8107db9e8c860d49c68527
BLAKE2b-256 checksum
How to use checksums
1afdd44ea7d71ad91c0c0f158846c6dadb1df0838c3edd7a4fb83b0255615a70
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.5 MB
Tags CPython 3.12 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
8efa26f3c4a7e39a9cbede7c70aa12b586402c4be3a95516e635734e209a86a6
BLAKE2b-256 checksum
How to use checksums
f10647d487c4c8e6d73cf22a3a79558635091eb263d3f51218b58639c67ffdbd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp312-cp312-macosx_14_0_arm64.whl

Download URL mdplib-1.0.4-cp312-cp312-macosx_14_0_arm64.whl
Size 112.8 kB
Tags CPython 3.12 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
168f143c7558c6eb3b0d536dd3a6a164aed99fb31ec28028e336545cb96cdf48
BLAKE2b-256 checksum
How to use checksums
2767654db63fdef9e98fdcba552a22ef341bf13319cd8e13095e5d3df95fdf9a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp311-cp311-win_amd64.whl

Download URL mdplib-1.0.4-cp311-cp311-win_amd64.whl
Size 295.5 kB
Tags CPython 3.11 Windows x86-64
SHA-256 checksum
How to use checksums
a50ac321359940b2b7f19ce656370c1c8f3441c65a376712454c141512d77eb6
BLAKE2b-256 checksum
How to use checksums
9ae850012246171d816e84454c5ea99cb8313571b1f88153d4f84bb246295b5e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.4 MB
Tags CPython 3.11 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
e3a9780d4f839e38a5eb40e8b6cc233d852def475440a1735c2a8c450e048312
BLAKE2b-256 checksum
How to use checksums
0f24bae9b52979f948cc849f3b8448602489e1b4dbee2f9d1be2e713d1116f27
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp311-cp311-macosx_14_0_arm64.whl

Download URL mdplib-1.0.4-cp311-cp311-macosx_14_0_arm64.whl
Size 109.8 kB
Tags CPython 3.11 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
ec1e2113d76c31a9d7cb70e000bc1957be54910c00db6c2e252d002dcd7a73f4
BLAKE2b-256 checksum
How to use checksums
516ff26b7ada4c73b93a15962051dd12796664311553c23d2b100af63d3cec57
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp310-cp310-win_amd64.whl

Download URL mdplib-1.0.4-cp310-cp310-win_amd64.whl
Size 295.0 kB
Tags CPython 3.10 Windows x86-64
SHA-256 checksum
How to use checksums
1c31144ec962713b0f8e27bf1ed3175cc028baac9b0cbb2ee412d98978760a2c
BLAKE2b-256 checksum
How to use checksums
a5b7b06162934ae87934f3712e48e9177ca67fcf78c0d1935a44608ae45003cc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl

Download URL mdplib-1.0.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
Size 1.4 MB
Tags CPython 3.10 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
b64a14596834325c5f572199981b1713ca8655ca294518f07765b90b8cf56b6f
BLAKE2b-256 checksum
How to use checksums
166c2047b3671476c41e81b5f0d903e9837f38bf6242e79a5ff4f82c7bffe1de
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release files / mdplib-1.0.4-cp310-cp310-macosx_14_0_arm64.whl

Download URL mdplib-1.0.4-cp310-cp310-macosx_14_0_arm64.whl
Size 108.6 kB
Tags CPython 3.10 macOS 14.0+ ARM64
SHA-256 checksum
How to use checksums
debe0efc3294b244c016b1c3d75a1014b8f4e7b1c6bd4cffd48bca2da26de4da
BLAKE2b-256 checksum
How to use checksums
1f04119d84fab942a4b4dd36c5ac4290f708e7a9df6198aea0485621d788cd51
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 6, 2026.

Transparency log

Release history Release notifications | RSS feed

This release

1.0.4 This release

16 release files

1.0.2

16 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page