MDPSolver
MDPSolver is a Python package for large Markov Decision Processes (MDPs) with infinite-horizons.
Features
- Fast solver: Our C++-based solver is substantially faster than other CPU-based MDP packages available for Python. See details in the documentation.
- Two optimality criteria: Discounted and Average reward.
- Three optimization algorithms: Value iteration, Policy iteration, and Modified policy iteration.
- Three value-update methods: Standard, Gauss–Seidel, and Successive over-relaxation.
- Supports sparse matrices.
- Employs parallel computing.
Installation
Prebuilt wheels are available for Linux, Windows, and macOS (14+) across Python 3.10–3.14. Install directly from PyPI with:
pip install mdpsolver
Quick start guide
The following shows how to get quickly started with mdpsolver.
Usage
Start by specifying the reward function and transition probabilities as lists. The following is an example of a simple MDP containing three states and two actions in each state.
#Import packages
import mdpsolver
#Rewards (3 states x 2 actions)
#e.g. choosing second action in first state gives reward=-1
rewards = [[5,-1],
[1,-2],
[50,0]]
#Transition probabilities (3 from_states x 2 actions x 3 to_states)
#e.g. choosing first action in third state gives a probability of 0.6 of staying in third state
tranMatWithZeros = [[[0.9,0.1,0.0],[0.1,0.9,0.0]],
[[0.4,0.5,0.1],[0.3,0.5,0.2]],
[[0.2,0.2,0.6],[0.5,0.5,0.0]]]
Now, create the model object and insert the problem parameters.
#Create model object
mdl = mdpsolver.model()
#Insert the problem parameters
mdl.mdp(discount=0.8,
rewards=rewards,
tranMatWithZeros=tranMatWithZeros)
We can now optimize the policy.
mdl.solve()
The optimized policy can be returned in a variety of ways. Here, we return the policy as a list and print directly in the terminal.
print(mdl.getPolicy())
#[1, 1, 0]
Sparse transition matrix?
mdpsolver has three alternative formats for large and highly sparse transition probability matrices.
(1) Elementwise representation (excluding elements containing zeros):
#[from_state,action,to_state,probability]
tranMatElementwise = [[0,0,0,0.9],
[0,0,1,0.1],
[0,1,0,0.1],
[0,1,1,0.9],
[1,0,0,0.4],
[1,0,1,0.5],
[1,0,2,0.1],
[1,1,0,0.3],
[1,1,1,0.5],
[1,1,2,0.2],
[2,0,0,0.2],
[2,0,1,0.2],
[2,0,2,0.6],
[2,1,0,0.5],
[2,1,1,0.5]]
mdl.mdp(discount=0.8,
rewards=rewards,
tranMatElementwise=tranMatElementwise)
(2) Probabilities and column (to_state) indices in separate lists:
tranMatProbs = [[[0.9,0.1],[0.1,0.9]],
[[0.4,0.5,0.1],[0.3,0.5,0.2]],
[[0.2,0.2,0.6],[0.5,0.5]]]
tranMatColumns = [[[0,1],[0,1]],
[[0,1,2],[0,1,2]],
[[0,1,2],[0,1]]]
mdl.mdp(discount=0.8,
rewards=rewards,
tranMatProbs=tranMatProbs,
tranMatColumns=tranMatColumns)
(3) Load the elementwise representation from a file:
mdl.mdp(discount=0.8,
rewards=rewards,
tranMatFromFile="transitions.csv")
Documentation
The documentation can be found in the wiki for MDPSolver.
How to cite
1 maintainer
Metadata
1 maintainer
Release files for mdpsolver 0.10.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mdpsolver-0.10.2.tar.gz | 226.2 kB | Details |
Built distributions (wheels)
Total release size: 6.6 MB
Release files / mdpsolver-0.10.2.tar.gz
| Download URL | mdpsolver-0.10.2.tar.gz |
|---|---|
| Size | 226.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2461a6b74765749257f75c6829fbb4ceb52f3aebe302dc43e8eeef4a73b88c6a
|
|
BLAKE2b-256 checksum How to use checksums |
d4864e00f9d7902f9eca84b1044fd546df463ea12e7baa8c7692e692a156ecc5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp314-cp314-win_amd64.whl
| Download URL | mdpsolver-0.10.2-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 653.5 kB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
a6a263dc4ef7e9db1f763edaed4aa1f3889422a2f16c101e76fe896142e071ef
|
|
BLAKE2b-256 checksum How to use checksums |
912dcd5379ac54bc59f16894025136346e29a5b81a7b91aacccd0c2a78e471ae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdpsolver-0.10.2-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 271.7 kB |
| Tags | CPython 3.14 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
4b45e5bd3d958559b41c01650fc1691c0e92e739e502835488f70979d08f227e
|
|
BLAKE2b-256 checksum How to use checksums |
9502f16725fac4d74a0d960bc2a40ab7efa7939265b1292af522734c3e034167
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp314-cp314-macosx_14_0_arm64.whl
| Download URL | mdpsolver-0.10.2-cp314-cp314-macosx_14_0_arm64.whl |
|---|---|
| Size | 374.2 kB |
| Tags | CPython 3.14 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
3873ea58beb3e84cca7057738f2d28f4bf081e8cbabb6537d329d01a0f035c8c
|
|
BLAKE2b-256 checksum How to use checksums |
3d758b8ec7dacf4558dba3acd4e2e66f58c16b0058b2ac982a5775b70aa91851
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp313-cp313-win_amd64.whl
| Download URL | mdpsolver-0.10.2-cp313-cp313-win_amd64.whl |
|---|---|
| Size | 628.3 kB |
| Tags | CPython 3.13 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
5ebaf133168c0ccbee6452736ed1021a6f538290c3f0dfd207c482887c0c85f9
|
|
BLAKE2b-256 checksum How to use checksums |
251da35749c3524662d55744704e1fd174d4ba24abb71ba70e2b01141efc28c7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdpsolver-0.10.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 271.5 kB |
| Tags | CPython 3.13 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
e0088e6f8cd0a070c46d1e070a77f1c1d640dbbe51f438377fa86104aef1e1f2
|
|
BLAKE2b-256 checksum How to use checksums |
81da1707e0b7a154eab15b82869f928172fb683efb3d76122473865937e30e5a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp313-cp313-macosx_14_0_arm64.whl
| Download URL | mdpsolver-0.10.2-cp313-cp313-macosx_14_0_arm64.whl |
|---|---|
| Size | 374.0 kB |
| Tags | CPython 3.13 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
322c788d085e2c92673116776bc5bb63c8fc7ff4faa4c663d6fed13bfade3547
|
|
BLAKE2b-256 checksum How to use checksums |
ae24a7821ac96142223c5b9ab8a71ae5988c777f253526abc99a6e5ef7783410
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp312-cp312-win_amd64.whl
| Download URL | mdpsolver-0.10.2-cp312-cp312-win_amd64.whl |
|---|---|
| Size | 628.3 kB |
| Tags | CPython 3.12 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
bc111a36f1d797780b7beff1be865525462a795933e11df74544ad47ddacc058
|
|
BLAKE2b-256 checksum How to use checksums |
9c14c95bae6f4f24598988a602aea92eb0ef3e8a89df1cc2aeed5df8101fbfc1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdpsolver-0.10.2-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 271.3 kB |
| Tags | CPython 3.12 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
66e76401074292573d6daf61d40c06c5803415fa9e40ba9cb1ed9945dbe4056e
|
|
BLAKE2b-256 checksum How to use checksums |
a04ca25ec5f5cf50446281b05d21974710e9e621573b4c30e1e88d0eab15c382
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp312-cp312-macosx_14_0_arm64.whl
| Download URL | mdpsolver-0.10.2-cp312-cp312-macosx_14_0_arm64.whl |
|---|---|
| Size | 373.9 kB |
| Tags | CPython 3.12 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
3af58475886aeb084cb5caa6df668c83838f2cd19ee0a6c6f93a50759cb219e2
|
|
BLAKE2b-256 checksum How to use checksums |
d6416ba694a5475e1091f27a2924b5d3050590f0d813490d13d3719ee3ffcf7a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp311-cp311-win_amd64.whl
| Download URL | mdpsolver-0.10.2-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 625.7 kB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
2c08a63565b22ef25423e791847684cb2a3c97bb119d7699fe4b0690e1b797a3
|
|
BLAKE2b-256 checksum How to use checksums |
027a8c65eb00f30f65286c0d0bb00feb28a76982510b44c043fb342accff6178
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdpsolver-0.10.2-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 270.4 kB |
| Tags | CPython 3.11 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
cafb065960ba9536149e3d1db692d5eaa5603e399cb0f481487d776c9200a024
|
|
BLAKE2b-256 checksum How to use checksums |
ca5ea36a7edd28fd5cf215ca352ba9389906ad55c476b615f9dfa28d5ae5aa26
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp311-cp311-macosx_14_0_arm64.whl
| Download URL | mdpsolver-0.10.2-cp311-cp311-macosx_14_0_arm64.whl |
|---|---|
| Size | 373.5 kB |
| Tags | CPython 3.11 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
f3364fe9b43f808f5720e8cae7b90f628f93a0ad70404bef77f3be5ea189e75f
|
|
BLAKE2b-256 checksum How to use checksums |
157b8e4911b9acd3b8b213254742d1b54465dd9ec18fa4bf4ba321f03af67a55
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp310-cp310-win_amd64.whl
| Download URL | mdpsolver-0.10.2-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 625.2 kB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
dce10bb4fec206a069f19f9701a0ba1bcd8d760f41a7eb6b8a19b7fdf180a431
|
|
BLAKE2b-256 checksum How to use checksums |
8ff58d99cef1cdf8125ca1612eb4f3144a0b93f811dac656f79bc9e66aa3718c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl
| Download URL | mdpsolver-0.10.2-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 269.0 kB |
| Tags | CPython 3.10 Linux glibc 2.24+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
4798e6149e0e6094ed47dd6847d1cad53bd9664d97e270f29319d95a798c859b
|
|
BLAKE2b-256 checksum How to use checksums |
858a411818ab6290e6f2aa7fcaf656d7187681fafefc3cdcff06f9792d1fa3be
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency logRelease files / mdpsolver-0.10.2-cp310-cp310-macosx_14_0_arm64.whl
| Download URL | mdpsolver-0.10.2-cp310-cp310-macosx_14_0_arm64.whl |
|---|---|
| Size | 372.0 kB |
| Tags | CPython 3.10 macOS 14.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
c8d929efbaaee4d9faf52bf8685c357c00a8cf1070c7396719616e974921336c
|
|
BLAKE2b-256 checksum How to use checksums |
78edb4f16a3f5fec1b02d400da24ddef82816e6a89569e0be683141e6bfaf703
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 9, 2026.
Transparency log
