Skip to main content

Proof Of Concept: Monotrail

This proof of concept shows how to use python packages without virtualenvs. It will install both python itself and your dependencies, given a requirement.txt or a pyproject.toml/poetry.lock in the directory. Install with pipx, pip or a binary from release or main

pipx install monotrail # or `pip install monotrail`
monotrail run python my_script.py

Every dependency is installed only once globally and hooked to your code. No venv directory, no explicit installation, no activate, no pyenv.

This is a proof of concept, so only most features are missing and will crash or produce nonsense. E.g. non-linux is badly tested, installation is awkward, error messages are suboptimal, only pypi is supported, startup is slow, only some requirement.txt syntax is supported, lockfiles from interactive mode aren't saved, some pkg_resources usage doesn't work, etc.

monotrail means to show you can use python without the traditional "installing packages in an environment". It also integrates python-build-standalone so you don't need to install python anymore.

It includes simple reimplementations of pipx and tox. There's also a pip package meant for notebooks, where you can interactively add packages at runtime, get an isolated package set per notebook and avoid version conflicts.

Usage

First download the binary and put it in PATH (e.g. via .local/bin):

wget -O ~/.local/bin/monotrail https://konstin.github.io/poc-monotrail/main/ubuntu/monotrail
chmod +x ~/.local/bin/monotrail

Make sure you have either a requirements.txt or pyproject.toml/poetry.lock:

monotrail run python my_script.py

Picking the python version and extras:

monotrail run -p 3.9 --extras fancy-pants python my_script.py --some-option

Running commands that used to be in .venv/bin:

monotrail run command pytest

There's also a python package with an entrypoint:

pip install monotrail
monotrail_python path/to/your/script.py

With jupyter notebooks

!pip install monotrail
import monotrail

monotrail.interactive(
    numpy="^1.21",
    pandas="^1"
)

Screenshot of the jupyter browser UI using monotrail

In google colab, you might want to import your git repository:

import monotrail

monotrail.from_git(
    "https://github.com/sokrypton/ColabFold", "63b42b8f5b5da418efecf6c4d11490a96595020d"
)

As pipx replacement:

monotrail ppipx --extras jupyter black .

As tox replacement:

monotrail run -p 3.8 -p 3.9 -p 3.10 command pytest

You can symlink monotrail to a file called python, python3 or python3.x and it'll work as python3.8 or the specified python version.

There is also a demo of the flat source layout, where you have the __init__.py directly in src instead of nesting src/srcery/__init__.py.

srcery
├── poetry.lock
├── pyproject.toml
└── src
    ├── __init__.py
    └── potion.py

Setting RUST_LOG=debug will give you details to track down bugs.

Background

monotrail first parses which python version you want (3.8 by default) and if not present downloads it from PyOxy. It doesn't run python as an executable but instead loads libpython.so and uses the C API.

Next, we search for a dependencies listing (poetry.lock or requirements.txt). If required we run poetry to resolve the dependencies (which we bootstrap through a pre-recorded poetry.lock for poetry itself). We install all missing packages to separate directories in .cache/monotrail and record all locations.

We initialize python and inject a custom PathFinder with everything and add it to sys.meta_path. When python searches where import something from, it goes through all the Finders in sys.meta_path until one returns a location. Ours knows the locations of the packages from the lockfile and python doesn't see anything else, so you can only load from the packages matching the lockfile.

Interactive mode does pretty much the same, except we skip the python installation and there's a check that the version of an already imported package didn't change.

Benchmarks (wheel installation)

One neat thing about venv-less installation is that we install every package version only once, so no more 3 different installations of pytorch. This takes a lot less disk space (even though clearing the cache is an unsolved problem) but most importantly it means that if you have used all required package versions once before "installation" is instantaneous. It also removes the need to recreate broken venvs.

By reimplementing wheel installation in rust, it also became a good bit faster. install-wheel-rs has a separate python interface so you can reuse it as a fast wheel installer on its own.

Installing a single large wheel (plotly)

$ rm -rf .venv-benchmark && virtualenv .venv-benchmark
$ VIRTUAL_ENV=.venv-benchmark hyperfine -p ".venv-benchmark/bin/pip uninstall -y plotly" \
  "target/release/monotrail venv-install test-data/popular-wheels/plotly-5.5.0-py2.py3-none-any.whl" \
  ".venv-benchmark/bin/pip install --no-deps test-data/popular-wheels/plotly-5.5.0-py2.py3-none-any.whl"
          
Benchmark #1: target/release/monotrail install test-data/popular-wheels/plotly-5.5.0-py2.py3-none-any.whl
  Time (mean ± σ):      5.797 s ±  0.069 s    [User: 3.796 s, System: 1.979 s]
  Range (min … max):    5.699 s …  5.906 s    10 runs
 
Benchmark #2: .venv-benchmark/bin/pip install --no-deps test-data/popular-wheels/plotly-5.5.0-py2.py3-none-any.whl
  Time (mean ± σ):      7.658 s ±  0.061 s    [User: 5.448 s, System: 2.085 s]
  Range (min … max):    7.598 s …  7.758 s    10 runs
 
Summary
  'target/release/monotrail install test-data/popular-wheels/plotly-5.5.0-py2.py3-none-any.whl' ran
    1.32 ± 0.02 times faster than '.venv-benchmark/bin/pip install --no-deps test-data/popular-wheels/plotly-5.5.0-py2.py3-none-any.whl'

A sample datascience stack (numpy, pandas, matplotlib)

test-data/poetry/data-science -E tqdm_feature
pip 22.0.4 from /home/konsti/monotrail/.venv-b/lib/python3.8/site-packages/pip (python 3.8)
Poetry version 1.1.13
Benchmark 1: .venv/bin/pip install -q -r requirements-benchmark.txt
  Time (mean ± σ):     11.745 s ±  1.159 s    [User: 9.637 s, System: 1.339 s]
  Range (min … max):   10.830 s … 13.048 s    3 runs
 
Benchmark 2: poetry install -q --no-root -E tqdm_feature
  Time (mean ± σ):     15.039 s ±  0.153 s    [User: 41.032 s, System: 5.934 s]
  Range (min … max):   14.894 s … 15.199 s    3 runs
 
Benchmark 3: monotrail poetry-install -E tqdm_feature
  Time (mean ± σ):      5.232 s ±  0.135 s    [User: 12.850 s, System: 2.334 s]
  Range (min … max):    5.089 s …  5.357 s    3 runs
 
Summary
  'monotrail poetry-install -E tqdm_feature' ran
    2.24 ± 0.23 times faster than '.venv/bin/pip install -q -r requirements-benchmark.txt'
    2.87 ± 0.08 times faster than 'poetry install -q --no-root -E tqdm_feature'

A mid-sized django project

test-data/poetry/mst -E import-json
pip 22.0.4 from /home/konsti/monotrail/.venv-b/lib/python3.8/site-packages/pip (python 3.8)
Poetry version 1.1.13
Benchmark 1: .venv/bin/pip install -q -r requirements-benchmark.txt
  Time (mean ± σ):     29.481 s ±  0.186 s    [User: 21.001 s, System: 3.313 s]
  Range (min … max):   29.331 s … 29.690 s    3 runs
 
Benchmark 2: poetry install -q --no-root -E import-json
  Time (mean ± σ):     70.291 s ±  1.366 s    [User: 355.966 s, System: 46.958 s]
  Range (min … max):   69.020 s … 71.735 s    3 runs
 
Benchmark 3: monotrail poetry-install -E import-json
  Time (mean ± σ):     10.714 s ±  1.423 s    [User: 43.583 s, System: 10.551 s]
  Range (min … max):    9.504 s … 12.282 s    3 runs
 
Summary
  'monotrail poetry-install -E import-json' ran
    2.75 ± 0.37 times faster than '.venv/bin/pip install -q -r requirements-benchmark.txt'
    6.56 ± 0.88 times faster than 'poetry install -q --no-root -E import-json'

Startup time

Hello world:

$ hyperfine --warmup 2 "python hello.py" "../target/release/monotrail run python hello.py"
Benchmark 1: python hello.py
  Time (mean ± σ):      17.3 ms ±   0.8 ms    [User: 14.5 ms, System: 2.9 ms]
  Range (min … max):    16.3 ms …  20.5 ms    159 runs
 
Benchmark 2: ../target/release/monotrail run python hello.py
  Time (mean ± σ):     218.4 ms ±   5.4 ms    [User: 161.6 ms, System: 55.9 ms]
  Range (min … max):   212.0 ms … 232.5 ms    13 runs
 
Summary
  'python hello.py' ran
   12.65 ± 0.64 times faster than '../target/release/monotrail run python hello.py'

Simplest numpy usage:

import sys
import numpy
print(
    f"hi from python {sys.version_info.major}.{sys.version_info.minor} and "
    f"numpy {numpy.__version__}"
)
$ hyperfine --warmup 2 "python numpy_version.py" "../target/release/monotrail run python numpy_version.py"
Benchmark 1: python numpy_version.py
  Time (mean ± σ):      99.8 ms ±   1.2 ms    [User: 127.3 ms, System: 171.7 ms]
  Range (min … max):    98.1 ms … 103.1 ms    29 runs
 
Benchmark 2: ../target/release/monotrail run python numpy_version.py
  Time (mean ± σ):     285.9 ms ±   1.8 ms    [User: 247.6 ms, System: 169.3 ms]
  Range (min … max):   283.4 ms … 289.9 ms    10 runs
 
Summary
  'python numpy_version.py' ran
    2.86 ± 0.04 times faster than '../target/release/monotrail run python numpy_version.py'

Dev setup

I use two venvs, the one I have activated is called .venv-b and another .venv where I install monotrail. Setup is

virtualenv .venv-b
. .venv-b/bin/activate
pip install -U maturin[zig] black pip pytest ipython
virtualenv .venv
# pytest runs the tests, jupyter nbconvert is for testing the notebook
.venv/bin/pip install pytest jupyter nbconvert

Metadata

Release files for monotrail 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for monotrail 0.1.0
File Size Uploaded Provenance Actions
monotrail-0.1.0.tar.gz 7.8 MB Without provenance Details for monotrail-0.1.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for monotrail 0.1.0
File Interpreter ABI Platform Actions
monotrail-0.1.0-cp37-abi3-win_amd64.whl CPython 3.7 abi3 Windows x86-64 Details for monotrail-0.1.0-cp37-abi3-win_amd64.whl
monotrail-0.1.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl CPython 3.7 abi3 Linux glibc 2.12+ x86-64 Details for monotrail-0.1.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
monotrail-0.1.0-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl CPython 3.7 abi3 macOS 10.9+ universal2 (ARM64, x86-64), macOS 10.9+ x86-64, macOS 11.0+ ARM64 Details for monotrail-0.1.0-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl

Total release size: 24.0 MB

Release files / monotrail-0.1.0.tar.gz

Download URL monotrail-0.1.0.tar.gz
Size 7.8 MB
Tags Source
SHA-256 checksum
How to use checksums
c19c9d11a951031e734f6558f1f477c42b4d391c509bb9aa350a632393d45ecc
BLAKE2b-256 checksum
How to use checksums
a7eac3402b61bbc161586ec85592e02121bdcdf5a65e96f76338a41b773ef734
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.13.2

Release files / monotrail-0.1.0-cp37-abi3-win_amd64.whl

Download URL monotrail-0.1.0-cp37-abi3-win_amd64.whl
Size 3.5 MB
Tags CPython 3.7 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
1d4034fb906f3f835e9633886016febed45eb8f434364116bf4eecdf8a82d46d
BLAKE2b-256 checksum
How to use checksums
543942e6ec358f0be7d8d4c6a1dbb74bebb66f715eb86c022d830327cd39e6bd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.13.2

Release files / monotrail-0.1.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

Download URL monotrail-0.1.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Size 4.7 MB
Tags CPython 3.7 Linux glibc 2.12+ x86-64 abi3
SHA-256 checksum
How to use checksums
29e27c8973cac7b81a7fb26c2cb0afa92f510e62385dcd7a7d965676d1d2b158
BLAKE2b-256 checksum
How to use checksums
d2160df005fe4343aa08f6bd1034ec56d2251cc79a530f61f0078c22a5c87636
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.13.2

Release files / monotrail-0.1.0-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl

Download URL monotrail-0.1.0-cp37-abi3-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Size 7.9 MB
Tags CPython 3.7 abi3 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
c462c59bc59b09f95ca4039afdf182a35401c6e572659d73e692119bbb3c08bb
BLAKE2b-256 checksum
How to use checksums
f822d8bb63f83afe7dfc5d0e73b004d129015acebfa234a61eafcf5f777b7a01
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/0.13.2

Security for monotrail 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

4 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page