mxdev 5.1.0

Enable to work with Python projects containing lots of packages, of which you only want to develop some.

Mixed development source packages on top of stable constraints using pip

mxdev [mɪks dɛv] is a utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some.

It builds on top of the idea to have stable version constraints and then develop from a VCS on top of it.

As part of the above use-case sometimes versions of the stable constraints need an override with a different (i.e. newer) version. Other software follow the same idea are mr.developer for Python's zc.buildout or mrs-developer for NPM packages.

Star us on Github

Overview

mxdev procedure is:

1. Configuration is read,
2. Requirements and constraints (given in the configuration) are read.
3. Sources from VCS are fetched into a target directory,
4. Modified constraints (handled packages commented, overridden versions replaced) and requirements (handled packages as editable from sources) are written.

mxdev will not run pip for you!

Installation

pip install mxdev

mxdev >=4.0 needs pip version 23 at minimum to work properly

Quick Start

1. Create mx.ini configuration file:

[settings]
requirements-in = requirements.txt
requirements-out = requirements-mxdev.txt
constraints-out = constraints-mxdev.txt

# Custom variables for reuse
github = git+ssh://[e-mail]/

[mypackage]
url = ${settings:github}myorg/mypackage.git
branch = main
extras = test

2. Run mxdev to fetch sources and generate files:

mxdev

3. Install with pip using the generated files:

pip install -r requirements-mxdev.txt

For more examples see the example/ directory.

Configuration

Configuration is done in an INI file (default: mx.ini) using configparser.ExtendedInterpolation syntax.

Settings Section [settings]

The main section must be called [settings], even if kept empty.

I/O Settings

Option	Description	Default
requirements-in	Input requirements file (can be URL). Empty value = generate from INI only	requirements.txt
requirements-out	Output requirements with development sources as -e entries	requirements-mxdev.txt
constraints-out	Output constraints (developed packages commented out)	constraints-mxdev.txt

Behavior Settings

Option	Description	Default
default-target	Target directory for VCS checkouts	./sources
threads	Number of parallel threads for fetching sources	4
smart-threading	Process HTTPS packages serially to avoid overlapping credential prompts (see below)	True
offline	Skip all VCS and HTTP fetches; use cached HTTP content from .mxdev_cache/ (see below)	False
default-install-mode	Default install-mode for packages: editable, fixed, or skip (see below)	editable
default-update	Default update behavior: yes or no	yes
default-use	Default use behavior (when false, sources not checked out)	True

Smart Threading

When smart-threading is enabled (default), mxdev uses a two-phase approach to prevent credential prompts from overlapping:

1. Phase 1: HTTPS packages without pushurl are processed serially (one at a time) to ensure clean, visible credential prompts
2. Phase 2: Remaining packages (SSH, local, HTTPS with pushurl) are processed in parallel for speed

Optimization: HTTPS URLs with pushurl defined are assumed to be read-only/public and processed in parallel, since the pushurl indicates authenticated write access is separate.

This solves the problem where parallel git operations would cause multiple credential prompts to overlap, making it confusing which package needs credentials.

When to disable: Set smart-threading = false if you have git credential helpers configured (e.g., credential cache, credential store) and never see prompts.

Offline Mode and HTTP Caching

When offline mode is enabled (or via -o/--offline flag), mxdev operates without any network access:

1. HTTP Caching: HTTP-referenced requirements/constraints files are automatically cached in .mxdev_cache/ during online mode
2. Offline Usage: In offline mode, mxdev reads from the cache instead of fetching from the network
3. Cache Miss: If a referenced HTTP file is not in the cache, mxdev will error and prompt you to run in online mode first

Example workflow:

# First run in online mode to populate cache
mxdev

# Subsequent runs can be offline (e.g., on airplane, restricted network)
mxdev -o

# Cache persists across runs, enabling true offline development

Cache location: .mxdev_cache/ (automatically added to .gitignore)

When to use offline mode:

• Working without internet access (airplanes, restricted networks)
• Testing configuration changes without re-fetching
• Faster iterations when VCS sources are already checked out

Note: Offline mode tolerates missing source directories (logs warnings), while non-offline mode treats missing sources as fatal errors.

Package Overrides

version-overrides

Override package versions which are already defined in a dependent constraints file. I.e. an upstream constraints.txt contains already somefancypackage==2.0.3. Given that, for some reason (like with my further developed sources), we need version 3.0.0 of the above package.

Then in this section, this can be defined as:

[settings]
version-overrides =
    somefancypackage==3.0.0
    otherpackage==33.12.1

It is possible to add as many overrides as needed. When writing the constraints-out, the new version will be taken into account. If there is a source section defined for the same package, the source will be used and entries here are ignored.

Note: When using uv pip install the version overrides here are not needed, since it supports overrides natively. With uv it is recommended to create an overrides.txt file with the version overrides and use uv pip install --override overrides.txt [..] to install the packages.

ignores

Ignore packages that are already defined in a dependent constraints file. No new version will be provided. This is specifically handy if a package is going to be installed editable from local file system (like -e .), but was already pinned in an upstream constraints file.

This can be defined as:

[settings]
ignores =
    somefancypackage
    otherpackage

main-package

mxdev can handle one Python package as main package directly via ini config. If defined, it will be added as last entry in the resulting requirements out file.

This can be defined as:

[settings]
main-package = -e .[test]

If the main package is defined in a dependent constraint file, its name must be added to ignores.

Advanced Settings

include

Include one or more other INI files.

The included file is read before the main file, so the main file overrides included settings. Included files may include other files. Innermost inclusions are read first.

If an included file is an HTTP-URL, it is loaded from there.

If the included file is a relative path, it is loaded relative to the parent's directory or URL.

Default: empty

directory

mxdev provides a default setting containing the current working directory which can be used inside package or custom sections:

[sectionname]
param = ${settings:directory}/some/path

Custom Variables

Additionally, custom variables can be defined as key = value pair. Those can be referenced in other values as ${settings:key} and will be expanded there.

[settings]
github = git+ssh://[e-mail]/
gitlab = git+ssh://[e-mail]/

Package Source Sections

Sections other than [settings] can define:

• Package sources: [PACKAGENAME] - VCS sources to checkout and develop
• Hook configuration: [hookname-section] - Settings for mxdev extensions (see EXTENDING.md)

For package sources, the section name is the package name: [PACKAGENAME]

Basic Package Options

Option	Type	Description	Default
url	required	VCS checkout URL	—
vcs	optional	Version control system: git, fs, svn, gitsvn, hg, bzr, darcs	git
branch	optional	Branch name or tag to checkout	main
extras	optional	Comma-separated package extras (e.g., test,dev)	empty
subdirectory	optional	Path to Python package when not in repository root	empty
target	optional	Custom target directory (overrides default-target)	default-target
pushurl	optional	Writable URL(s) for pushes. Supports single URL or multiline list for pushing to multiple remotes. Not applied after initial checkout.	—

VCS Support Status:

• git (stable, tested)
• fs (stable, tested) - local directory pseudo-VCS
• svn, gitsvn, hg, bzr, darcs (unstable, tests need rewrite)

Installation Options

Option	Description	Default
install-mode	editable: Install with -e (development mode) fixed: Install without -e (production/Docker) skip: Only clone, don't install ⚠️ direct is deprecated, use editable	default-install-mode
use	When false, source is not checked out and version not overridden	default-use

Git-Specific Options

Option	Description	Default
depth	Git clone depth (shallow clone). Set GIT_CLONE_DEPTH=1 env var for global default	full clone
submodules	Submodule handling: always, checkout, recursive (see below)	always

Git Submodule Modes

• always (default): Git submodules will always be checked out, updated if already present
• checkout: Submodules only fetched during checkout, existing submodules stay untouched
• recursive: Fetches submodules recursively, results in git clone --recurse-submodules on checkout and submodule update --init --recursive on update

Multiple Push URLs

You can configure a package to push to multiple remotes (e.g., mirroring to GitHub and GitLab):

[my-package]
url = https://github.com/org/repo.git
pushurl =
    [e-mail]:org/repo.git
    [e-mail]:org/repo.git
    [e-mail]:org/repo.git

When you run git push in the checked-out repository, Git will push to all configured pushurls sequentially.

Note: Multiple pushurls only work with the git VCS type. This mirrors Git's native behavior where a remote can have multiple push URLs.

Usage

Run mxdev (for more options run mxdev --help).

Mxdev will

1. read the configuration from mx.ini,
2. fetch the packages defined in the config file and
3. write a requirements and constraints file.

Now, use the generated requirements and constraints files with i.e. pip install -r requirements-mxdev.txt.

Example Configuration

Example mx.ini

This looks like so:

[settings]
requirements-in = requirements.txt
requirements-out = requirements-mxdev.txt
constraints-out = constraints-mxdev.txt

version-overrides =
    baz.baaz==1.9.32

ignores =
    my.ignoredpackage

# custom variables
github = git+ssh://[e-mail]/
mygit = git+ssh://[e-mail]/

[foo.bar]
url = ${settings:github}orga/foo.bar.git
branch = fix99
extras = test,baz

[kup.fancyproject]
url = ${settings:mygit}customers/fancycorp/kup.fancyproject.git
branch = fix99
extras = test,baz

More Examples

For comprehensive examples demonstrating all features, see the example/ directory.

Real-World Examples

• "new" plone.org backend
• Conestack

Extending

The functionality of mxdev can be extended by hooks. This is useful to generate additional scripts or files or automate any other setup steps related to mxdev's domain.

See EXTENDING.md for complete documentation on creating mxdev extensions.

Rationale

Problem

There is a constraint file like -c constraints.txt with a package foo.bar with a version pin. Then it is not possible to install this package in a requirements file editable like -r requirements.txt with -e git+ssh://[e-mail]/orga/foo.bar.git@fix-99. Neither it is possible to override inherited version constraints with custom ones.

Idea

A pre-processor fetches (as this can be an URL) and expands all -c SOMEOTHER_FILE_OR_URL and -r SOMEOTHER_FILE_OR_URL files into one, filtering out all packages given in a configuration file. For each of those packages, a -e ... entry is generated instead and written to a new TARGET.txt. Same is true for version overrides: a new entry is written to the resulting constraints file while the original version is disabled. The configuration is read from a file mx.ini in ExtendedInterpolation INI syntax (YAML would be nice, but the package must have as less dependencies as possible to other packages).

Trivia

Mx (generally pronounced like mix [mɪks], or [məks] in the UK) is meant to be a gender-neutral alternative to the titles Mr. and Ms. but also associates with the word "mix".

Misc

The VCS-related code is taken from mr.developer. Thanks to Florian Schulze and Contributors.

---

Contributing

If you want to help with the development (improvement, update, bug-fixing, ...) of mxdev this is a great idea!

The code is located in the GitHub MXStack Organization / mxdev.

You can fork it, work on the project and create a pull request.

Maintainers are Jens Klein and the BlueDynamics Alliance developer team.

We appreciate any contribution! If you have an idea, found a bug, want to drop us a question, or a release is needed, please just file an issue at the mxdev issue tracker.

---

Changes

5.1.0

• Feature: Git repositories can now specify multiple push URLs using multiline syntax in the pushurl configuration option. This enables pushing to multiple remotes (e.g., GitHub + GitLab mirrors) automatically. Syntax follows the same multiline pattern as version-overrides and ignores. Example: pushurl = followed by indented URLs on separate lines. When git push is run in the checked-out repository, it will push to all configured pushurls sequentially, mirroring Git's native multi-pushurl behavior. Backward compatible with single pushurl strings. [jensens, 2025-11-03]
• Feature: Added --version command-line option to display the current mxdev version. The version is automatically derived from git tags via hatch-vcs during build. Example: mxdev --version outputs "mxdev 5.1.0" for releases or "mxdev 5.1.1.dev27+g62877d7" for development versions. [jensens, 2025-11-03]

5.0.2 (2025-10-23)

• Fix #70: HTTP-referenced requirements/constraints files are now properly cached and respected in offline mode. Previously, offline mode only skipped VCS operations but still fetched HTTP URLs. Now mxdev caches all HTTP content in .mxdev_cache/ during online mode and reuses it during offline mode, enabling true offline operation. This fixes the inconsistent behavior where -o/--offline didn't prevent all network activity. [jensens]
• Improvement: Enhanced help text for -n/--no-fetch, -f/--fetch-only, and -o/--offline command-line options to better explain their differences and when to use each one. [jensens]

5.0.1 (2025-10-23)

• Fix #65: Check source directories exist before writing to requirements-mxdev.txt. In offline mode: missing sources log WARNING and are written as comments (expected behavior). In non-offline mode: missing sources log ERROR and mxdev exits with RuntimeError (fatal error indicating checkout failure). This fixes mxmake two-stage installation workflow and prevents silent failures when sources fail to check out. [jensens]
• Fix: Configuration parsing no longer logs "Can not parse override:" errors when version-overrides is empty. Empty lines in version-overrides and ignores are now properly skipped during parsing. Also fixed bug where ignores lines were not properly stripped of whitespace. [jensens]
• Fix: Three tests that were accidentally marked as skipped during PR #66 merge are now fixed and passing: test_resolve_dependencies_simple_file (fixed assertion to check line contents), test_write_output_with_ignores (fixed to use read() for proper ignore processing), and test_write_relative_constraints_path_different_dirs (fixed to include constraints content). [jensens]
• Chore: Improved test coverage for main.py from 42% to 100%. Added comprehensive tests for the main() function covering all CLI argument combinations (--verbose, --silent, --offline, --threads, --no-fetch, --fetch-only), ensuring robust testing of the entry point and all code paths. [jensens]
• Chore: Updated test fixture data versions to resolve Dependabot security alerts. Updated urllib3 from 1.26.9 to 2.5.0 and requests from 2.28.0 to 2.32.4 in test data files. These are test fixtures only and were never actual dependencies or security risks. Resolves GitHub Dependabot alerts #1-7. [jensens]
• Performance: Smart threading now processes HTTPS URLs with pushurl in parallel. When a package defines both an HTTPS url and a pushurl (typically SSH), the HTTPS URL is assumed to be read-only/public and won't prompt for credentials, making it safe for parallel processing. This improves checkout performance for the common pattern of public repos with separate push URLs. [jensens]
• Fix: Add 'synchronize' event to pull_request workflow triggers. This ensures CI runs when PRs are updated with new commits (e.g., after rebasing or pushing new changes), not just when opened or reopened. [jensens]
• Chore: Optimize GitHub Actions to prevent duplicate workflow runs on pull requests. Restrict push trigger to only run on main branch, so PRs only trigger via pull_request event. This reduces CI resource usage by 50% for PR workflows. [jensens]
• Fix: process_line() now correctly comments out packages in override_keys and ignore_keys for both requirements and constraints files. Previously, these settings only applied to constraints files (variety="c"). Now they work for requirements files (variety="r") as well, with the message "-> mxdev disabled (version override)" for override_keys in requirements. [jensens]

5.0.0 (2025-10-22)

• Breaking: Drop support for Python 3.8 and 3.9. Minimum required version is now Python 3.10. [jensens]
• Breaking: Modernize type hints to use Python 3.10+ syntax (PEP 604: X | Y instead of Union[X, Y])
• Use built-in generic types (list, dict, tuple) instead of typing.List, typing.Dict, typing.Tuple [jensens]
• Chore: Replace black with ruff for faster linting and formatting. Configure ruff with line-length=120 and appropriate rule selections. Keep isort for import sorting with plone profile and force-alphabetical-sort. This modernizes the tooling stack for better Python 3.10+ support and faster CI runs. [jensens]
• Feature: #54: Add fixed install mode for non-editable installations to support production and Docker deployments. The new editable mode replaces direct as the default (same behavior, clearer naming). The direct mode is now deprecated but still works with a warning. Install modes: editable (with -e, for development), fixed (without -e, for production/Docker), skip (clone only). [jensens]
• Fix #35: Add smart-threading configuration option to prevent overlapping credential prompts when using HTTPS URLs. When enabled (default), HTTPS packages are processed serially first to ensure clean credential prompts, then other packages are processed in parallel for speed. Can be disabled with smart-threading = false if you have credential helpers configured. [jensens]
• Fix #34: The offline configuration setting and --offline CLI flag are now properly respected to prevent VCS fetch/update operations. Previously, setting offline = true in mx.ini or using the --offline CLI flag was ignored, and VCS operations still occurred. [jensens]
• Fix #46: Git tags in branch option are now correctly detected and handled during updates. Previously, updating from one tag to another failed because tags were incorrectly treated as branches. [jensens]
• Fix #22 and #25: Constraints file path in requirements-out is now correctly calculated as a relative path from the requirements file's directory. This allows requirements and constraints files to be in different directories. Previously, the path was written from the config file's perspective, causing pip to fail when looking for the constraints file. On Windows, paths are now normalized to use forward slashes for pip compatibility. [jensens]
• Fix #53: Per-package target setting now correctly overrides default-target when constructing checkout paths. [jensens]
• Fix #55: UnicodeEncodeError on Windows when logging emoji. The emoji is now conditionally displayed only when the console encoding supports it (UTF-8), avoiding errors on Windows cp1252 encoding. [jensens]

4.1.1 (2025-10-20)

• Modernize release method with hatchling. See RELEASE.md [jensens]
• Modernize tox setup. [jensens]
• Modernize Github workflows. [jensens]
• Enhance test coverage [jensens]
• Fix Makefile. [jensens]

4.1.0 (2025-06-03)

• Support environment variable GIT_CLONE_DEPTH for setting a default git depth for all checkouts. Useful for CI. [maurits]

• Fix #47: Do not add packages with capital names uncommented at the bottom ignore list when checked out. [petschki]

4.0.3 (2024-05-17)

• Fix #45: Packages with capital names do not get ignored when checked out. [jensens]

4.0.2 (2024-03-13)

• Fix #42: deprecated use of pkg_resoures to load entry points and parse requirements. This enables mxdev to work on Python 3.12, where pkg_resources is no longer installed by default in virtual_envs. [jensens]

4.0.1 (2024-03-01)

• Fix specifying out a revision (#40) [pbauer]

4.0.0 (2024-02-28)

• Breaking: Remove --pre on sources from generated requirements-mxdev.txt. Usually it is not needed any longer, at least withy pip 23.x. This is a breaking change if you rely on the --pre option being present in the generated file. Now the --pre option should be added to pip install when the generated file is used. This change enables the use of the generated file with the alternative pip replacement uv. [jensens]

• Breaking: Drop official support for Python 3.7 (it is end of life). [jensens]

• Document mx.ini sections vcs setting. [jensens]

3.1.0 (2023-12-10)

• Feature: Provide directory default setting [rnix]
• Feature: Include other INI config files [jensens]

3.0.0 (2023-05-08)

• Removed leftover print [jensens]

3.0.0b3 (2023-04-23)

• Fix usage of --install-option='pre' and use --pre option in requirements files instead. The install options are deprecated in pip 23 which Plone switched to recently. More info: https://github.com/pypa/pip/issues/11358 https://discuss.python.org/t/passing-command-line-arguments-to-pip-install-after-install-options-deprecation/22981/6 [thet, fredvd]

• Fix reading sections from the config parser without defaults if the section contains a setting that also exists as default. [rnix]

• Do not write constraints out to the file if no constraints are defined. [rnix]

• Add the main-package option to the settings. [rnix]

3.0.0b2 (2023-02-07)

• In this package, use pyproject.toml and markdown for README et al. [jensens]

• Add use option to sources, and default-use to the settings. default-use is true by default. When false, the source is not checked out, and the version for this package is not overridden. [maurits]

3.0.0b1 (2022-11-21)

• Do not use libvcs, but recycled and updated (type hints, tests) mr.developer VCS code. Code for GIT is tested well, code for SVN, Mercurial, Bazaar and DARCS needs contributors with knowledge in this area. Additional options, like pushurl, ... (see README) were added. pip style VCS URLs are not supported any longer. [jensens, rnix, zworkb]

• Config parser options are now considered case-sensitive. [rnix]

• Do not fail mxdev run if requirements.txt is missing. [rnix]

• Add flag to only fetch repositories and skip generating files. [rnix]

• Add flag to skip fetching of repositories. [rnix]

• Add support for custom hooks. [rnix]

• Rename sources.ini to mx.ini in the documentation. [rnix]

• Introduce state object and pass it to read/fetch/write. State object contains all required runtime data. [rnix]

2.0.0 (2022-01-31)

• Depend on pip 22, where interdependency mode is no longer needed. Remove all interdependency-related code. [jensens]

• Better error message if the requirements-in file does not exist. [jensens]

• Better last message with the full pip command. [jensens]

• Allow empty requirements-in configuration. [jensens]

1.1.0 (2021-12-29)

• Feature: Ignore existing constraints. New setting ignores with a list of packages (one per line) to ignore constraints without providing a version. [jensens]

1.0.1 (2021-12-21)

• Fix: If a developed package depends on another developed package the dependent package was ignored sometimes (!?). Instead, the last release was taken. Solution: Install it with the --pre option in order to allow the other non-final/in-development release. [jensens]

1.0.0 (2021-12-12)

• Defaults for "outfiles" are *-mxdev.txt now. [jensens]

1.0.0b4 (2021-12-07)

• Fix interdependency mode. [jensens]

1.0.0b3 (2021-12-07)

• Fix: Do not apply override disabling on requirements. [jensens]

1.0.0b2 (2021-12-07)

• Add feature: version overrides. [jensens]

1.0.0b1 (2021-12-04)

• Add -s or --silent option. [jensens]

• Beautified output. [jensens]

• Fixed missing CR if *.txt does not end with a newline. [jensens]

1.0.0a9 (2021-12-01)

• Added auto correction for pip URLs, so that GitHub or GitLab URLs can be used as copied in sources.ini. [zworkb]

1.0.0a8 (2021-11-30)

• Added interdependency handling to avoid manual dependency order resolution. [jensens, gogobd]

• Added skip mode to exclude packages from installation (clone/update only). [jensens, gogobd]

• Removed position feature. [jensens, gogobd]

1.0.0a7 (2021-11-30)

• Removed Workaround for libvcs and depend on libvcs>=0.10.1. [jensens]

1.0.0a6 (2021-11-30)

• Workaround for libvcs bug https://github.com/vcs-python/libvcs/issues/295 [jensens, gogobd]

1.0.0a5 (2021-11-30)

• Workaround for libvcs bug https://github.com/vcs-python/libvcs/issues/293 [jensens, gogobd]

1.0.0a4 (2021-11-29)

• Fix: editable can be configured to be processed before or after initial requirements. [jensens]

1.0.0a3 (2021-11-23)

• Fix #1: Re-run of pip vanishes committed changes [jensens]

1.0.0a2 (2021-11-21)

• Fix/simplify packaging. [jensens]

• Implement subdirectory editable install [jensens]

• Implement package extras [jensens]

1.0.0a1 (2021-11-21)

• Initial work. [jensens]

---

License

Copyright (c) 2022-2025, mxstack Contributors

All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.