devpi-server 6.3.0

devpi-server: reliable private and pypi.org caching server

devpi-server: server for private package indexes and PyPI caching

PyPI cache

You can point pip or easy_install to the root/pypi/+simple/ index, serving as a transparent cache for pypi-hosted packages.

User specific indexes

Each user (which can represent a person, project or team) can have multiple indexes and upload packages and docs via standard twine or setup.py invocations. Users and indexes can be manipulated through devpi-client and a RESTful HTTP API.

Index inheritance

Each index can be configured to merge in other indexes so that it serves both its uploads and all releases from other index(es). For example, an index using root/pypi as a parent is a good place to test out a release candidate before you push it to PyPI.

Good defaults and easy deployment

Get started easily and create a permanent devpi-server deployment including pre-configured templates for nginx and process managers.

Separate tool for Packaging/Testing activities

The complementary devpi-client tool helps to manage users, indexes, logins and typical setup.py-based upload and installation workflows.

See https://doc.devpi.net on how to get started and further documentation.

Support

If you find a bug, use the issue tracker at Github.

For general questions use the #devpi IRC channel on freenode.net or the [e-mail] mailing list.

For support contracts and paid help contact merlinux.eu.

Changelog

6.3.0 (2021-11-19)

Features

• Use aiohttp (asyncio) for fetching release links from mirrors to return stale links immediately in case of a timeout, but update the database in the background for the next request.

Bug Fixes

• fix #853: prevent duplicate mirror indexes in sro method when there are multiple inheritance chains to the same mirror

• fix #860: don’t check for existing files and validate them during mirror links update, the operation is way to expensive and there is a low possibility for errors.

• Add missing lazy package dependency. Previously this was only a transitive dependency coming from the devpi-common package.

6.2.0 (2021-08-12)

Bug Fixes

• Optimized some database access patterns. A new index is added to the database on first startup. For large databases that can take a while.

• Improved performance of loads from database.

• Optimized memory and cache use for database access.

• Use frozenset for project name cache of mirror indexes. This mitigates memory fragmentation on some Linux distributions.

6.1.0 (2021-07-11)

Deprecations and Removals

• Renamed ‘pypi_submit’ permission to ‘upload’. The old permission still works, but will be removed in a later major release.

Features

• Allow patching index with same json layout as the output of fetching json for an index.

• Allow user and index URLs to work with a trailing slash.

Bug Fixes

• fix #631: race condition in fetching project links from mirrors.

6.0.1 (2021-06-23)

Bug Fixes

• fix #843: add explicit ruamel.yaml dependency declaration after strictyaml vendored it.

6.0.0 (2021-05-16)

Deprecations and Removals

• Remove deprecated command line options which were replaced by separate scripts.

• Dropped support for Python 2.7, 3.4 and 3.5. Python 3.x versions will be supported until their EOL (see https://devguide.python.org/#status-of-python-branches). After that, any release might break support for EOLed versions.

• Removed deprecated --start, --stop and --status options.

Features

• fix #140: support force flag for deletion on non-volatile indexes.

• fix #725: new option mirror_whitelist_inheritance for indexes. The union setting is the old behaviour and used for existing indexes to not break existing installations. With it the whitelist of each index in the inheritance order is merged into the current whitelist. This could lead to unexpected whitelisting. The new intersection setting is used for all new indexes and it intersects the whitelist at each step in the inheritance order which is more secure and never causes unexpected whitelisting.

• fix #792: support data-yanked attribute from PEP 592 for mirror indexes.

• fix #827: add --listen option corresponding to listen kwarg of waitress server.

• Replicas download files asynchronously from the metadata and will do so with multiple parallel requests. This means the metadata will be in sync faster and downloads will process quicker. Missing files will be downloaded on demand if they haven’t been fetched yet. The new --file-replication-threads option allows controlling the amount of parallel downloads. Event processing waits until files for that serial are available. Since newest files are downloaded first, event processing might wait until all files are downloaded.

• Add devpi-gen-secret script to generate a file with a random secret and proper permissions.

• Installers will get simple results directly instead of a redirect when an index is used without /+simple or without a trailing slash.

• Much faster mirror project names parsing. For PyPI the speedup can be about 30x.

• Do some validity and security checks on the secret provided by --secretfile.

• The server secret for token signing is now derived via argon2 from the data provided by --secretfile. Existing login tokens are invalidated by this.

• Add --trusted-proxy, --trusted-proxy-count and --trusted-proxy-headers to support proxy headers with waitress.

• The user creation and modification time is now stored. Adding or removing an index doesn’t count as a modification.

Bug Fixes

• fix #210: the original fix was incomplete and the test for it was subtly wrong.

• fix #451: packages not on mirror_whitelist no longer query the mirror

• fix #680: indexes with multiple mirror bases now work correctly with default secure whitelist settings.

• Handle cases where the Content-Type header from a mirror can be an empty string.

Other Changes

• The X-Outside-URL header now takes precedence over the --outside-url option. This allows the option to be the fallback when there is no proxy in front, instead of overwriting the header.

• Warning! Once you used 6.0.0 with a replica you have to check that all files have been downloaded with devpi-fsck before attempting to downgrade to 5.x.y, as those older versions have no mechanism to re-download those.

• Add new devpiserver_auth_request hook and deprecate devpiserver_auth_user hook.

• Require pyramid>=2.

• Use secrets.token_bytes instead of os.urandom for salts and server secrets.

• Replicas need to use the same secret as the master for the --secretfile option to be able to authenticate with the master.

• Replicas will no longer proxy to the master to determine the authentication status now that the secret must be shared between master and replicas.

• When using --restrict-modify those users can’t delete their own user object to prevent lockout.

• The secret file must be user accessible only, devpi-server will not start if it is not.