This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!
Project Description

Python tools to help with the development & deployment of company-private Python packages.

It was developped to use Sonatype Nexus as a private Pypi mirror (until it supports this natively), but should be adaptable to any repository supporting artifact upload with HTTP.

It is composed of 2 distincts :

  • nexus_uploader : a simple module to be used as a setup_requires entry in setup.py, in order to easily upload Python packages onto a Nexus.
  • pyRequirements2nexus : a CLI tool to convert standard Python requirements.txt into nexus-requirements.txt files, made of URLs pointing to installable packages that pyRequirements2nexus mirrored on your Nexus.

Features

nexus_uploader features

  • easy integration in setup.py
  • HTTP BasicAuth to connect to Sonatype Nexus

pyRequirements2nexus features

  • full dependency resolution of Python packages, working with both Pypi public ones & private ones on a Nexus
  • to install packages, the end machine only require a connexion to the Nexus host, not to the Internet
  • support -e editable packages in requirements.txt
  • support package URL fallbacks as comments in requirements.txt
  • a list of packages already included in the base environment can be specified (e.g. if you are using Anaconda), so that they will be excluded from the final nexus-requirements.txt

Limitations

  • only support == version locking in requirements.txt (not >=). This is intentional, to ensure package versions do not change unexpectedly.

How to upload home-made Python packages to my Nexus ?

Here is a handy recipe you can put in your setup.py :

setup(
    ...
    setup_requires=['nexus_uploader']
)

Usage:

$ python setup.py sdist nexus_upload --repository http://nexus/content/repositories/Snapshots/com/vsct/my_project --username $REPOSITORY_USER --password $REPOSITORY_PASSWORD

Important note

Contrary to requirements.txt files, there is no way to include URLs to dependencies in setup.py install_requires (nor setup_requires). To work around this limitation, pyrequirements2nexus takes advantage of the deprecated “dependency_links” mechanism. Using them, it is able to discover the full chain of Python packages dependencies, both from Pypi AND from your local Nexus.

To easily generate the “dependency_links.txt” in your Nexus-hosted private package out of your requirements.txt, use the following recipe in your its setup.py :

with open(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'requirements.txt')) as requirements_file:
    requirements = requirements_file.readlines()
    dependency_links = [req for req in requirements if req.startswith('http')]
    install_requires = [req for req in requirements if not req.startswith('http')]

setup(
    ...
    install_requires=install_requires,
    dependency_links=dependency_links,
)

Because “dependency_links” are not supported since pip 1.6, they will NOT be installed by pip install normal recursive dependency-retrieval algorithm. You should always use the flat nexus-requirements.txt with pip install.

pyRequirements2nexus usage

pip install nexus_uploader
pyRequirements2nexus --help

Also take a look at jenkins-install-python-requirements.sh for an example of how we use it on our Jenkins.

Installation of nexus-requirements.txt on an end machine

pip install --user --no-index --no-deps --no-cache-dir --upgrade --requirement nexus-requirements.txt

Supported requirements.txt format

http://nexus/content/repositories/repo_id/my/project/group/mypkgname/1.0/mypkgname-1.0-py2.py3-none-any.tar.gz
nose==1.3.7   # -> transformed into an URL like this: http://nexus/content/repositories/repo_id/my/project/group/...
-e ../my/relative/path  # http://nexus/content/repositories/...fallback_url...

Contributing

pre-commit hooks installation:

pip install -r dev-requirements
pre-commit install

Unit tests:

PYTHONPATH=. py.test tests/

Smoke tests using Pypi:

PYTHONPATH=. ipython3 --pdb tests/smoke_test_extract_classifier_and_extension.py 200

FAQ

pip install - Download error on https://pypi.python.org / Couldn’t find index page for

The stack trace :

Collecting http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (from -r scripts/requirements.pip (line 12))
  Downloading http://nexus/content/repositories/pip/com/vsct/pip/jsonschema/2.5.1/jsonschema-2.5.1-py2.py3-none-any.tar.gz (50kB)
    Complete output from command python setup.py egg_info:
    Download error on https://pypi.python.org/simple/vcversioner/: [Errno -2] Name or service not known -- Some packages may not be found!
    Couldn't find index page for 'vcversioner' (maybe misspelled?)
    Download error on https://pypi.python.org/simple/: [Errno -2] Name or service not known -- Some packages may not be found!
    No local packages or download links found for vcversioner

Explanation : https://github.com/Julian/jsonschema/issues/276

Solution :

$ cat <<EOF > ~/.pydistutils.cfg
[easy_install]
allow_hosts = nexus
find_links = http://nexus/content/repositories/pip/com/vsct/pip/vcversioner/2.14.0.0/
EOF

How to generate a “–default-packages” file out of an Anaconda .sh installer

grep -aF 'extract_dist ' Anaconda3-2.4.1-Linux-x86_64.sh \
    | perl -p -e 's/extract_dist (.+?[0-9])[^.]*$/\1\n/;' -e 's/^(.+)-(.+)$/\1 == \2/;' \
    | grep -vE '^(_cache|_license|anaconda|python) ' > anaconda3-2.4.1_included_packages.txt

Tip for easily removing packages from your nexus

pip install --user repositorytools
export REPOSITORY_USER=...
export REPOSITORY_PASSWORD=
artifact delete http://nexus/content/repositories/pip/com/vsct/pip/ultrajson/1.35/ultrajson-1.35-macosx-10.6-intel.tar.gz

ToDo

  • Python 2 support
  • detect if package releases on Pypi require gcc compilation (are they using setuptools/distutils Extension in setup.py ?)
  • classifier-based selection of Python packages
  • add support for md5 & sha1 upload/checks
Release History

Release History

1.0.8

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.7

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.6

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.5

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.4

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
nexus_uploader-1.0.8.tar.gz (14.9 kB) Copy SHA256 Checksum SHA256 Source Dec 13, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting