Private Python Package Index
Pyshop is a private repository for python packages.
The aim is to split private projects in distinct private packages and keep a setup.py clean and working, by declaring all dependencies, exactly as public packages on PyPI.
Pyshop also proxies and caches packages from PyPI safely using SSL and checking server certificate.
Pyshop uses clear and simple ACLs to manage privileges:
an installer group that can only download release files,
a developer group that can download and upload release files and browse the website,
an admin group that has developer privileges and accounts management.
Since pyshop is intended to host private packages, every user, including pip, must be authenticated by login and password.
Using A virtualenv with Python 3
$ cd /srv $ sudo mkdir pyshop $ sudo chown $(whoami) pyshop $ cd pyshop $ pyvenv . (pyshop)$ source bin/activate (pyshop)$ pip install "pyshop[waitress]" (pyshop)$ cp pyshop.sample.ini pyshop.ini (pyshop)$ vim pyshop.ini # change at least the pyshop.cookie_key setting (pyshop)$ pyshop_setup pyshop.ini # Create the database (pyshop)$ pserve pyshop.ini # start pyshop CTRL+C to stop
You should edit the pyshop.ini file in order to configure the pyshop.cookie_key and the host:port that hosts the service. When the server is running visit the website, http://localhost:8000/ by default, to check everything is fine.
For production usage, you should create accounts with the developer group. Visit http://localhost:8000/pyshop/user with the admin account to create accounts.
You also should also use an https reverse proxy. Python packaging core uses HTTP basic authentication: it sends user/password in clear.
The pythop.sample.ini file use waitress as the default WSGI server, but, if you are familiar with another WSGI server that support paste format, you could use it.
Daemonize with systemd on linux
Pyramid 1.8 has removed deamonized options you have to use a process manager.
Here is a simple way to daemonise it undex linux that use systemd
(pyshop)$ curl -o pyshop.service https://raw.githubusercontent.com/mardiros/pyshop/master/pyshop.sample.service (pyshop)$ sudo mv pyshop.service /etc/systemd/system/pyshop.service
Pyramid 1.8 has removed deamonized options: http://docs.pylonsproject.org/projects/pyramid/en/latest/whatsnew-1.8.html#backwards-incompatibilities
Currently, there is an image of pyshop used for development purpose, it support both MySQL and PostgreSQL. The PostgreSQL integration is fully operation, you can run a new Pyshop install using docker-compose, with the command:
docker-compose up pgpyshop
It will create the database with the default pyshop users:
privileged user: login admin, password: changeme
unprivileged user: login pip, password changeme
If you want to use a different orchestrator, you have to link the postgresql container to Pyshop container with the name postgresql.localdomain
The MySQL support does not automate the database setup right now.
The official Docker image of Pyshop is available here on the Docker Hub:
Configuring your environment
Here are all configuration files you will need to modify for usual python tools to use your newly deployed private repository.
Configuration used by pip. This is a user file, you can set a developer or the generic pip account.
[global] # when mirroring a package, pyshop retrieves information from PyPI and # stores it in its database. Be patient, it is not so long. default-timeout = 120 timeout = 120 [install] index-url = http://pip:changeme@localhost:8000/simple/ [search] index = http://pip:changeme@localhost:8000/pypi
setup.cfg and pydistutils.cfg
setup.cfg and pydistutils.cfg are used when running python setup.py develop to install your package or when using easy_install. You should use a generic account with installer privileges only, shared by all developers.
This setting can be set per project or in user $HOME (see setuptools documentation for details)
[easy_install] index-url = http://pip:changeme@localhost:8000/simple/
This should work now:
python setup.py develop
Configuration used by setuptools to upload files. All developers should have this configuration in their $HOME to upload packages.
[distutils] index-servers = pyshop [pyshop] username: admin # or create an account in pyshop admin interface password: changeme repository: http://localhost:8000/simple/
This should work now:
python setup.py sdist upload -v -r pyshop
1.3.0 - Released on 2017-06-29
Handle package with hyphen, undescore and dot properly.
Implement xmlrpc api for pip search for python 3 and by default.
1.2.5 - Released on 2017-06-03
version used to build 1.2.4 was too old and build a broken package.
1.2.4 - Released on 2017-05-30
Fix for Python 3
Update README / install process for pyramid>=1.8 default to python 3
1.2.3 - Released on 2016-06-22
For ldap accounts, do not copy the ldap password into local db
Use https for XML-RPC calls
1.2.2 - Released on 2015-02-16
Update pyramid dependencies
1.2.1 - Released on 2015-02-15
Update requests dependencies
1.2.0 - Released on 2015-02-15
Implement roles edition
Filter package by names, if the input is not a classifier
Add purge option to packages (Chris Powell)
Add remove file from disk (Chris Powell)
Fix XML-RPC view
1.1.1 - Released on 2014-11-04
Fix packaging (DeaconDesperado)
1.1.0 - Released on 2014-10-31
Add support of MySQL
Use version sanization regex matching PEP386/440.
Fix invalid commit call in LDAP user creation.
Do not fetch user from database again while creating it from LDAP information, it will return the currently being created user due to autoflush and if other thread was about to do the same, it would be protected from doing anything wrong by the SQL transaction anyway.
Consider LDAP created users as local as well.
Fix upload of local package with waitress
Fix creation of the repository directory when creating a local package
Fix pyshop_setup command with Python 3
Add compatibility with python 3.3 and 3.4
Add ldap support (mgoacolou)
Replace scss by bootstrap (mgoacolou)
Authorize developper to upload a patched version of a mirrored package without any restriction. Some mirrored package may have bugs that are critical for you, and it’s better to get a mirrored package than rely on an external sources.
Fix support of SQLAlchemy 0.9.x
Fix packaging (missing LICENSE)
Fix postgresql admin account pages
Add support for posgresql
Remove some links in the simple route.
Don’t display home pages
Don’t display download links in case there is archive available. (That fix some install for south for example)
Fix mirroring of package when the case and underscore are not “correct”
Fix mirroring of external links browsed (in case it’s not a package)
Fix tests for python 2.6 (unittest2 required and be installed manually)
Fix pyramid 1.2 compatibility
Rename command pyshop_install to python_setup
Give the possibility to use the prefix_route via the settings ‘pyshop.prefix_route’
Give the possibility to disable xmlrpc servive via ‘pyshop.enable_xmlrpc’
Tolerant underscore/hyphen usage in package name
Tolerant with trailing slash in urls
Fix unit tests
add key “pyshop.mirror.cache.ttl” in config file. This settings allow to set the cache time of a package before refresh it on pypi.
add a button in the web interface to force the refresh of mirrored package. This permit to force reset the ttl of a package on the web interface.
add key “pyshop.upload.rewrite_filename” in config file. This settings disable the rename of package file uploaded on the server.
change “satanize” to “sanitize” keys in config file. Modify this settings in your config file after a migration.
add key “pyshop.mirror.wheelify” in config file. This settings require users to use recent version of pip and setuptools virtualenv 1.10.1 is OK. This is experimental. User and Proxy Server must run the same OS on the same architecture to use that feature.
Use requests for xmlrpc queries too. - validate certificate if https is used - unified proxies configuration (use environment vars)
SAWarning/DeprecationWarning removed - Fix mirroring link for external files
Mirror the download_url of the release file
Handle hiphen and underscore [fizyk]
Enhance mimetime handling [fizyk]
Bugfixes and DeprecationWarning removed
Fix package version comparison
Handle bdist_wheel format
To handle the wheel format, (some package like Twisted 13 use it), for previous install, you must run a migration script like this.
$ pyshop_migrate development.ini 0.7.5
The sqlite database file will be altered, YOU MAY backup it before run the script.
Fix local package usage (broken since 0.7.1) [fizyk]
Remove all certificates and extra handling for PyPI validation as PyPI now uses a certificate that can be validated without these. [disko]
Remove unused certificates for pypi validation
Securize download from pypi by forcing https and validate certificate pypi.python.org certificate chain is embed in the pyshop package
Fix package order on web page
Sanitize version number on upload. This is configurable with settings pyshop.upload.satanize and pyshop.upload.satanize.regex
Settings pyshop.satanize and pyshop.satanize.regex have been renamed to pyshop.mirror.satanize and pyshop.mirror.satanize.regex
Fix first connection of the web application
Fix the usage of http proxy (forcing request version)
Add Link to display all release versions
Fix ugly version number sorting
Fix release file upgrade (allow developper to override release file)
Rename user views to account
Add view to let the connected user to update his account
The setting pyshop.satanize.reg has been renamed to pyshop.satanize.regex
The setting cookie_key has been renamed to pyshop.cookie_key
Fix bug on package upload. don’t close the stream while writing it.
Add basic tests on packages view
work with pip, setuptools
secure access with login/password
tests for python 2.7 only
compatible with python 2.6