Skip to main content

This project can be used to mirror the pypi index using the new warehouse api.

Project description

pypi-downloader

Description

This project can be used to mirror the pypi index using the new warehouse API.

This project consists of two scripts:

  1. a multithreaded version of the main script, pypi-downloader-mt.py, as command: pypi-downloader
  2. a helper script to get the current list of packages from the pypi index site currently located at: https://pypi.org/, as command: pypi-packages

Config file

If a config file is specified as a command line parameter, the config file uses the YAML format.

The config file consists of four sections:

  1. logging - Specifies a logging.dictConfig dictionary
  2. threads - Number of threads to use
  3. packages - List of packages to download, if no packages are specified, all packages are downloaded from the pypi index site
  4. blacklist - List of packages to not download

Note: For logging, this module uses the root logger only. Note: Values specified in the config file (threads and packages) can be overridden by values specified on the command line.

Config file examples

Default configuration

logging:
  version: 1
  formatters:
    simple:
      format: '[%(levelname)s]: %(message)s'
  handlers:
    console1:
      class: logging.StreamHandler
      level: ERROR
      formatter: simple
      stream: ext://sys.stderr
    console2:
      class: logging.StreamHandler
      level: DEBUG
      formatter: simple
      stream: ext://sys.stdout
  root:
    level: INFO
    stream: ext://sys.stdout
    handlers: [console1, console2]
threads: 1
packages:
blacklist:

Default configuration with packages and blacklist specified and non default thread count

logging:
  version: 1
  formatters:
    simple:
      format: '[%(levelname)s]: %(message)s'
  handlers:
    console1:
      class: logging.StreamHandler
      level: ERROR
      formatter: simple
      stream: ext://sys.stderr
    console2:
      class: logging.StreamHandler
      level: DEBUG
      formatter: simple
      stream: ext://sys.stdout
  root:
    level: INFO
    stream: ext://sys.stdout
    handlers: [console1, console2]
threads: 5
packages:
  - tox
  - mypy
blacklist:
  - pyyaml

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pypi-downloader, version 2.0.4
Filename, size File type Python version Upload date Hashes
Filename, size pypi_downloader-2.0.4-py3-none-any.whl (9.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pypi-downloader-2.0.4.tar.gz (9.0 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page