strip out unnecessary pip packages from requirements
TLDR: requirements without unnecessary
For the purpose of this exercise, an unnecessary pip package is any package that is not being imported by YOUR own Python code. Unless you've provided a configuration override stating that you want it.
Simplest example: initialize, scan and build in current directory
$ pipstripper --init --build pip-stripper configuration generated @ sample/tst.seedworkdir01/pip-stripper.yaml build phase - generating requirements at: sample/tst.seedworkdir01/requirements.dev.txt sample/tst.seedworkdir01/requirements.txt $cat requirements.txt Django==2.2 Jinja2==2.10.1 celery==4.3.0 cx-Oracle==6.4.1 psycopg2==2.8.2 python-dateutil==2.8.0 $ cat requirements.dev.txt pyquery==1.4.0
How it works
Lots of your packages may not be needed. Let's say you've installed
A linter and autoformatter has no need to be on a server:
pip-stripper most likely won't find
import black anywhere, so it will not put it in
usage: pipstripper [-h] [--config CONFIG] [--noscan] [--build] [--init] [--workdir WORKDIR] [--verbose] optional arguments: -h, --help show this help message and exit --config CONFIG config file. if not provided will look for pip- stripper.yaml in --workdir, current directory --noscan don't scan to classify packages. build phase will re-use existing pip-stripper.scan.yaml. [False]. --build read pip-stripper.scan.yaml to create requirements.prod/dev.txt [False] --init initialize the config file (as pip-stripper.yaml) if it doesn't exist --workdir WORKDIR work directory [defaults to config file's value or current directory] --verbose verbose mode. adds extra zzz_debug: entry to pip- stripper.scan.yaml [False]
Initialization (defaults to False)
The first option
--init will create pip-stripper.yaml, the configuration file for pip-stripper.
This is the only file you should edit manually!!!
Scan phase (will run unless you specify
This will scan your Python source files in
--workdir and use it to create pip-stripper.scan.yaml.
This is the file that contains instructions for the build phase.
Don't edit pip-stripper.scan.yaml!
- adjust the configuration in pip-stripper.yaml
- re-run the scan
Scanning also creates 2 work files,
tmp.pip-stripper.imports.rpt, tracking pip packages and its best guesses at python imports, respectively.
Build (defaults to False)
--build takes what it finds in pip-stripper.scan.yaml and uses it to populate
If those requirements files don't suit you, you may need to edit pip-stripper.yaml.
This allows you to specify:
- specify which packages are just workstation-level and shouldn't go into requirements.
- hardcode packages that need to go into either.
- Associating your source directories to either
You may have to enter pip to python import alias names manually (alias matching is something that needs work).
hardcoded_aliases: PyYAML: yaml
Hardcoding package to requirement mapping:
psycopg2 is typically never really imported in a Django or SQLAlchemy context, but rather derived from the configuration, you need to specify it yourself as below. Same thing with
django-redis-cache which is configured in django's
settings.py as package path rather than an import.
ClassifierPip: #the following are used to "hardcode" package names to given buckets. buckets: prod: - psycopg2 - django-redis-cache tests: - nose - pytest workstation: # that's a workstation only package, so it's held back - black
Associating Python directories to requirements:
This is a typical regex-based configuration telling which buckets the directories count as:
prod is the default outcome. First match wins, and
tests is the only one needed.
ClassifierImport: regex_dirs: workstation:  dev:  tests: - "/tests/" prod:  default_bucket: "prod"
case 1: hardcoding
pip freeze line like
First, it looks for a matching entry in
ClassifierPip:buckets in pip-stripper.yaml, (basically a hardcoded decision by the user of where to put it).
ClassifierPip: buckets: workstation: - black prod: - psycopg2
This will result in
psycopg2==2.7.7 going into requirements.prod.txt (when needed, requirements lines are always copied from the
pip freeze output).
case 2: import classification.
grep-ing the Python code found this line:
./tests/helper_pyquery.py:57: from pyquery import PyQuery
Each file path is run against the regex specified by
ClassifierImport:regex_dirs, so pyquery ends up in the
ClassifierImport: regex_dirs: tests: - "/tests/" prod:  default_bucket: "prod"
--build pass looks at where buckets get mapped:
Builder: req_mapper: dev: buckets: - dev - tests prod: buckets: - prod
case 3 multiple imports
./tests/helper_pyquery.py:57: from pyquery import PyQuery ./myserver/foobar.py:22: from pyquery import PyQuery
pyquery is put in
prod buckets. But also in
myserver did not match any
ClassifierImport:regex_dirs, meaning that
default_bucket: "prod" was used.
enter bucket precedence
ClassifierPip: bucket_precedence: - prod - tests - dev - workstation
pyquery ends up only in
case 4. no import match was found and nothing was hardcoded.
will get left out of requirements. Which is not to say that it won't end up
pip installed on your server if it is a dependency of some other package (
pipdeptree can help you there).
Babel==2.6.0 - pytz [required: >=0a, installed: 2018.9]
Build Phase - the result of the
--scan phase gets put into pip-stripper.scan.yaml:
Notice our friend
black? We've explicitly classified it as workstation, so the scan didn't label it as unknown.
pips: buckets: dev: .... tests: - pyquery prod: - psycopg2 unknown: - Babel workstation: - black
Look at the end for the
warnings: section. In this case,
repr was used with Python 2.7 but isn't necessary with Python 3, so I won't worry about it. A typical reason for a missing import is that automatic aliasing to link the pip name and import name didn't work.
warnings: - missing import:repr
And that's it. The outcome?
my raw pip freeze weighs in at 158 packages:
$ wc -l requirements.freeze_raw.txt 158 requirements.freeze_raw.txt
my stripped down requirements ended up with 24 packages total:
$ wc -l requirements*txt | egrep 'prod|dev' 6 requirements.dev.txt 18 requirements.prod.txt
On my test environment,
pip install -r requirements.prod.txt -r requirements.dev.txt got me 48 packages, after dependencies were pulled in.
$ pip freeze | wc -l 48
This is all very nice, but hopefully you have sufficient tests to allow you to be confident you didn't miss anything!
The way you can test is to create a new virtualenv, pip install both requirements files and then run your tests.
WARNING: be cautious in hardcoding package associations in dev/tests buckets rather than prod. Your tests could run to success, but production would still fail.
These are pretty conservative as nose and pytest are really only testing tools.
ClassifierPip: buckets: tests: - nose - nose2 - pytest
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.