python-package-sync-tool 0.5.2

Small tool to sync package from different machines

PythonPackageSyncTool

PythonPackageSyncTool is a Python utility to "fix" requirements.txt.

It is used to make manipulation on requirements.txt that is produces by pip freeze

Getting Help

QuickStart

First of all it is better to install alex_ber_utils (this dependency is used in setup.py).

pip3 install -U alex_ber_utils

Than:

pip3 install -U python-package-sync-tool

In some directory create config.yml. Here is example (https://github.com/alex-ber/alpine-anaconda3/blob/master/python_package_sync_tool_config.yml)

treeroot:
    source: installed.txt
    destination: rm.txt
    mutual_exclusion: True
    remove:
      - conda
      - conda-package-handling
      - icc-rt
      - intel-openmp
      - mkl
      - mkl-fft
      - mkl-random
      - mkl-service
      - pycosat
      - sip
      - tbb
    add:
  #    - graphviz==2.38.0
  #    - entrypoints==0.2.3
  #    - numpy==1.16.2
  #    - PyYAML==5.1
  #    - sasl==0.2.1
  #    - scipy==1.2.1
  #    - tables==3.4.4
  #    - terminado==0.8.2
  #    - tornado==5.1.1
  #    - Twisted==19.2.0

Create installed.txt with installed packages (in pip freeze format).

Now, you can run

python3 -m alexber.reqsync.data

You can also override parameters in the yaml file:

python3 -m alexber.reqsync.data --treeroot.add=some_new_package:1.0.0

shorthand

python3 -m alexber.reqsync.data --add=some_new_package:1.0.0

is also supported.

Of course, you can provide all parameters via CLI and not to use YAML file.

You can also supply --general.log (no shorthand here) parameter (you can do it also via config.yml). There is some default log configuration (to write everything to stdout).

Note: config.yml now also supports format that alex-ber-utils https://github.com/alex-ber/AlexBerUtils support for a long time.

Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for full supported list of general parameters.

See documentation here

My parser module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718

My ymlparsers module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6

My major init_app_conf module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c

Note:

Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.

You can specified multiple packages using comma delimiter.

You can specify path to your config file using --config_file (or --general.config_file). It can be absolute or relative.

Alternatively,

cd /opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/

Note: This is Path where you're actually install my utility, it can be different in your machine.

If you use venv it will look something like:

cd /opt/MyProject/venv/Lib/site-packages/alexber/reqsync

Alternatively you can create script file for yourself, named, say, driver.py:

#!/usr/bin/python3

import alexber.reqsync.app as app

if __name__ == "__main__":
   app.main()

Than create file config.yml near your script (see data/config.yml) or provide all parameter using command line argruments. Use ':' in places where you should naturally write '==' (see explanation below).

Parameters 'source' and 'destination' are required. You should also provide (requirements) file for 'source'.

mutual_exclusion has default value True.

Now, type

python3 -m alexber.reqsync.data --add=some_new_package:1.0.0

or if you're using script (driver.py) go the directory with the script and type

./driver.py --add=some_new_package:1.0.0

or if you install my tool to Anaconda/Python/venv that has it's bin folder is in the Path you can run

python_package_sync_tool --add=some_new_package:1.0.0

or alternativley

you can run

reqsync --add=some_new_package:1.0.0

This will run quick check whether package is not in remove list. If it is, the utility will fail. You can override this beahivor by supplying --mutual_exclusion=False.

Then, this will add some_new_package with version 1.0.0 to the requirements-dest.txt

Note:

Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.

You can specified multiple packages using comma delimiter.

You can specifiy path to your config file using --config_file (or --general.config_file).

It can be absolute or relative. If you're running using script (driver.py), that it can be relative to the directory whether you put your script. If you're running as the module (python3 -m), it can be relative to /opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/ (exact path can be different, see above).

You can supplied multiply packages by using comma:

python3 -m alexber.reqsync.data --add=some_new_package:1.0.0,another_new_package:2.0.0

or if you're using script (driver.py) go the directory with the script and type

./driver.py --add=some_new_package:1.0.0,another_new_package:2.0.0

Installing from Github

python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip

Optionally installing tests requirements.

python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip#egg=python-package-sync-tool[tests]

Or explicitly:

wget https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip -O master.zip; unzip master.zip; rm master.zip

And then installing from source (see below).

Installing from source

python3 -m pip install . # only installs "required"

python3 -m pip install .[test] # installs dependencies for tests

Using Docker

alexberkovich/python_package_sync_tool:latest contains all python_package_sync_tool dependencies. This Dockerfile is very simple, you can take relevant part for you and put them into your Dockerfile.

Alternatively, you can use it as base Docker image for your project and add/upgrade another dependencies as you need.

For example:

FROM alexberkovich/python_package_sync_tool:latest

COPY requirements.txt etc/requirements.txt

RUN set -ex && \
    #latest pip,setuptools,wheel
    pip install --upgrade pip setuptools wheel && \
    pip install python_package_sync_tool 
    pip install -r etc/requirements.txt 

CMD ["/bin/sh"]
#CMD tail -f /dev/null

where requirements.txt is requirements for your project.

From the directory with setup.py

python3 setup.py test #run all tests

pytest

Requirements

PythonPackageSyncTool requires the following modules.

• Python 3.7+

• PyYAML==5.1

• alex-ber-utils==0.2.5

Changelog

All notable changes to this project will be documented in this file.

#https://pypi.org/manage/project/python-package-sync-tool/releases/

[0.5.2] - 2020-12-17

Removed

• alexber.reqsync.utils.app_conf removed.

Changed

• Pytest version updated (also some tests dependencies changed)
• alex-ber-utils dependency updated.
• Logger information was moved from app.py to config.yml file.
• config.yml now also supports format that alex-ber-utils https://github.com/alex-ber/AlexBerUtils support for a long time.

Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for full supported list of general parameters.

See documentation here

My parser module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718

My ymlparsers module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6

My major init_app_conf module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c

• Logger is enabled, relative path is resolved correctly, and other little tweaks, see

Integrating Python’s logging and warnings packages [https://medium.com/analytics-vidhya/integrating-pythons-logging-and-warnings-packages-7ffd6f65e02d]

Making relative path to file to work [https://alex-ber.medium.com/making-relative-path-to-file-to-work-d5d0f1da67bf]

• config.xml - to remove package was added mkl*-related packagages and anaconda-specific packages.

• REAMDE.md was updated to reflect all changes above.

Added

• Optional Docker support added.

[0.5.1] - 2020-10-25

Added

• Support for Python 3.8

Changed

[0.4.3] - 2019-10-17

Changed

• anaconda-navigator and conda-build added to config.yml.

[0.4.2] - 2019-10-16

Changed

• navigator-updater added to config.yml.

[0.4.1] - 2019-05-30

Changed

• Bug fix: adding packages before all existing one works incorrect.
• Removing alexber.reqsync.utils.parsers. It was fully duplicated by alexber.utils.parsers. So, all usage was change to the latest (part of alex-ber-utils).

[0.3.1] - 2019-05-23

Changed

• Bug fix: adding packages before all existing one works incorrect.

Added

• Unit test for bug fix that check adding packages before all existing one.
• More detail assertion to integration tests.
• Unit-test for non-sorted requirements-src.
• Unit-test that check run with empty add and empty remvoe.
• Unit-test that check removing single package.
• Unit-test that check remove first package in requirements-src.
• Unit-test that check remove last package in requirements-src.
• Unit-test that check that empty lines in requirements-src are ignored.
• Unit-test that check correct usage of file input buffer and file output buffer.

[0.2.11] - 2019-05-22

Changed

• Dependency alex-ber-utils bumped up to 0.2.5.

[0.2.8] - 2019-05-22

Changed

• Dependency alex-ber-utils bumped up to 0.2.4.

[0.2.6] - 2019-05-22

Changed

• Dependency alex-ber-utils bumped up to 0.2.3.

[0.2.5] - 2019-05-22

Changed

• Fixed bug in setup.py, incorrect order between VERSION and UploadCommand (no tag was created on upload)
• Dependency alex-ber-utils bumped up to 0.2.2.

[0.2.4] - 2019-05-22

Changed

• Adding dependency alex-ber-utils 0.2.1 to README.md.

[0.2.3] - 2019-05-22

Changed

• Upgrading urllib3, SQLAlchemy, pycrypto dependenies beacause of volnurabilities issues.

[0.2.2] - 2019-05-22

Changed

• Fixing python-package-sync-tool.
• Creating alias reqsync to python-package-sync-tool.
• Some minour fixed.

[0.2.1] - 2019-05-22

Changed

• Changing dependency version of alex-ber-utils to 0.2.1.

[0.2.0] - 2019-05-22

Changed

• Only bumping up version.

[0.1.9] - 2019-05-22

Changed

• Added alex-ber-utils as dependency.
• Deleting old README-old.rst file.
• requirements-src.txt updated.
• Clarification added to README.md that alex_ber_utils should be installed first.
• formatting CHANGELOG.MD (minor fix)
• Added alternative of usage of python_package_sync_tool to README.md
• Fixing bug that --add is empty
• Factor out tests_data to seperate folder, use importlib.resources API.
• Updated README-old.rst

[0.1.8] - 2019-05-20

Changed

• README.md change, key '-U' added to pip3 install.

[0.1.7] - 2019-05-20

Removed

• Some project cleanup.

Changed

• CHANGELOG and REAMDE now use Markdown format.
• REAMDE totally rewritten.
• Fixing bugs in the core algorithm. Simplifying code.
• Fixing correct handling of package adding to the buttom of the list.

[0.1.6] - 2019-05-20

Added

• __init__.py file added to alexber.reqsync.data.

[0.1.5] - 2019-05-20

Added

• Small tool to sync package from different machines.