pyproject-ops 0.8.1

Python project Ops automation.

---

Welcome to pyproject_ops Documentation

What is this project?

There are various folder structures for Python projects, each with its own merits. Drawing from my extensive Python career experience, I have developed a personal best practice. To streamline the creation of the codebase skeleton according to this practice, I have introduced a tool called cookiecutter-pyproject.

By adopting this folder structure setup, pyproject_ops is an automation tool, capable of handling common tasks throughout the Python project development lifecycle. These tasks include “creating virtual environments”, “installing dependencies”, “running tests”, “building documentation sites”, and more.

This tool is, in essence, my secret weapon for efficiently managing over 120 Python open source projects, all available on PyPI at https://pypi.org/user/machugwu/. Each of these projects is equipped with essential features such as continuous integration (CI), code coverage testing, matrix testing, and documentation websites.

A little history about this project:

I previously had an automation tool called pygitrepo that was tailored to my former development workflow. pygitrepo primarily relied on setup.py and requirements.txt files. Even today, pyproject_ops continues to support the old convention while also accommodating pyproject.toml and poetry. This versatility enables me to write less code when automating Python development workflows.

How to Use

Use as a CLI

$ pip install pyproject_ops
$ pyops --help
NAME
    pyops - python project ops command line interface.

SYNOPSIS
    pyops COMMAND | <flags>

DESCRIPTION
    python project ops command line interface.

FLAGS
    -v, --version=VERSION
        Type: bool
        Default: False

COMMANDS
    COMMAND is one of the following:

     build_doc
       ** 📔 Build documentation website locally

     build_doc_only
       📔 Build documentation website locally without checking doc dependencies

     bump_version
       🔼 Bump semantic version.

     cov
       ** 🧪 Run code coverage test

     cov_only
       🧪 Run code coverage test without checking test dependencies

     deploy_latest_doc
       🚀 📔 Deploy Documentation Site To S3 as Latest Doc

     deploy_versioned_doc
       🚀 📔 Deploy Documentation Site To S3 as Versioned Doc

     install
       ** 💾 Install main dependencies and Package itself

     install_all
       ** 💾 💻 🧪 📔 🤖 Install All Dependencies

     install_automation
       💾 🤖 Install Dependencies for Automation Script

     install_dev
       💾 💻 Install Development Dependencies

     install_doc
       💾 📔 Install Document Dependencies

     install_test
       💾 🧪 Install Test Dependencies

     int
       ** 🧪 Run integration test

     int_only
       🧪 Run integration test without checking test dependencies

     poetry_export
       Export requirements-*.txt from poetry.lock file

     poetry_lock
       ** Resolve dependencies using poetry, update poetry.lock file

     publish
       📦 Publish package to PyPI

     test
       ** 🧪 Run test

     test_only
       🧪 Run test without checking test dependencies

     venv_create
       ** 🐍 Create Virtual Environment

     venv_remove
       ** 🗑 🐍 Remove Virtual Environment

     view_cov
       👀 🧪 View coverage test output html file locally in web browser.

     view_doc
       ** 👀 📔 View documentation website locally

     view_latest_doc
       👀 📔 View the latest documentation website on S3

Use as a Python library

>>> from pyproject_ops.api import PyProjectOps
>>> pyops = PyProjectOps.from_pyproject_toml("pyproject.toml")
>>> pyops.create_virtualenv()
>>> pyops.pip_install_all()
>>> pyops.poetry_lock()
>>> pyops.poetry_install_all()
>>> pyops.run_cov_test()
>>> pyops.view_cov()
>>> pyops.build_doc()
>>> pyops.view_cov()
>>> pyops.python_build()
>>> pyops.twine_upload()

Folder Structure

Below is the folder structured used in pyproject_ops. The first item is the relative path from the project root directory. The second item is the attribute name that you can use to access the path in pyproject_ops. The third item is the description of the path. You can find a concrete example at HERE

• .venv: PyProjectOps.dir_venv, The virtualenv directory.

• .venv/bin: PyProjectOps.dir_venv_bin, The bin folder in virtualenv.

• .venv/bin/pip: PyProjectOps.path_venv_bin_pip, The pip command in virtualenv.

• .venv/bin/pytest: PyProjectOps.path_venv_bin_pytest, The pytest command in virtualenv.

• .venv/bin/python: PyProjectOps.path_sys_executable, The current Python interpreter path.

• .venv/bin/python: PyProjectOps.path_venv_bin_python, The python executable in virtualenv.

• .venv/bin/twine: PyProjectOps.path_bin_twine, The twine CLI command path.

• build: PyProjectOps.dir_build, The build folder for Python or artifacts build.

• build/glue: PyProjectOps.dir_build_glue, The AWS glue artifacts build folder.

• build/glue/extra_py_files: PyProjectOps.dir_build_glue_extra_py_files, The AWS glue extra Python files build folder.

• build/glue/extra_py_files.zip: PyProjectOps.path_build_glue_extra_py_files_zip, The AWS glue extra Python files zip file path.

• build/lambda: PyProjectOps.dir_build_lambda, The AWS Lambda artifacts build folder.

• build/lambda/layer.zip: PyProjectOps.path_build_lambda_layer_zip, The AWS Lambda layer zip file path.

• build/lambda/python: PyProjectOps.dir_build_lambda_python, The AWS Lambda layer build folder. This folder contains the dependencies.

• build/lambda/python/aws: PyProjectOps.path_build_lambda_bin_aws, This is the AWS CLI executable path in Lambda layer.

• build/lambda/source.zip: PyProjectOps.path_build_lambda_source_zip, The AWS Lambda source code deployment package zip file path.

• config: PyProjectOps.dir_config, The folder that stores the config files.

• config/config.json: PyProjectOps.path_config_json, Path to the JSON file that stores the non-sensitive config.

• dist: PyProjectOps.dir_dist, The dist folder for Python package distribution (.whl file).

• docs: PyProjectOps.dir_sphinx_doc, Sphinx docs folder.

• docs/build: PyProjectOps.dir_sphinx_doc_build, The temp Sphinx doc build folder.

• docs/build/html: PyProjectOps.dir_sphinx_doc_build_html, The built Sphinx doc build HTML folder.

• docs/build/html/index.html: PyProjectOps.path_sphinx_doc_build_index_html, The built Sphinx doc site entry HTML file path.

• docs/source: PyProjectOps.dir_sphinx_doc_source, Sphinx docs source code folder.

• docs/source/conf.py: PyProjectOps.dir_sphinx_doc_source_conf_py, Sphinx docs conf.py file path.

• docs/source/pyproject_ops: PyProjectOps.dir_sphinx_doc_source_python_lib, The generated Python library API reference Sphinx docs folder.

• htmlcov: PyProjectOps.dir_htmlcov, The code coverage test results HTML output folder.

• htmlcov/index.html: PyProjectOps.path_htmlcov_index_html, The code coverage test results HTML file.

• lambda_app: PyProjectOps.dir_lambda_app, The AWS Lambda app handler file and Lambda related code directory.

• lambda_app/.chalice/config.json: PyProjectOps.path_chalice_config, The AWS Chalice framework’s config file path.

• lambda_app/.chalice/deployed: PyProjectOps.dir_lambda_app_deployed, The generated deployed.json file for AWS Chalice framework’s.

• lambda_app/app.py: PyProjectOps.path_lambda_app_py, The app.py file for AWS Chalice framework.

• lambda_app/lambda_function.py: PyProjectOps.path_lambda_function_py, The lambda_function.py handler file for AWS Lambda, if you are not using

• lambda_app/update_chalice_config.py: PyProjectOps.path_lambda_update_chalice_config_script, Example: ${dir_project_root}/lambda_app/update_chalice_config.py

• lambda_app/vendor: PyProjectOps.dir_lambda_app_vendor, The vendor folder for AWS Chalice framework’s packaging.

• lambda_app/vendor/pyproject_ops: PyProjectOps.dir_lambda_app_vendor_python_lib, The source python library folder in AWS Chalice framework’s vendor folder.

• poetry-lock-hash.json: PyProjectOps.path_poetry_lock_hash_json, The poetry-lock-hash.json file path. It is the cache of the poetry.lock file hash.

• poetry.lock: PyProjectOps.path_poetry_lock, The poetry.lock file path.

• pyproject.toml: PyProjectOps.path_pyproject_toml, The pyproject.toml file path.

• pyproject_ops: PyProjectOps.dir_python_lib, The current Python library directory.

• pyproject_ops/_version.py: PyProjectOps.path_version_py, Path to the _version.py file where the package version is defined.

• requirements-automation.txt: PyProjectOps.path_requirements_automation, The requirements-automation.txt file path.

• requirements-dev.txt: PyProjectOps.path_requirements_dev, The requirements-dev.txt file path.

• requirements-doc.txt: PyProjectOps.path_requirements_doc, The requirements-doc.txt file path.

• requirements-test.txt: PyProjectOps.path_requirements_test, The requirements-test.txt file path.

• requirements.txt: PyProjectOps.path_requirements, The requirements.txt file path.

• tests: PyProjectOps.dir_tests, Unit test folder.

• tests_int: PyProjectOps.dir_tests_int, Integration test folder.

• tests_load: PyProjectOps.dir_tests_load, Load test folder.

Develop and Release Strategy

This project is a “meta” project for other projects, it is very hard to test. I keep using this project in many of my production projects, and continuously improving it. I will merge all the changes manually into this every three months.

Install

pyproject_ops is released on PyPI, so all you need is:

$ pip install pyproject_ops

To upgrade to latest version:

$ pip install --upgrade pyproject_ops