sample-project-ivi 0.0.3

A sample project to demonstrate packaging and testing.

python-template

This is a template for a project with the following features:

• Setuptools for Python project
• Sphinx documentation
• Pytest unit tests with coverage
• Pylint code quality checks
• Type checking with mypy
• Black code formatting
• Gitlab CI/CD

Documentation 📖

Auto generated documentation can be found here. :point_left:

Setup Sphinx

Sphinx is a documentation generator that can be used to generate documentation from docstrings in the code.

All commands are to be run from the project directory.

Setup Sphinx for a New Project

1. Delete the existing docs directory if it exists.

   rm -rf docs
   

2. Create Sphinx Project

   We will use the sphinx-quickstart command to create a new Sphinx project. The files will be created in the docs directory.

   sphinx-quickstart docs
   

3. Edit conf.py

   Edit the docs/conf.py file to add the following lines to the top of the file:

   import os
   import sys
   sys.path.insert(0, os.path.abspath('..'))
   

   Add the following lines to the extensions list:

   extensions = [
       "sphinx.ext.autodoc",  # for autodoc
       "sphinx.ext.autosummary",  # for autosummary
       "sphinx.ext.viewcode",  # for source code
       "sphinx.ext.napoleon",  # for google style docstrings
       "sphinx_autodoc_typehints",  # for type hints
       "sphinx_copybutton",  # for copy button
       "sphinx-prompt",  # for prompt
       "recommonmark",  # for markdown
   ]
   

   Change the theme to sphinx_rtd_theme or another theme of your choice if you prefer:

   html_theme = "sphinx_rtd_theme"
   

   Add the following lines to the end of the file:

   # generate autosummary even if no references
   autosummary_generate = True
   autosummary_imported_members = True
   

4. Generate Documentation

   Run the following command to generate the documentation:

   sphinx-apidoc -f -e -M -o docs/ sample_project_ivi/
   

   The above command will generate the documentation for the sample_project_ivi package. There will be a modules.rst file in the output directory. This file will be used to generate the table of contents for the documentation.

5. Edit index.rst

   Add modules to the table of contents by adding the following lines to the docs/index.rst file:

   .. toctree::
   :maxdepth: 2
   :caption: Contents:
   
   modules
   

6. Build Documentation

   Run the following command to build the documentation:

   # for html documentation
   sphinx-build -b html docs docs/_build
   
   # for pdf documentation (requires latex to be configured)
   sphinx-build -M latexpdf docs docs/_build
   

   The documentation will be generated in the docs/_build directory.

Setuptools

Setup Setuptools for a New Project

• Use setuptools to package your code.

• Setuptools lets you easily download, build, install, upgrade, and uninstall Python packages.

• Setuptools can be configured using the following files:

  • setup.py
  • setup.cfg

• Setuptools lets you install your code using pip.

  pip install git+REPO_URL
  # or
  pip install .
  # or in editable mode
  pip install -e .
  

• You can also save configurations for your project in the setup.cfg file.

• Sample setup.py file:

  from setuptools import setup, find_packages
  
  setup(
      name="project_name",
      version="0.1.0",
      description="Project description",
      author="Author name",
      author_email="Author email",
      url="Project url",
      license="License name",
      # find_packages() finds all the packages in the src directory
      # package_dir={"": "src"} can be used to specify the source directory
      packages=find_packages(),
      # package_data={"": ["data/*.txt"]} can be used to include data files
      # the following command will include all txt files in the data directory
      # hence hardcoded paths in the code can be avoided and the code can be made more portable
      package_data={"src": ["data/*.txt"]},
      # install_requires can be used to specify the dependencies of the project
      # will be installed automatically when the project is installed 
      install_requires=[
          "package1",
          "package2",
      ],
      # extras_require can be used to specify the dependencies of the project
      # will not be installed automatically when the project is installed
      extras_require={
          "dev": [
              "package3",
              "package4",
          ],
      },
      # entry_points can be used to specify the command line scripts
      entry_points={
          "console_scripts": [
              "script_name=package.module:main",
          ],
      },
  )
  

• Check the Setuptools documentation for more information.

Unit testing with Pytest

Setup Pytest for a New Project

• Use pytest to write and run tests.

• Unittesting makes sure that your code works as expected.

• Pytest can automatically find and run tests in files named test_*.py or *_test.py.

• Pytest can be run using the following command:

  pytest test_file.py
  # or
  python -m pytest test_file.py
  # or to run all tests in a directory 'tests'
  pytest tests/
  

• Here is a sample test file:

  import pytest
  
  def test_function():
      assert 1 == 1
  

• Some sample tests can be found in the tests.

• Check the Pytest documentation for more information.

• Use coverage to check the test coverage of your code.

• Coverage identifies which parts of your code are executed during testing and reports can be generated that show which parts of your code are missed by the tests.

• Coverage can be run using the following command:

  coverage run -m pytest file.py
  coverage report
  

• Check the Coverage documentation for more information.

• tox can be used to run all the above tools in one command.

• Tox creates a virtual environment for each Python version you want to test.

• Tox can be run using the following command:

  tox
  

• Configuration for tox can be specified in the following files:

  • tox.ini
  • pyproject.toml
  • setup.cfg

• A sample tox configuration file:

  [tox]
  envlist = py{38,311} # specify the python versions to test
  
  [testenv]
  deps =
      pytest
      coverage
  commands =
      pytest
      coverage run -m pytest
      coverage report
  

• Check the Tox documentation for more information.

• Sample tox configuration files can be found in the tox directory.

Code Quality Checks

Setup Code Quality Checks for a New Project

Pylint

• Use linters to check your code for errors and style.

• Pylint is one of the tools that can be used to lint your code.

• flake8 is another popular tool that can be used to lint your code.

• Code can be linted using the following command:

  # recommended (pylint and flake8)
  pylint file.py
  # or
  flake8 file.py
  

• Pylint can provide you with a score for your code. This score can be used to track the quality of your code and also be used to enforce a minimum score. In the template repository, the minimum score is set to 9.0. If the score is below 9.0, the CI pipeline will fail. This is done with the following command:

  pylint --fail-under=9 uam
  

• It can be used to enforce a coding style, find bugs and unused code.

• Check the Pylint documentation for more information.

• ruff can also be used to lint your code. ruff is based on rust is much faster than pylint and flake8.

• Example with pylint:

  # Example code test.py
  def calculate_average(numbers):
      total = sum(numbers)
      number_of_items = len(numbers)
      average = total / len(numbers)
      print("The average is:", average)
  
  
  numbers = [1, 2, 3, 4, 5]
  calculate_average(numbers)
  

  pylint test.py
  # Output
  ************* Module example
  test.py:10:0: C0304: Final newline missing (missing-final-newline)
  test.py:1:0: C0114: Missing module docstring (missing-module-docstring)
  test.py:2:0: C0116: Missing function or method docstring (missing-function-docstring)
  test.py:2:22: W0621: Redefining name 'numbers' from outer scope (line 9) (redefined-outer-name)
  test.py:4:4: W0612: Unused variable 'number_of_items' (unused-variable)
  
  ------------------------------------------------------------------
  Your code has been rated at 2.86/10 (previous run: 4.29/10, -1.43)
  

  The same suggestions can be seen in VSCode as well once the extensions are installed.

Mypy

• Use type hints to specify the type of variables and arguments.

• Type hints are optional in Python, but they are recommended.

• Type hints can be very useful for documentation and debugging.

• Type hints are specified using the following syntax:

  variable: type
  
  def function(argument: type) -> type:
      pass
  
  class Class:
      def method(self, argument: type) -> type:
          pass
  

• Check the PEP 484 for more information.

• Tools like mypy can be used to check your code for type errors.

  mypy file.py
  mypy src/
  

• It is a static type checker that can identify type errors in your code.

  def function(argument: int) -> int:
      return argument + "1"
  

  mypy file.py
  error: Unsupported operand types for + ("int" and "str")
  

• Check the Mypy documentation for more information.

• Example 1

  # Example code test_1.py
  def add_numbers(a: int, b: int) -> int:
      return a + b
  
  result = add_numbers(5, "10")  # Type mismatch: 'str' cannot be added to 'int'
  print(result)
  

  mypy test_1.py
  # Output
  test.py:5: error: Argument 2 to "add_numbers" has incompatible type "str"; expected "int"  [arg-type]
  Found 1 error in 1 file (checked 1 source file)
  

• Example 2

  # Example code test_2.py
  import numpy as np
  
  def calculate_mean(data: np.ndarray) -> float:
      return np.mean(data)
  
  numbers = [1, 2, 3, 4, 5]  # Incorrect type: List[int] instead of np.ndarray
  mean = calculate_mean(numbers)
  print(mean)
  

  mypy test_2.py
  # Output
  test.py:8: error: Argument 1 to "calculate_mean" has incompatible type "List[int]"; expected "ndarray[Any, Any]"  [arg-type]
  Found 1 error in 1 file (checked 1 source file)
  

• You need to provide type hints for your code to be checked by mypy. Else it will not be able to check the type of the variables.

• Mypy will also check for type errors in third-party libraries.

Black

• Use black to format your code.

• Black lets you format your code in a consistent way.

• Black can be run using the following command:

  black --check --diff file.py # will show the changes that will be made without making them
  # or
  black file.py # will format the file in place
  

• It can be integrated with your editor to format your code on save.

• Check the Black documentation for more information.

• Example:

  # before
  def my_function():
  x=1
  y = 2
  z=3
  if x>y:
  print("x is greater than y")
  else:
  print("y is greater than or equal to x")
  

  ↓

  # after
  def my_function():
      x = 1
      y = 2
      z = 3
      if x > y:
          print("x is greater than y")
      else:
          print("y is greater than or equal to x")
  

• Use isort to sort your imports.

• Isort does sort import in the following order:

  • standard library imports
  • related third party imports
  • local application/library specific imports

• Isort can be run using the following command:

  isort --check-only --diff file.py # will show the changes that will be made without making them
  # or
  isort file.py # will format the file in place
  

• It can also be integrated with your editor to sort your imports on save.

• Here is an example of how isort sorts imports:

  # before
  import os
  import sys
  import django
  import requests
  import uam
  

  ↓

  # after
  import os
  import sys
  
  import django
  import requests
  
  import uam
  

  • Check the Isort documentation for more information.