Skip to main content
Documentation Status Unit tests and formatting Recording Slides

Reproducibility is important for software: if it’s not reproducible, it’s not useful. Even if you don’t plan on sharing your code, imagine coming back to a project after a few weeks, or having to install it on a new machine. You’ll be all the more thankful to your past self if you have a clear way to install and run your code.

This repository is a collection of tips and tricks for developing stable and reproducible Python code. There is a slight focus on scientific computing, but the general principles can apply to most Python projects. If you’re reading this from GitHub, please check out the documentation for a more in-depth explanation of the topics covered.

The intended audience is myself (as I often find myself going to past projects to find how I did something!), but also for students and anyone who is interested in learning some new tricks or even sharing their own! I try to follow the principles laid out here on development and reproducibility, so feel free to point out any lapses or suggest improvements, either by opening an issue or pull request.

As is typical in open source, there are many ways to do the same thing. But hopefully this gives you a starting point. Feel free to pick and choose the features that you like. This flexibility is one of the best (and worst parts) of open source. Some of the things we cover:

  • Packaging and distribution with Poetry.

  • Virtual environments with Conda and virtualenv.

  • Version control with Git/GitHub.

  • Reproducible examples with Hydra.

  • Documentation with Sphinx.

  • Code formatting with Black, Flake8, and isort.

  • Unit tests and continuous integration with pytest and GitHub Actions.

  • Remote development with VS Code <https://code.visualstudio.com/> and SSH.

The accompanying slides and video are from a tutorial given at LauzHack’s Deep Learning Bootcamp. Feel free to modify and use it for your own purposes.

Installation

This “dummy” package can be installed with pip:

pip install pydevtips

Or from source. Firstly, Poetry must be installed: https://python-poetry.org/docs/#installation

# -- recommend using pipx
pipx install poetry

# -- or with official installer
# curl -sSL https://install.python-poetry.org | python3 -

# I recommend creating virtual environment in the project folder
poetry config virtualenvs.in-project true

Then the following commands can be run:

# get source code
git clone git@github.com:ebezzam/python-dev-tips.git
cd python-dev-tips

# create new environment, press enter to accept
# -- important to set python version, otherwise `python` executable may not exist
# -- (would be `python3` instead)
conda create -n project_env python=3.11

# view available environments
conda info --envs

# activate environment
conda activate project_env

# install package locally
(project_env) poetry install --with dev

# install pre-commit hooks
(project_env) pre-commit install

# run tests
(project_env) pytest

# deactivate environment
(project_env) conda deactivate

Or without conda and just Poetry:

# get source code
git clone git@github.com:ebezzam/python-dev-tips.git
cd python-dev-tips

# install package locally
poetry install --with dev
# -- `poetry env info` to see where the virtual environment is located

# install pre-commit hooks
poetry run pre-commit install

# run tests
poetry run pytest

## NOTE that Python related commands need to be run
## with `poetry run` or by activating the virtual environment
## https://python-poetry.org/docs/basic-usage/#activating-the-virtual-environment

Examples

Examples can be found in the examples and notebooks folders. Scripts from the examples folder should be run from the root of the repository, e.g.:

python examples/real_convolve.py

Parameter setting is done with hydra. More on that in the Reproducible examples section of the documentation.

TODO

  • switch to ruff for code formatting: https://docs.astral.sh/ruff/

  • numba: https://numba.pydata.org/

  • picking a license

  • change documentation links to main branch

  • github page

  • point out features in scripts: object-oriented, asserts, tqdm, type hints

  • manifest file to not include file in package

  • GitHub actions for releasing to PyPi when changes to version

  • pytorch compatible

  • Cython / C++

Metadata

Release files for pydevtips 0.0.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pydevtips 0.0.4
File Size Uploaded Provenance
pydevtips-0.0.4.tar.gz 5.6 kB Without provenance Details

Built distribution (wheel)

Table of built distributions (wheels) for pydevtips 0.0.4
File Interpreter ABI Platform
pydevtips-0.0.4-py3-none-any.whl Python 3 none any Details

Total release size: 11.6 kB

Release files / pydevtips-0.0.4.tar.gz

Download URL pydevtips-0.0.4.tar.gz
Size 5.6 kB
Tags Source
SHA-256 checksum
How to use checksums
482357ceea3d05f3d3b1521c535a01e86500668c94db9e276d1f8847a6a6316e
BLAKE2b-256 checksum
How to use checksums
c045cd24d6cc805e7ec2ff2011d28e36b1b64974655ecead1cfd71ea1533e004
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.8.4 CPython/3.13.0 Darwin/22.6.0

Release files / pydevtips-0.0.4-py3-none-any.whl

Download URL pydevtips-0.0.4-py3-none-any.whl
Size 6.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5c0c3f38af22750b44dd23bdeabe415c1f46d8851d82297a9a53d238b09d603a
BLAKE2b-256 checksum
How to use checksums
521c93f513a9746f2e9dab338e6cd0dd0f085ead4691cbcd96b8bf3dd9a41634
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.8.4 CPython/3.13.0 Darwin/22.6.0

Security for pydevtips 0.0.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page