Data cleaning package

version: 0.2.1

This repository collects python modules to clean and prepare data for downstream analyses or presentation in reports. For example, the module contains functionality to created formatted publication ready baseline tables or supplementary excel tables.

Please consult the package documentation.

Installation

At present, the repository is undergoing development and no packages exist yet on PyPy or in Conda. Therefore it is recommended that it is installed in either of the two ways listed below. First, clone this repository and then cd to the root of the repository.

git clone git@gitlab.com:SchmidtAF/clean-data.git
cd clean-data

Installation using conda dependencies

A conda environment is provided in a yaml file in the directory ./resources/conda/. A new conda environment called clean-data can be built using the command:

# From the root of the repository
conda env create --file ./resources/conda/envs/conda_create.yaml

To add to an existing environment use:

# From the root of the repository
conda env update --file ./resources/conda/envs/conda_update.yaml

Next the package can be installed:

make install

Development

For development work, install the package in editable mode with Git commit hooks configured:

make install-dev

This command installs the package in editable mode and configures Git commit hooks, allowing you to run git pull to update the repository or switch branches without reinstalling.

Alternatively, you can install manually:

python -m pip install -e .
python .setup_git_hooks.py

Git Hooks Configuration

When setting up a development environment, the setup-hooks command configures Git hooks to enforce conventional commit message formatting and spell check using codespell.

To view the commit message format requirements, run:

./.githooks/commit-msg -help

For frequent use, add this function to your shell configuration (~/.bashrc or ~/.zshrc):

commit-format-help() {
    local git_root
    git_root=$(git rev-parse --show-toplevel 2>/dev/null)
    
    if [ -z "$git_root" ]; then
        echo "Error: Not inside a git repository"
        return 1
    fi
    
    local hook_path="$git_root/.githooks/commit-msg"
    
    if [ ! -f "$hook_path" ]; then
        echo "Error: commit-msg hook not found"
        return 1
    fi
    
    "$hook_path" --help
}

If you have already installed the package in editable mode without running _setup_git_hooks.py, you can configure the hooks manually at any time by running:

_setup_git_hooks.py

Validating the package

After installing the package from GitLab, you may wish to run the test suite to confirm everything is working as expected:

# From the root of the repository
pytest tests

Usage

Please have a look at the examples in resources for some possible recipes.