np-validator 0.10.0

Awesome `np-validator` is a Python cli/package created with https://github.com/TezRomacH/python-package-template

np-validator

np-validator is just a simple Python cli/package that validates data sources for the neuropixel pipline using templated workflows.

Quick overview

API

Auto-generated steps

By default run_validation can be run on a list of filepaths. A list of ValidationStep will be auto-generated if a valid project name is supplied. If no project name is supplied project name will be "default".

from np_validator import run_validation

filepaths = [
  "some/sort/of/path/prefix/uuid-maybe.mapping.pkl",
  "some/sort/of/path/prefix/uuid-maybe.behavior.pkl",
  "this/will/be/ignored/uuid-maybe.replay.pkl",
]

results = run_validation(filepaths, project="default")

Manually generating validation steps

from np_validator import Processor, Validator, ValidationStep, run_validation

# make a basic filesize validator
fs_validator = Validator(
  name="meets_filesize_threshold",
  args={
    "threshold": 10,
  },
)

# add validator to a validation step
validation_step_0 = ValidationStep(
  path_suffix=".mapping.pkl",
  validators=[fs_validator, ],
)

# make a validation step with a processor
# processors convert the data source from it's basic state, a filepath, to an easier to use object

# unpickle, unpickles an arbitrary filepath into a python object
unpickler = Processor(
  name="unpickle",
)

# has_dict_key, checks if a dict-like interface has a key at path
session_uuid_validator = Validator(
  name="has_dict_key",
  args={
    "path": ["session_uuid", ],
  }
)

# assembling it all together as a validation step

validation_step_1 = ValidationStep(
  path_suffix=".behavior.pkl",
  processor=unpickler,
  validators=[session_uuid_validator, ],
)

# running a validation
filepaths = [
  "some/sort/of/path/prefix/uuid-maybe.mapping.pkl",
  "some/sort/of/path/prefix/uuid-maybe.behavior.pkl",
  "this/will/be/ignored/uuid-maybe.replay.pkl",
]
results = run_validation(
  filepaths,
  [validation_step_0, validation_step_1, ],
)

Updating autogenerated steps

from np_validator import Processor, Validator, ValidationStep, autogenerate_validation_steps, update_project_validation_steps

# autogenerated steps are organized by projects
validation_step_0 = ValidationStep(
  path_suffix=".mapping.pkl",
  validators=[
    Validator(
      name="meets_filesize_threshold",
      args={
        "threshold": 10,
      },
    ),
  ],
)

validation_step_1 = ValidationStep(
  path_suffix=".behavior.pkl",
  processor=Processor(
    name="unpickle",
  ),
  validators=[
    Validator(
      name="has_dict_key",
      args={
        "path": ["session_uuid", ],
      }
    ),
  ],
)

# updating project validation steps overwrites the previous steps

update_project_validation_steps(
  "pretest",  # project name
  [validation_step_0, validation_step_1],
)

# to append, get the current steps with autogenerate_validation_steps and append to it
current = autogenerate_validation_steps("default")
updated = current + [validation_step_0, validation_step_1]
update_project_validation_steps(
  "default",  # project name
  updated, # append new steps
)

Documentation

For more detailed documentation on using this package please refer to the docs.

Contributing

Processor

To add a new processor, add the function to np_validator/processors.py. Ideally create a test for it in tests. Run the tests* to help ensure that no regressions have been introduced. Each processor is expected to have one required argument which is expected to be a string filepath.

Validator

To add a new validator, add the function to np_validator/validators.py. Ideally create a test for it in tests. Run the tests* to help ensure that no regressions have been introduced.

*Tests automatically run on pull_request.

Installation

pip install -U np-validator

or install with Poetry

poetry add np-validator

Makefile usage

Makefile contains a lot of functions for faster development.

1. Download and remove Poetry

To download and install Poetry run:

make poetry-download

To uninstall

make poetry-remove

2. Install all dependencies and pre-commit hooks

Install requirements:

make install

Pre-commit hooks coulb be installed after git init via

make pre-commit-install

3. Codestyle

Automatic formatting uses pyupgrade, isort and black.

make codestyle

# or use synonym
make formatting

Codestyle checks only, without rewriting files:

make check-codestyle

Note: check-codestyle uses isort, black and darglint library

Update all dev libraries to the latest version using one comand

make update-dev-deps

4. Code security

make check-safety

This command launches Poetry integrity checks as well as identifies security issues with Safety and Bandit.

make check-safety

5. Type checks

Run mypy static type checker

make mypy

6. Tests with coverage badges

Run pytest

make test

7. All linters

Of course there is a command to rule run all linters in one:

make lint

the same as:

make test && make check-codestyle && make mypy && make check-safety

8. Docker

make docker-build

which is equivalent to:

make docker-build VERSION=latest

Remove docker image with

make docker-remove

More information about docker.

9. Cleanup

Delete pycache files

make pycache-remove

Remove package build

make build-remove

Delete .DS_STORE files

make dsstore-remove

Remove .mypycache

make mypycache-remove

Or to remove all above run:

make cleanup

📈 Releases

You can see the list of available releases on the GitHub Releases page.

We follow Semantic Versions specification.

We use Release Drafter. As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.

List of labels and corresponding titles

Label	Title in Releases
enhancement, feature	🚀 Features
bug, refactoring, bugfix, fix	🔧 Fixes & Refactoring
build, ci, testing	📦 Build System & CI/CD
breaking	💥 Breaking Changes
documentation	📝 Documentation
dependencies	⬆️ Dependencies updates

You can update it in release-drafter.yml.

GitHub creates the bug, enhancement, and documentation labels for you. Dependabot creates the dependencies label. Create the remaining labels on the Issues tab of your GitHub repository, when you need them.

🛡 License

This project is licensed under the terms of the MIT license. See LICENSE for more details.

Additonal information

Initializing a new repo

Initialize your code

1. Initialize git inside your repo:

cd np-validator && git init

2. If you don't have Poetry installed run:

make poetry-download

3. Initialize poetry and install pre-commit hooks:

make install
make pre-commit-install

4. Run the codestyle:

make codestyle

5. Upload initial code to GitHub:

git add .
git commit -m ":tada: Initial commit"
git branch -M main
git remote add origin https://github.com/np_validator/np-validator.git
git push -u origin main

Set up bots

• Set up Dependabot to ensure you have the latest dependencies.
• Set up Stale bot for automatic issue closing.

Poetry

Want to know more about Poetry? Check its documentation.

Details about Poetry

Poetry's commands are very intuitive and easy to learn, like:

• poetry add numpy@latest
• poetry run pytest
• poetry publish --build

etc

Building and releasing your package

Building a new version of the application contains steps:

• Bump the version of your package poetry version <version>. You can pass the new version explicitly, or a rule such as major, minor, or patch. For more details, refer to the Semantic Versions standard.
• Make a commit to GitHub.
• Create a GitHub release.
• And... publish 🙂 poetry publish --build

🎯 What's next

Well, that's up to you 💪🏻. I can only recommend the packages and articles that helped me.

• Typer is great for creating CLI applications.
• Rich makes it easy to add beautiful formatting in the terminal.
• Pydantic – data validation and settings management using Python type hinting.
• Loguru makes logging (stupidly) simple.
• tqdm – fast, extensible progress bar for Python and CLI.
• IceCream is a little library for sweet and creamy debugging.
• orjson – ultra fast JSON parsing library.
• Returns makes you function's output meaningful, typed, and safe!
• Hydra is a framework for elegantly configuring complex applications.
• FastAPI is a type-driven asynchronous web framework.

Articles:

• Open Source Guides.
• A handy guide to financial support for open source
• GitHub Actions Documentation.
• Maybe you would like to add gitmoji to commit names. This is really funny. 😄

🚀 Features

Development features

• Supports for Python 3.9 and higher.
• Poetry as the dependencies manager. See configuration in pyproject.toml and setup.cfg.
• Automatic codestyle with black, isort and pyupgrade.
• Ready-to-use pre-commit hooks with code-formatting.
• Type checks with mypy; docstring checks with darglint; security checks with safety and bandit
• Testing with pytest.
• Ready-to-use .editorconfig, .dockerignore, and .gitignore. You don't have to worry about those things.

Deployment features

• GitHub integration: issue and pr templates.
• Github Actions with predefined build workflow as the default CI/CD.
• Everything is already set up for security checks, codestyle checks, code formatting, testing, linting, docker builds, etc with Makefile. More details in makefile-usage.
• Dockerfile for your package.
• Always up-to-date dependencies with @dependabot. You will only enable it.
• Automatic drafts of new releases with Release Drafter. You may see the list of labels in release-drafter.yml. Works perfectly with Semantic Versions specification.

Open source community features

• Ready-to-use Pull Requests templates and several Issue templates.
• Files such as: LICENSE, CONTRIBUTING.md, CODE_OF_CONDUCT.md, and SECURITY.md are generated automatically.
• Stale bot that closes abandoned issues after a period of inactivity. (You will only need to setup free plan). Configuration is here.
• Semantic Versions specification with Release Drafter.