Skip to main content

data_manifestor

Build status Python Version Dependencies Status

Code style: black Security: bandit Pre-commit Semantic Versions License Coverage Report

Awesome data_manifestor is a Python cli/package created with https://github.com/TezRomacH/python-package-template

API

Run a local comparison.

from data_manifestor import compare_manifest_to_local

# By default if no template_dict is supplied, it will be inferred based on meta information serialized in the local path
result = compare_manifest_to_local(
  "some/path/to/local/experiment/directory",
)

To change what manifestor templates get inferred, you will need to update template associated with the project you want to affect.

from data_manifestor import Template, update_project_data_manifestor_template

template = Template(
  name="some sort of name for this template",
  path_patterns=[
    "parent/dir/of/path/you/want/to/check/exists",
  ]
)

update_project_data_manifestor_template(
  "pretest",
  template,
)

Documentation

For more detailed documentation on using this package please refer to the docs.

Installation

pip install -U data_manifestor

or install with Poetry

poetry add data_manifestor

Then you can run

data_manifestor --help

or with Poetry:

poetry run data_manifestor --help

Makefile usage

Makefile contains a lot of functions for faster development.

1. Download and remove Poetry

To download and install Poetry run:

make poetry-download

To uninstall

make poetry-remove

2. Install all dependencies and pre-commit hooks

Install requirements:

make install

Pre-commit hooks coulb be installed after git init via

make pre-commit-install

3. Codestyle

Automatic formatting uses pyupgrade, isort and black.

make codestyle

# or use synonym
make formatting

Codestyle checks only, without rewriting files:

make check-codestyle

Note: check-codestyle uses isort, black and darglint library

Update all dev libraries to the latest version using one comand

make update-dev-deps
4. Code security

make check-safety

This command launches Poetry integrity checks as well as identifies security issues with Safety and Bandit.

make check-safety

5. Type checks

Run mypy static type checker

make mypy

6. Tests with coverage badges

Run pytest

make test

7. All linters

Of course there is a command to rule run all linters in one:

make lint

the same as:

make test && make check-codestyle && make mypy && make check-safety

8. Docker

make docker-build

which is equivalent to:

make docker-build VERSION=latest

Remove docker image with

make docker-remove

More information about docker.

9. Cleanup

Delete pycache files

make pycache-remove

Remove package build

make build-remove

Delete .DS_STORE files

make dsstore-remove

Remove .mypycache

make mypycache-remove

Or to remove all above run:

make cleanup

📈 Releases

You can see the list of available releases on the GitHub Releases page.

We follow Semantic Versions specification.

We use Release Drafter. As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.

List of labels and corresponding titles

Label Title in Releases
enhancement, feature 🚀 Features
bug, refactoring, bugfix, fix 🔧 Fixes & Refactoring
build, ci, testing 📦 Build System & CI/CD
breaking 💥 Breaking Changes
documentation 📝 Documentation
dependencies ⬆️ Dependencies updates

You can update it in release-drafter.yml.

GitHub creates the bug, enhancement, and documentation labels for you. Dependabot creates the dependencies label. Create the remaining labels on the Issues tab of your GitHub repository, when you need them.

🛡 License

License

This project is licensed under the terms of the MIT license. See LICENSE for more details.

Additional information

Initializing a new repo

Initialize your code

  1. Initialize git inside your repo:
cd data_manifestor && git init
  1. If you don't have Poetry installed run:
make poetry-download
  1. Initialize poetry and install pre-commit hooks:
make install
make pre-commit-install
  1. Run the codestyle:
make codestyle
  1. Upload initial code to GitHub:
git add .
git commit -m ":tada: Initial commit"
git branch -M main
git remote add origin https://github.com/data_manifestor/data_manifestor.git
git push -u origin main

Set up bots

  • Set up Dependabot to ensure you have the latest dependencies.
  • Set up Stale bot for automatic issue closing.

Poetry

Want to know more about Poetry? Check its documentation.

Details about Poetry

Poetry's commands are very intuitive and easy to learn, like:

  • poetry add numpy@latest
  • poetry run pytest
  • poetry publish --build

etc

Building and releasing your package

Building a new version of the application contains steps:

  • Bump the version of your package poetry version <version>. You can pass the new version explicitly, or a rule such as major, minor, or patch. For more details, refer to the Semantic Versions standard.
  • Make a commit to GitHub.
  • Create a GitHub release.
  • And... publish 🙂 poetry publish --build

🎯 What's next

Well, that's up to you 💪🏻. I can only recommend the packages and articles that helped me.

  • Typer is great for creating CLI applications.
  • Rich makes it easy to add beautiful formatting in the terminal.
  • Pydantic – data validation and settings management using Python type hinting.
  • Loguru makes logging (stupidly) simple.
  • tqdm – fast, extensible progress bar for Python and CLI.
  • IceCream is a little library for sweet and creamy debugging.
  • orjson – ultra fast JSON parsing library.
  • Returns makes you function's output meaningful, typed, and safe!
  • Hydra is a framework for elegantly configuring complex applications.
  • FastAPI is a type-driven asynchronous web framework.

Articles:

🚀 Features

Development features

Deployment features

Open source community features

Metadata

Release files for data-manifestor 0.5.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for data-manifestor 0.5.6
File Size Uploaded Provenance Actions
data_manifestor-0.5.6.tar.gz 15.7 kB Without provenance Details for data_manifestor-0.5.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for data-manifestor 0.5.6
File Interpreter ABI Platform Actions
data_manifestor-0.5.6-py3-none-any.whl Python 3 none any Details for data_manifestor-0.5.6-py3-none-any.whl

Total release size: 26.6 kB

Release files / data_manifestor-0.5.6.tar.gz

Download URL data_manifestor-0.5.6.tar.gz
Size 15.7 kB
Tags Source
SHA-256 checksum
How to use checksums
da9f6addbe292663d836d90c90af367233f5b2d7b085574d83697b8dd62993ee
BLAKE2b-256 checksum
How to use checksums
2530e72a92db897aa72456ee3982283c802c96669fe8aa332aa7f95bdb867094
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.14 CPython/3.9.13 Darwin/20.1.0

Release files / data_manifestor-0.5.6-py3-none-any.whl

Download URL data_manifestor-0.5.6-py3-none-any.whl
Size 10.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
edd4df71b5596297ae4c84ede99d06751820d0ad9b8565b6f1e6f07025e2f29b
BLAKE2b-256 checksum
How to use checksums
d897ecaa0de933acad56615a52aa794fb4b34f1f0e92d5e5ec123a9a470bc606
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.1.14 CPython/3.9.13 Darwin/20.1.0

Security for data-manifestor 0.5.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page