GNU Tools for python
Project description
GNUTOOLS
Modules • Code structure • Installing the application • Makefile commands • Environments • Running the application Ressources
Gnutools is a Python package that provides a few perks:
- Up to 3x speedup processing the disk by using system commands instead of python libs.
- A simple interface with optimized command executed from the system.
- A list of functions to ease the file processing in python.
You can reuse your favorite Python packages such as NumPy, SciPy and Cython to extend ZakuroCache integration.
Modules
At a granular level, Gnutools is a library that consists of the following components:
Component | Description |
---|---|
gnutools | Contains the implementation of Gnutools |
gnutools.audio | Audio processsing |
gnutools.fs | File system processing |
gnutools.concurrent | Concurrent processing |
gnutools.grid | Grid search |
gnutools.remote | Download files from gdrive |
gnutools.tests | Unit tests |
gnutools.utils | Utilitaries |
Code structure
from setuptools import setup
from gnutools import __version__
setup(
name="gnutools-python",
version=__version__,
packages=[
"gnutools",
"gnutools.audio",
"gnutools.concurrent",
"gnutools.fs",
"gnutools.grid",
"gnutools.remote",
"gnutools.tests",
"gnutools.utils",
],
long_description="".join(open("README.md", "r").readlines()),
long_description_content_type="text/markdown",
include_package_data=True,
url="https://github.com/JeanMaximilienCadic/gnutools-python",
license="MIT",
author="Jean Maximilien Cadic",
python_requires=">=3.6",
install_requires=[r.rsplit()[0] for r in open("requirements.txt")],
author_email="git@cadic.jp",
description="GNU Tools for python",
classifiers=[
"Programming Language :: Python :: 3.6",
"License :: OSI Approved :: MIT License",
],
)
Installing the application
To clone and run this application, you'll need the following installed on your computer:
Install the package:
# Clone this repository and install the code
git clone https://github.com/JeanMaximilienCadic/gnutools-python
# Go into the repository
cd gnutools-python
Makefile commands
Exhaustive list of make commands:
install_wheels
sandbox_cpu
sandbox_gpu
build_sandbox
push_environment
push_container_sandbox
push_container_vanilla
pull_container_vanilla
pull_container_sandbox
build_vanilla
build_wheels
auto_branch
Environments
Docker
Note
Running this application by using Docker is recommended.
To build and run the docker image
make build
make sandbox
PythonEnv
Warning
Running this application by using PythonEnv is possible but not recommended.
make install_wheels
Running the application
- Pathlib
10.1s
to scan856631
files
from pathlib import Path
results = [f for f in Path("/mnt/hdd/backup/ASR").glob("**/*.wav")]
- gnutools
3.7s
to scan856631
files
from gnutools.fs import listfiles
results = listfiles("/mnt/hdd/backup/ASR", [".wav"])
Ressources
- Vanilla: https://en.wikipedia.org/wiki/Vanilla_software
- Sandbox: https://en.wikipedia.org/wiki/Sandbox_(software_development)
- All you need is docker: https://www.theregister.com/2014/05/23/google_containerization_two_billion/
- Dev in containers : https://code.visualstudio.com/docs/remote/containers
- Delta lake partitions: https://k21academy.com/microsoft-azure/data-engineer/delta-lake/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
File details
Details for the file gnutools_python-2.3.5-py3-none-any.whl
.
File metadata
- Download URL: gnutools_python-2.3.5-py3-none-any.whl
- Upload date:
- Size: 12.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.8.0 colorama/0.4.4 importlib-metadata/4.6.4 keyring/23.5.0 pkginfo/1.8.2 readme-renderer/34.0 requests-toolbelt/0.9.1 requests/2.25.1 rfc3986/1.5.0 tqdm/4.64.1 urllib3/1.26.5 CPython/3.10.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | ede5203473c935fa9803bc6958bdd329d9095b58745a596c018a99a5c5ca00d6 |
|
MD5 | bfb6a2a1800a6d6a84dcf63365e32366 |
|
BLAKE2b-256 | 29d84e275ad718235eec3bba9d12cb2764d879ff21d303c5d8ba4482fe26b2bf |