gnutools-python 2.3.3a1

GNU Tools for python

GNUTOOLS

Modules • Code structure • Installing the application • Makefile commands • Environments • Running the application Ressources

Gnutools is a Python package that provides a few perks:

• Up to 3x speedup processing the disk by using system commands instead of python libs.
• A simple interface with optimized command executed from the system.
• A list of functions to ease the file processing in python.

You can reuse your favorite Python packages such as NumPy, SciPy and Cython to extend ZakuroCache integration.

Modules

At a granular level, Gnutools is a library that consists of the following components:

Component	Description
gnutools	Contains the implementation of Gnutools
gnutools.audio	Audio processsing
gnutools.fs	File system processing
gnutools.concurrent	Concurrent processing
gnutools.grid	Grid search
gnutools.remote	Download files from gdrive
gnutools.tests	Unit tests
gnutools.utils	Utilitaries

Code structure

from setuptools import setup
from gnutools import __version__

setup(
    name="gnutools-python",
    version=__version__,
    packages=[
        "gnutools",
        "gnutools.audio",
        "gnutools.concurrent",
        "gnutools.fs",
        "gnutools.grid",
        "gnutools.remote",
        "gnutools.tests",
        "gnutools.utils",
    ],
    long_description="".join(open("README.md", "r").readlines()),
    long_description_content_type="text/markdown",
    include_package_data=True,
    url="https://github.com/JeanMaximilienCadic/gnutools-python",
    license="MIT",
    author="Jean Maximilien Cadic",
    python_requires=">=3.6",
    install_requires=[r.rsplit()[0] for r in open("requirements.txt")],
    author_email="[e-mail]",
    description="GNU Tools for python",
    classifiers=[
        "Programming Language :: Python :: 3.6",
        "License :: OSI Approved :: MIT License",
    ],
)

Installing the application

To clone and run this application, you'll need the following installed on your computer:

• Git
• Docker Desktop
  • Install Docker Desktop on Mac
  • Install Docker Desktop on Windows
  • Install Docker Desktop on Linux
• Python

Install the package:

# Clone this repository and install the code
git clone https://github.com/JeanMaximilienCadic/gnutools-python

# Go into the repository
cd gnutools-python

Makefile commands

Exhaustive list of make commands:

install_wheels
sandbox_cpu
sandbox_gpu
build_sandbox
push_environment
push_container_sandbox
push_container_vanilla
pull_container_vanilla
pull_container_sandbox
build_vanilla
build_wheels
auto_branch 

Environments

Docker

Note

Running this application by using Docker is recommended.

To build and run the docker image

make build
make sandbox

PythonEnv

Warning

Running this application by using PythonEnv is possible but not recommended.

make install_wheels

Running the application

• Pathlib 10.1s to scan 856631 files

from pathlib import Path
results = [f for f in Path("/mnt/hdd/backup/ASR").glob("**/*.wav")]

• gnutools 3.7s to scan 856631 files

from gnutools.fs import listfiles
results = listfiles("/mnt/hdd/backup/ASR", [".wav"])

Ressources

• Vanilla: https://en.wikipedia.org/wiki/Vanilla_software
• Sandbox: https://en.wikipedia.org/wiki/Sandbox_(software_development)
• All you need is docker: https://www.theregister.com/2014/05/23/google_containerization_two_billion/
• Dev in containers : https://code.visualstudio.com/docs/remote/containers
• Delta lake partitions: https://k21academy.com/microsoft-azure/data-engineer/delta-lake/