jacktrade 0.13.0

A collection of commonly used Python utilities.

jacktrade

Jack of all trades, master of none - a collection of commonly used Python utilities. Install using:

pip install jacktrade

The package consists of the following submodules:

• Benchmark
• Buffers
• Cache
• Collections
• Files
• Multicore
• Pickler
• Sysenv

Benchmark

Contains a CodeTimer class which is used to elegantly and precisely time a piece of code:

from jacktrade import CodeTimer
from time import sleep

with CodeTimer() as ct:
    # Enter code to time here
    sleep(0.1)  # Simulates a piece of code

# Prints: "Code execution took 100 ms."
(ct.ns, ct.us, ct.ms, ct.s)  # Access code duration in nano/micro/milli/seconds.

A CodeTimer instance can also be used as a decorator and store results of repeated function calls inside the list provided as results argument:

from jacktrade import CodeTimer
from time import sleep

results = []  # A list for storing timing results

@CodeTimer(no_print=True, results=results)
def sleep_ms(milliseconds: int):
    sleep(milliseconds / 1000)

sleep_ms(100)  # Call wrapped function
sleep_ms(200)  # Call again
print(int(results[0].ms))  # Prints: "100"
print(int(results[1].ms))  # Prints: "200"

Buffers

Contains a StringBuffers class, whose purpose is to reduce the number of I/O operations when writing to files. By speficying buffer_size parameter, the contents of the buffer are automatically flushed to disk when the buffer fills up. The class handles any number of simultaneously "open" files.

from jacktrade import StringBuffers

output_file = "out.txt"
buffers = StringBuffers(output_dir="text", buffer_size=3)
buffers.add(output_file, "Hello")   # Nothing is written out
buffers.add(output_file, " world")  # Nothing is written out
buffers.add(output_file, "!")  # "Hello world!" is written to ./text/out.txt

Cache

Contains a @cached_method decorator which can be used to cache the calls of instance and class methods. The cache is attached to the object's instance (or the class in case of class methods), so it gets garbage collected together with the object.

from jacktrade import cached_method
from time import sleep

class Worker:

    @cached_method
    def instance_method(self, base, exp):
        sleep(base**exp)  # Long-running task

    @classmethod
    @cached_method
    def class_method(cls, base, exp):
        sleep(base**exp)  # Long-running task


w = Worker()
w.instance_method(2, 5)  # Takes 2^5 = 32 seconds to return
w.instance_method(2, 5)  # Returns immediately

Worker.class_method(3, 3)  # Takes 3^3 = 27 seconds to return
Worker.class_method(3, 3)  # Returns immediately

Collections

Contains utility functions for working with collections, namely dictionaries and iterables. Usage examples include:

from jacktrade import *

# Dict utilities
dict_data = {"a": 1, "b": {"c": 2}}
flatten_dict(dict_data)             # Returns: [1, 2]
get_first_dict_item(dict_data)      # Returns: ("a", 1)
get_first_dict_key(dict_data)       # Returns: "a"
get_first_dict_value(dict_data)     # Returns: 1

# Iterable utilities
list_data = [1, 2, [3, 4], 5, 6]
flatten_list(list_data)             # Returns: [1, 2, 3, 4, 5, 6]
chunkify(list_data, chunk_size=2)   # Yields: [1, 2], [[3, 4], 5], [6]
limit_iterator(list_data, limit=3)  # Yields: 1, 2, [3, 4]

MasterDict is a wrapper class holding multiple dictionaries. It provides methods for simultaneously deleting keys from all underlying dictionaries, as well as clearing them. It is intended to hold caches and reliably empty them with a single method call.

from jacktrade import MasterDict

dicts = MasterDict(a={1: "1", 2: "2"}, b={1: "1", 3: "3"}, c={4: "4"})
dicts.delete_keys(1, 3)  # Delete keys 1 and 3 from all dictionaries
dicts.as_dict()  # Returns:  {'a': {2: '2'}, 'b': {}, 'c': {4: '4'}}
dicts.clear_all()  # Clear all dictionaries
dicts.as_dict()  # Returns:  {'a': {}, 'b': {}, 'c': {}}

BaseMapping is a generic base class used to create dict subclasses which automatically map keys to values from a collection of objects of the same type. It is used like so:

from jacktrade import BaseMapping

class NameAgeLookup(BaseMapping):
    """
    Maps a person's name to its age if the person is over 18 years old.
    """

    def __init__(self, persons):
        super().__init__(
            items=persons,
            key_getter=lambda p: p["name"],
            value_getter=lambda p: p["age"],
            condition=lambda p: p["age"] > 18,
        )

mapping = NameAgeLookup(
    [
        {"name": "Jack", "age": 15},
        {"name": "Mike", "age": 27},
        {"name": "Pete", "age": 39},
    ]
)

# Assertions pass
assert mapping == {"Mike": 27, "Pete": 39}
assert mapping.invert() == {27: "Mike", 39: "Pete"}

Permutations class is used for parametrisation, returning all possible combinations of input parameters:

from jacktrade import Permutations

p = Permutations(a=[1, 2, 3], b=["A", "B"])
p.args    # Returns: [(1, "A"), (1, "B"), (2, "A"), (2, "B"), ...]
p.kwargs  # Returns: [{"a": 1, "b": "A"}, {"a": 1, "b": "B"}, ...]
for kwargs in p:
    # yields:
    # {"a": 1, "b": "A"}
    # {"a": 1, "b": "B"}
    # ...

Files

Provides utilities for working with files. Currently it contains only a single function for merging CSV files.

from jacktrade import merge_csv_files

# Merges A.csv and B.csv into AB.csv without duplicating headers
merge_csv_files(["A.csv", "B.csv"], "AB.csv")

# Merges A.csv and B.csv into AB.csv verbatim, treating headers as data
merge_csv_files(["A.csv", "B.csv"], "AB.csv", has_headers=False)

Command line utility

The package also provides a command line utility to do the same actions as above:

merge csv AB.csv A.csv B.csv  # Merge A.csv and B.csv into AB.csv
merge csv --no-headers AB.csv A.csv B.csv  # Treat headers as data

The command line utility also allows the user to specify directories and glob patterns, like so:

merge csv merged.csv dir1 dir2  # Merges all *.csv files in dir1 & dir2 (if they exist) into merged.csv
merge csv --glob merged.csv "1*.csv" "3*.csv" # Merges all CSV files whose names begin with 1 or 3

Multicore

Provides an elegant and memory-efficient way to process data using multiple cores. The main advantage of using do_multicore_work function over manually using concurrent.futures or multiprocessing modules is that new jobs are only submitted for execution when a CPU core is available. This optimises CPU and RAM usage. Using the aforementioned modules directly, it is all too easy to inadvarently cause memory leaks and crash the interpreter (if not the whole system).

Usage example (does not work in the interactive interpreter):

from jacktrade import do_multicore_work

def worker(first, second) -> tuple:
    """Receives two arguments and returns them as a tuple."""
    return (first, second)

def worker_done_callback(future):
    """Called whenever a worker process terminates and returns a result."""
    print(future.result())

if __name__ == "__main__":
    do_multicore_work(
        worker, args=[(1, 2), (3, 4), (5, 6)], worker_done_callback=worker_done_callback
    )    # Prints: (1, 2)\n(3, 4)\n(5, 6)\n

Pickler

This tiny module contains two convenience functions for pickling and unpickling Python objects, making it possible to do so with a single function call (a feature missing from pickle module):

from jacktrade import pickle_object, unpickle_object

pickle_object(obj := [1, 2, 3], filename := "obj.pickle")   # Pickles obj to obj.pickle file
assert unpickle_object(filename) == obj     # Unpickle obj.pickle and test equality with obj

Sysenv

Contains utilities for interacting with the operating system and the environment.

from jacktrade import in_virtual_environment, suspend, hibernate, restart, shutdown

# Power management
suspend()    # Put the machine into standby
hibernate()  # Hibernate the machine
restart()    # Restart the machine
shutdown()   # Shut down the machine

# Miscellaneous
in_virtual_environment()  # True if called inside venv, else False