A Python utility belt containing simple tools, a stdlib like feel, and extra batteries.
Project description
Ubelt is a small library of robust, tested, documented, and simple functions that extend the Python standard library. It has a flat API that all behaves similarly on Windows, Mac, and Linux (up to some small unavoidable differences). Almost every function in ubelt was written with a doctest. This provides helpful documentation and example usage as well as helping achieve 100% test coverage (with minor exceptions on Windows).
Goal: provide simple functions that accomplish common tasks not yet addressed by the python standard library.
Constraints: Must be low-impact pure python; it should be easy to install and use.
Method: All functions are written with docstrings and doctests to ensure that a baseline level of documentation and testing always exists (even if functions are copy/pasted into other libraries)
Motto: Good utilities lift all codes.
Read the docs here: http://ubelt.readthedocs.io/en/latest/
These are some of the tasks that ubelt’s API enables:
extended pathlib (ub.Path)
hash common data structures like list, dict, int, str, etc. (hash_data)
hash files (hash_file)
cache a block of code (Cacher, CacheStamp)
time a block of code (Timer)
show loop progress with less overhead than tqdm (ProgIter)
download a file with optional caching and hash verification (download, grabdata)
run shell commands (cmd)
find a file or directory in candidate locations (find_path, find_exe)
string-format nested data structures (repr2)
color text with ANSI tags (color_text)
horizontally concatenate multiline strings (hzcat)
make a directory if it doesn’t exist (ensuredir)
delete a file, link, or entire directory (delete)
create cross platform symlinks (symlink)
expand environment variables and tildes in path strings (expandpath)
import a module using the path to that module (import_module_from_path)
check if a particular flag or value is on the command line (argflag, argval)
get paths to cross platform data/cache/config directories (ensure_app_cache_dir, …)
memoize functions (memoize, memoize_method, memoize_property)
build ordered sets (oset)
short defaultdict and OrderedDict aliases (ddict and odict)
map a function over the keys or values of a dictionary (map_keys, map_vals)
perform set operations on dictionaries (dict_union, dict_isect, dict_diff, dict_subset, …)
perform dictionary operations like histogram, inversion, and sorting (dict_hist, invert_dict, sorted_keys, sorted_vals)
argmax/min/sort on lists and dictionaries (argmin, argsort,)
find duplicates in a list (find_duplicates)
group a sequence of items by some criterion (group_items)
Ubelt is small. Its top-level API is defined using roughly 40 lines:
from ubelt.util_arg import (argflag, argval,)
from ubelt.util_cache import (CacheStamp, Cacher,)
from ubelt.util_colors import (NO_COLOR, color_text, highlight_code,)
from ubelt.util_const import (NoParam,)
from ubelt.util_cmd import (cmd,)
from ubelt.util_dict import (AutoDict, AutoOrderedDict, ddict, dict_diff,
dict_hist, dict_isect, dict_subset, dict_union,
dzip, find_duplicates, group_items, invert_dict,
map_keys, map_vals, named_product, odict,
sorted_keys, sorted_vals, varied_values,)
from ubelt.util_download import (download, grabdata,)
from ubelt.util_download_manager import (DownloadManager,)
from ubelt.util_func import (compatible, identity, inject_method,)
from ubelt.util_format import (FormatterExtensions, repr2,)
from ubelt.util_futures import (Executor, JobPool,)
from ubelt.util_io import (delete, readfrom, touch, writeto,)
from ubelt.util_links import (symlink,)
from ubelt.util_list import (allsame, argmax, argmin, argsort, argunique,
boolmask, chunks, compress, flatten, iter_window,
iterable, peek, take, unique, unique_flags,)
from ubelt.util_hash import (hash_data, hash_file,)
from ubelt.util_import import (import_module_from_name,
import_module_from_path, modname_to_modpath,
modpath_to_modname, split_modpath,)
from ubelt.util_indexable import (IndexableWalker, indexable_allclose,)
from ubelt.util_memoize import (memoize, memoize_method, memoize_property,)
from ubelt.util_mixins import (NiceRepr,)
from ubelt.util_path import (Path, TempDir, augpath, ensuredir, expandpath,
shrinkuser, userhome,)
from ubelt.util_platform import (DARWIN, LINUX, POSIX, WIN32,
ensure_app_cache_dir, ensure_app_config_dir,
ensure_app_data_dir, find_exe, find_path,
get_app_cache_dir, get_app_config_dir,
get_app_data_dir, platform_cache_dir,
platform_config_dir, platform_data_dir,)
from ubelt.util_str import (codeblock, ensure_unicode, hzcat, indent,
paragraph,)
from ubelt.util_stream import (CaptureStdout, CaptureStream, TeeStringIO,)
from ubelt.util_time import (Timer, timestamp,)
from ubelt.util_zip import (split_archive, zopen,)
from ubelt.orderedset import (OrderedSet, oset,)
from ubelt.progiter import (ProgIter,)Installation:
Ubelt is distributed on pypi as a universal wheel and can be pip installed on Python 3.6+. Installations are tested on CPython and PyPy implementations. For Python 2.7 and 3.5, the last supported version was 0.11.1.
pip install ubelt
Note that our distributions on pypi are signed with GPG. The signing public key is D297D757; this should agree with the value in dev/public_gpg_key.
History:
Ubelt is a migration of the most useful parts of utool(https://github.com/Erotemic/utool) into a standalone module with minimal dependencies.
The utool library contains a number of useful utility functions, but it also contained non-useful functions, as well as the kitchen sink. A number of the functions were too specific or not well documented. The ubelt is a port of the simplest and most useful parts of utool.
Note that there are other cool things in utool that are not in ubelt. Notably, the doctest harness ultimately became xdoctest. Code introspection and dynamic analysis tools were ported to xinspect. The more IPython-y tools were ported to xdev. Parts of it made their way into scriptconfig. The init-file generation was moved to mkinit. Some vim and system-y things can be found in vimtk.
Development on ubelt started 2017-01-30 and development of utool mostly stopped on utool was stopped later that year, but recieved patches until about 2020. Ubelt achived 1.0.0 and removed support for Python 2.7 and 3.5 on 2022-01-07.
Function Usefulness
When I had to hand pick a set of functions that I thought were the most useful I chose these and provided some comment on why:
import ubelt as ub
ub.Cacher # configuration based on-disk cachine
ub.Path # an extension of the pathlib.Path
ub.cmd # combines the best of subprocess.Popen and os.system
ub.hash_data # hash mutable python containers, useful with Cacher to config strings
ub.repr2 # readable representations of nested data structures
ub.download # why is this not a one liner --- also see grabdata for the same thing, but builtin caching.
ub.AutoDict # one of the most useful tools in Perl, recursive default dicts of dicts
ub.JobPool # easy multi-threading / multi-procesing / or single-threaded processing
ub.modname_to_modpath # (works via static analysis)
ub.modpath_to_modname # (works via static analysis)
ub.import_module_from_path # (Unlike importlib, this does not break pytest)
ub.import_module_from_name # (Unlike importlib, this does not break pytest)But a better way might to objectively measure the frequency of usage and built a histogram of usefulness. I generated this histogram using python dev/gen_api_for_docs.py, which roughly counts the number of times I’ve used a ubelt function in another project. Note: this measure is biased towards older functions.
Function name |
Usefulness |
|---|---|
1941 |
|
264 |
|
242 |
|
208 |
|
186 |
|
184 |
|
163 |
|
159 |
|
155 |
|
152 |
|
144 |
|
142 |
|
137 |
|
127 |
|
111 |
|
109 |
|
106 |
|
106 |
|
100 |
|
98 |
|
93 |
|
90 |
|
82 |
|
78 |
|
77 |
|
76 |
|
75 |
|
69 |
|
61 |
|
60 |
|
59 |
|
53 |
|
53 |
|
47 |
|
40 |
|
36 |
|
36 |
|
36 |
|
31 |
|
30 |
|
30 |
|
30 |
|
29 |
|
28 |
|
28 |
|
27 |
|
27 |
|
27 |
|
26 |
|
26 |
|
23 |
|
23 |
|
23 |
|
23 |
|
22 |
|
22 |
|
20 |
|
20 |
|
19 |
|
18 |
|
18 |
|
17 |
|
17 |
|
17 |
|
16 |
|
16 |
|
14 |
|
14 |
|
12 |
|
9 |
|
7 |
|
7 |
|
7 |
|
6 |
|
6 |
|
5 |
|
5 |
|
4 |
|
4 |
|
4 |
|
4 |
|
3 |
|
2 |
|
2 |
|
2 |
|
1 |
|
1 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
|
0 |
Examples
The most up to date examples are the doctests. We also have a Jupyter notebook: https://github.com/Erotemic/ubelt/blob/main/docs/notebooks/Ubelt%20Demo.ipynb
Here are some examples of some features inside ubelt
Timing
Quickly time a single line.
>>> import math
>>> import ubelt as ub
>>> timer = ub.Timer('Timer demo!', verbose=1)
>>> with timer:
>>> math.factorial(100000)
tic('Timer demo!')
...toc('Timer demo!')=0.1453sLoop Progress
ProgIter is a no-threads attached Progress meter that writes to stdout. It is a mostly drop-in alternative to tqdm. The advantage of ``ProgIter`` is that it does not use any python threading, and therefore can be safer with code that makes heavy use of multiprocessing.
Note: ProgIter is also defined in a standalone module: pip install progiter)
>>> import ubelt as ub
>>> def is_prime(n):
... return n >= 2 and not any(n % i == 0 for i in range(2, n))
>>> for n in ub.ProgIter(range(1000), verbose=2):
>>> # do some work
>>> is_prime(n)
0/1000... rate=0.00 Hz, eta=?, total=0:00:00, wall=14:05 EST
1/1000... rate=82241.25 Hz, eta=0:00:00, total=0:00:00, wall=14:05 EST
257/1000... rate=177204.69 Hz, eta=0:00:00, total=0:00:00, wall=14:05 EST
642/1000... rate=94099.22 Hz, eta=0:00:00, total=0:00:00, wall=14:05 EST
1000/1000... rate=71886.74 Hz, eta=0:00:00, total=0:00:00, wall=14:05 ESTCaching
Cache intermediate results in a script with minimal boilerplate. It looks like 4 lines of boilerplate is the best you can do with Python 3.8 syntax. See <https://raw.githubusercontent.com/Erotemic/ubelt/main/ubelt/util_cache.py>`__ for details.
>>> import ubelt as ub
>>> cfgstr = 'repr-of-params-that-uniquely-determine-the-process'
>>> cacher = ub.Cacher('test_process', cfgstr)
>>> data = cacher.tryload()
>>> if data is None:
>>> myvar1 = 'result of expensive process'
>>> myvar2 = 'another result'
>>> data = myvar1, myvar2
>>> cacher.save(data)
>>> myvar1, myvar2 = dataHashing
The ub.hash_data constructs a hash corresponding to a (mostly) arbitrary ordered python object. A common use case for this function is to construct the cfgstr mentioned in the example for ub.Cacher. Instead of returning a hex, string, ub.hash_data encodes the hash digest using the 26 lowercase letters in the roman alphabet. This makes the result easy to use as a filename suffix.
>>> import ubelt as ub
>>> data = [('arg1', 5), ('lr', .01), ('augmenters', ['flip', 'translate'])]
>>> ub.hash_data(data)[0:8]
5f5fda5eThere exists an undocumented plugin architecture to extend this function to arbitrary types. See ubelt/util_hash.py for details.
Command Line Interaction
The builtin Python subprocess.Popen module is great, but it can be a bit clunky at times. The os.system command is easy to use, but it doesn’t have much flexibility. The ub.cmd function aims to fix this. It is as simple to run as os.system, but it returns a dictionary containing the return code, standard out, standard error, and the Popen object used under the hood.
>>> import ubelt as ub
>>> info = ub.cmd('gcc --version')
>>> print(ub.repr2(info))
{
'command': 'gcc --version',
'err': '',
'out': 'gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609\nCopyright (C) 2015 Free Software Foundation, Inc.\nThis is free software; see the source for copying conditions. There is NO\nwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n\n',
'proc': <subprocess.Popen object at 0x7ff98b310390>,
'ret': 0,
}Also note the use of ub.repr2 to nicely format the output dictionary.
Additionally, if you specify verbose=True, ub.cmd will simultaneously capture the standard output and display it in real time.
>>> import ubelt as ub
>>> info = ub.cmd('gcc --version', verbose=True)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.A common use case for ub.cmd is parsing version numbers of programs
>>> import ubelt as ub
>>> cmake_version = ub.cmd('cmake --version')['out'].splitlines()[0].split()[-1]
>>> print('cmake_version = {!r}'.format(cmake_version))
cmake_version = 3.11.0-rc2This allows you to easily run a command line executable as part of a python process, see what it is doing, and then do something based on its output, just as you would if you were interacting with the command line itself.
Lastly, ub.cmd removes the need to think about if you need to pass a list of args, or a string. Both will work. This utility has been tested on both Windows and Linux.
Cross-Platform Resource and Cache Directories
If you have an application which writes configuration or cache files, the standard place to dump those files differs depending if you are on Windows, Linux, or Mac. Ubelt offers a unified functions for determining what these paths are.
The ub.ensure_app_cache_dir and ub.ensure_app_resource_dir functions find the correct platform-specific location for these files and ensures that the directories exist. (Note: replacing “ensure” with “get” will simply return the path, but not ensure that it exists)
The resource root directory is ~/AppData/Roaming on Windows, ~/.config on Linux and ~/Library/Application Support on Mac. The cache root directory is ~/AppData/Local on Windows, ~/.config on Linux and ~/Library/Caches on Mac.
Example usage on Linux might look like this:
>>> import ubelt as ub
>>> print(ub.shrinkuser(ub.ensure_app_cache_dir('my_app')))
~/.cache/my_app
>>> print(ub.shrinkuser(ub.ensure_app_resource_dir('my_app')))
~/.config/my_appPaths
Ubelt extends pathlib.Path by adding several new (often chainable) methods. Namely, augment, delete, expand, ensuredir, shrinkuser. It also modifies behavior of touch to be chainable.
>>> # Ubelt extends pathlib functionality
>>> import ubelt as ub
>>> dpath = ub.Path('~/.cache/ubelt/demo_path').expand().ensuredir()
>>> fpath = dpath / 'text_file.txt'
>>> aug_fpath = fpath.augment(suffix='.aux', ext='.jpg').touch()
>>> aug_dpath = dpath.augment('demo_path2')
>>> assert aug_fpath.read_text() == ''
>>> fpath.write_text('text data')
>>> assert aug_fpath.exists()
>>> assert not aug_fpath.delete().exists()
>>> assert dpath.exists()
>>> assert not dpath.delete().exists()
>>> print(f'{fpath.shrinkuser()}')
>>> print(f'{dpath.shrinkuser()}')
>>> print(f'{aug_fpath.shrinkuser()}')
>>> print(f'{aug_dpath.shrinkuser()}')
~/.cache/ubelt/demo_path/text_file.txt
~/.cache/ubelt/demo_path
~/.cache/ubelt/demo_path/text_file.aux.jpg
~/.cache/ubelt/demo_pathdemo_path2Symlinks
The ub.symlink function will create a symlink similar to os.symlink. The main differences are that 1) it will not error if the symlink exists and already points to the correct location. 2) it works* on Windows (*hard links and junctions are used if real symlinks are not available)
>>> import ubelt as ub
>>> dpath = ub.ensure_app_cache_dir('ubelt', 'demo_symlink')
>>> real_path = join(dpath, 'real_file.txt')
>>> link_path = join(dpath, 'link_file.txt')
>>> ub.writeto(real_path, 'foo')
>>> ub.symlink(real_path, link_path)Downloading Files
The function ub.download provides a simple interface to download a URL and save its data to a file.
>>> import ubelt as ub
>>> url = 'http://i.imgur.com/rqwaDag.png'
>>> fpath = ub.download(url, verbose=0)
>>> print(ub.shrinkuser(fpath))
~/.cache/ubelt/rqwaDag.pngThe function ub.grabdata works similarly to ub.download, but whereas ub.download will always re-download the file, ub.grabdata will check if the file exists and only re-download it if it needs to.
>>> import ubelt as ub
>>> url = 'http://i.imgur.com/rqwaDag.png'
>>> fpath = ub.grabdata(url, verbose=0, hash_prefix='944389a39')
>>> print(ub.shrinkuser(fpath))
~/.cache/ubelt/rqwaDag.pngNew in version 0.4.0: both functions now accepts the hash_prefix keyword argument, which if specified will check that the hash of the file matches the provided value. The hasher keyword argument can be used to change which hashing algorithm is used (it defaults to "sha512").
Grouping
Group items in a sequence into a dictionary by a second id list
>>> import ubelt as ub
>>> item_list = ['ham', 'jam', 'spam', 'eggs', 'cheese', 'bannana']
>>> groupid_list = ['protein', 'fruit', 'protein', 'protein', 'dairy', 'fruit']
>>> ub.group_items(item_list, groupid_list)
{'dairy': ['cheese'], 'fruit': ['jam', 'bannana'], 'protein': ['ham', 'spam', 'eggs']}Dictionary Histogram
Find the frequency of items in a sequence
>>> import ubelt as ub
>>> item_list = [1, 2, 39, 900, 1232, 900, 1232, 2, 2, 2, 900]
>>> ub.dict_hist(item_list)
{1232: 2, 1: 1, 2: 4, 900: 3, 39: 1}Find Duplicates
Find all duplicate items in a list. More specifically, ub.find_duplicates searches for items that appear more than k times, and returns a mapping from each duplicate item to the positions it appeared in.
>>> import ubelt as ub
>>> items = [0, 0, 1, 2, 3, 3, 0, 12, 2, 9]
>>> ub.find_duplicates(items, k=2)
{0: [0, 1, 6], 2: [3, 8], 3: [4, 5]}Dictionary Manipulation
Take a subset of a dictionary.
>>> import ubelt as ub
>>> dict_ = {'K': 3, 'dcvs_clip_max': 0.2, 'p': 0.1}
>>> subdict_ = ub.dict_subset(dict_, ['K', 'dcvs_clip_max'])
>>> print(subdict_)
{'K': 3, 'dcvs_clip_max': 0.2}Take only the values, optionally specify a default value.
>>> import ubelt as ub
>>> dict_ = {1: 'a', 2: 'b', 3: 'c'}
>>> print(list(ub.take(dict_, [1, 2, 3, 4, 5], default=None)))
['a', 'b', 'c', None, None]Apply a function to each value in the dictionary (see also ub.map_keys).
>>> import ubelt as ub
>>> dict_ = {'a': [1, 2, 3], 'b': []}
>>> newdict = ub.map_vals(len, dict_)
>>> print(newdict)
{'a': 3, 'b': 0}Invert the mapping defined by a dictionary. By default invert_dict assumes that all dictionary values are distinct (i.e. the mapping is one-to-one / injective).
>>> import ubelt as ub
>>> mapping = {0: 'a', 1: 'b', 2: 'c', 3: 'd'}
>>> ub.invert_dict(mapping)
{'a': 0, 'b': 1, 'c': 2, 'd': 3}However, by specifying unique_vals=False the inverted dictionary builds a set of keys that were associated with each value.
>>> import ubelt as ub
>>> mapping = {'a': 0, 'A': 0, 'b': 1, 'c': 2, 'C': 2, 'd': 3}
>>> ub.invert_dict(mapping, unique_vals=False)
{0: {'A', 'a'}, 1: {'b'}, 2: {'C', 'c'}, 3: {'d'}}AutoDict - Autovivification
While the collections.defaultdict is nice, it is sometimes more convenient to have an infinitely nested dictionary of dictionaries.
>>> import ubelt as ub
>>> auto = ub.AutoDict()
>>> print('auto = {!r}'.format(auto))
auto = {}
>>> auto[0][10][100] = None
>>> print('auto = {!r}'.format(auto))
auto = {0: {10: {100: None}}}
>>> auto[0][1] = 'hello'
>>> print('auto = {!r}'.format(auto))
auto = {0: {1: 'hello', 10: {100: None}}}String-based imports
Ubelt contains functions to import modules dynamically without using the python import statement. While importlib exists, the ubelt implementation is simpler to user and does not have the disadvantage of breaking pytest.
Note ubelt simply provides an interface to this functionality, the core implementation is in xdoctest (over as of version 0.7.0, the code is statically copied into an autogenerated file such that ubelt does not actually depend on xdoctest during runtime).
>>> import ubelt as ub
>>> module = ub.import_module_from_path(ub.expandpath('~/code/ubelt/ubelt'))
>>> print('module = {!r}'.format(module))
module = <module 'ubelt' from '/home/joncrall/code/ubelt/ubelt/__init__.py'>
>>> module = ub.import_module_from_name('ubelt')
>>> print('module = {!r}'.format(module))
module = <module 'ubelt' from '/home/joncrall/code/ubelt/ubelt/__init__.py'>Related to this functionality are the functions ub.modpath_to_modname and ub.modname_to_modpath, which statically transform (i.e. no code in the target modules is imported or executed) between module names (e.g. ubelt.util_import) and module paths (e.g. ~/.local/conda/envs/cenv3/lib/python3.5/site-packages/ubelt/util_import.py).
>>> import ubelt as ub
>>> modpath = ub.util_import.__file__
>>> print(ub.modpath_to_modname(modpath))
ubelt.util_import
>>> modname = ub.util_import.__name__
>>> assert ub.modname_to_modpath(modname) == modpathHorizontal String Concatenation
Sometimes its just prettier to horizontally concatenate two blocks of text.
>>> import ubelt as ub
>>> B = ub.repr2([[1, 2], [3, 4]], nl=1, cbr=True, trailsep=False)
>>> C = ub.repr2([[5, 6], [7, 8]], nl=1, cbr=True, trailsep=False)
>>> print(ub.hzcat(['A = ', B, ' * ', C]))
A = [[1, 2], * [[5, 6],
[3, 4]] [7, 8]]External tools
Some of the tools in ubelt also exist as standalone modules. I haven’t decided if its best to statically copy them into ubelt or require on pypi to satisfy the dependency. There are some tools that are not used by default unless you explicitly allow for them.
Code that is currently statically included:
ProgIter - https://github.com/Erotemic/progiter
Code that is currently linked via pypi:
OrderedSet - https://github.com/LuminosoInsight/ordered-set
Code that is completely optional, and only used in specific cases:
Numpy - ub.repr2 will format a numpy array nicely by default
xxhash - this can be specified as a hasher to ub.hash_data
Pygments - used by the util_color module.
Similar Tools
UBelt is one of many Python utility libraries. Other similar libraries are listed here.
Libraries that contain a broad scope of utilities:
Boltons: https://github.com/mahmoud/boltons
Libraries that contain a specific scope of utilities:
More-Itertools: https://pypi.org/project/more-itertools/
CyToolz: https://github.com/pytoolz/cytoolz/
Libraries that contain one specific data structure or utility:
timerit - https://github.com/Erotemic/timerit
Notes.
PRs are welcome.
Also check out my other projects (many of which are powered by ubelt):
ProgIter https://github.com/Erotemic/progiter
xdoctest https://github.com/Erotemic/xdoctest
xinspect https://github.com/Erotemic/xinspect
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for ubelt-1.0.1-py2.py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 69b44438c1d3a113de5c46d90dda2509b40b3e46b06853a5205d4cc868b97f7f |
|
| MD5 | 940a0528c5e8b736b43526493e3c5f29 |
|
| BLAKE2b-256 | 5e1c6ce7070b1f000cd52ffda71d0492ae6d641ad5e97a487bdc060c18737fc6 |
