ensure_dir(directory)

Create directory if it doesn’t exist.

expand(directory)

Expand ~ to $HOME, and expand other environment variables to their values, in the provided path. Designed for UNIX-based platforms: unclear what will happen on others.

generate_downloader(headers, args, max_per_hour=30)

Produce a callable downloader. Provide headers for URL call, and provide some args (for the rate limiting (see that later in the document)), which will be used to identify this function in rate-limiting. max_per_hour is optional: use it to decide how many times per hour downloads can happen. The default is 30 times per hour. The returned downloader is called as follows:

downloader = generate_downloader({"foo": "bar"}, "an identifier string")
downloader(url=url, dest=dest)

where url is where to download FROM and dest is where to SAVE the file. It will make the directory for the file if it does not exist. A text progress bar is shown during download, and while rate limit is blocking.

max_clamp(val, limit)

Return val if it is less than limit, otherwise return limit.

parse_int_string(int_string)

Given a string like “1 23 4-8 32 1”, return a unique list of those integers in the string, and the integers in the ranges in the string. For this example, the output would be [1, 4, 5, 6, 7, 8, 23, 32]. Non-numbers ignored. Not necessarily sorted

rate_limited(max_per_hour, *args)

A decorator to rate-limit a function (ensures that it runs no more than max_per_hour times per hour by sleeping sometimes). Called like this:

@util.rate_limited(120, name)
def a_function(var1, var2="foo"):

sanitize(filename, platform=None)

Remove disallowed characters from potential filename. Provide platform to sanitize for that platform, otherwise current platform is found and used.

set_up_logging(log_filename="log", verbosity=0)

Set up a logger with reasonable settings. Return it for log calls.

random_line(file_path, encoding=FORCED_ENCODING)

At time of writing, FORCED_ENCODING is “UTF-8”. Refer to code for latest. Return a line from the provided file at random.