uxf 0.35.0

A pure Python library supporting Uniform eXchange Format, a plain text human readable optionally typed storage format.

Python UXF Library

UXF is a plain text human readable optionally typed storage format. UXF may serve as a convenient alternative to csv, ini, json, sqlite, toml, xml, or yaml.

For details of the Uniform eXchange Format (UXF) supported by this library, see the UXF Overview. (PyPI link to UXF Overview.)

• Introduction
• Python UXF Types
• API
  • Reading and Writing UXF Data
  • API Notes
  • Classes
  • Functions
  • Constants
  • Command Line Usage

Introduction

The Python uxf library works out of the box with the standard library, and one dependency, editabletuple. And it will use dateutil available.

• Install: python3 -m pip install uxf
• Run: python3 -m uxf -h # this shows the command line help
• Use: import uxf # see the uxf.py module docs for the API

Using uxf as an executable (e.g., python3 -m uxf ...) provides a means of doing .uxf to .uxf conversions (e.g., compress or uncompress or to use the standard pretty print format). The executable can also be used for linting and for replacing imports to ensure that UXF files are stand-alone.

Installed alongside uxf.py are uxflint.py, and uxfconvert.py which might prove useful to see how to use uxf. For example, uxfconvert.py can losslessly convert .uxf to .json or .xml and back. It can also do some simple conversions to and from .csv, to .ini, and to and from .sqlite, but these are really to illustrate use of the uxf APIs. And also see the UXF test files in the ../testdata folder, the Python examples in the eg folder, and the Python tests in the t folder.

If you just want to create a small standalone .pyz, you could simply copy py/uxf.py as uxf.py into your project folder and include it in your .pyz file.

Python UXF Types

Most Python types convert losslessly to and from UXF types. In particular:

Python Type	UXF type
None	null
bool	bool
bytes	bytes
bytearray	bytes
datetime.date	date
datetime.datetime	datetime
int	int
float	real
str	str
uxf.List	list
uxf.Map	map
uxf.Table	table

A List is a Python collections.UserList subclass with .data (the list), .comment and .vtype attributes. A .vtype holds a str representing the name of the corresponding UXF type (e.g., int, real, etc.) or a ttype (the name of a custom table type; see TClass). If a .vtype is None it means that any UXF type may be stored.

Similarly a Map is a Python collections.UserDict subclass with .data (the dict), .comment, .ktype, and .vtype attributes. The .ktype is a str like the .vtype except that it may only be None (accept any valid ktype) or one of bytes, date, datetime, int, or str. The .vtype works just the same as for Lists.

The Table class has .records, .comment, and .tclass attributes; with .tclass holding a TClass which in turn holds the table's ttype (i.e., its table type name), and the table's field names and (optional) types. In all cases a type of None signifies that any type valid for the context may be used.

Non-UXF types can be represented using ttypes. For example, for points you could define a ttype such as: =point x:real y:real. Then you could include single points like (point 1.5 7.2), or many of them such as (point 1.5 7.2 8.3 -9.4 14.8 0.6).

Collection types such as set, frozenset, tuple, or collections.deque are automatically converted to a List when they are encountered. Of course, these are one way conversions.

API

The simplest part of the API loads and saves (dumps) UXF data from/to strings or files.

Reading and Writing UXF Data

load(filename_or_filelike): -> uxo
loads(uxt): -> uxo

The load() function reads UXF data from a file or file-like object, and the loads() function reads UXF data from a string. The returned uxo is of type Uxf.

dump(filename_or_filelike, data)
dumps(data) -> uxt

The dump() function writes the data to a file or file-like object, and the dumps() function writes the data into a string that's then returned (here called uxt to indicate UXF text). The data can be a Uxf object or a single list, List, dict, Map, or Table.

If the data contains values of types that aren't supported by UXF, they could either be transformed in advance (e.g., to a custom table type, a ttype).

See also the examples in the eg folder and the tests in the t folder.

API Notes

A UXF object (called a uxo in these docs) has a .data attribute that is always a List or Map or Table. The first two have essentially the same APIs as list and dict respectively. The Table API is a little similar to a list. Individual records can be accessed using [], but to work with records or fields the API provides get_record(), set_record(), get_field(), set_field(), amongst others. For small tables, records can be accessed with the first, second, third, or fourth properties. The best way to append new records is to use the Table's append() method.

The uxf module distinguishes between a ttype (the name of a user-defined table) and a TClass (the Python class which represents a user-defined table). A TClass has .ttype and .fields attributes.

For reading UXF data it is easiest to iterate, and to do so recursively, if the data has nested collections. Note that the visit.py example provides a way of doing this using the visitor design pattern. The uxfconvert.py example shows alternative approaches to manual iteration.

Note that for imports, if the filename is relative it is searched for in the same folder as the importing UXF file, and if not found there, each of the paths in UXF_PATH is tried in turn (if any).

Note that the __version__ is the module version (i.e., the version of this implementation), while the VERSION is the maximum UXF version that this module can read (and the UXF version that it writes).

Classes

The classes are documented in importance order. Here are alphabetically ordered links: Error, Field, List, Map, TClass, Table, Uxf.

Uxf

A Uxf object represents UXF data.

The easiest way to obtain a Uxf object is to use one of the module-level functions, load() or loads(). Of course, you can also create Uxfs programatically (as shown in many of the examples and tests).

The .data attribute holds a List, Map, or Table. Since these classes can nest, a Uxf can represent any arbitrary data structure.

The .custom attribute is an opional str used for customizing the file format.

The .tclasses attribute is a possibly empty dict where each key is a ttype (i.e., the .tclass.ttype which is a TClass's name) and each value is a TClass object.

The .comment is an optional file-level str.

The .imports attribute is a possibly empty dict where each key is a ttype and where each value is a str holding that type's import text.

The .import_filenames property is a utility useful for some UXF processors. it yields all the unique import filenames.

To create a Uxf programmatically (rather than by using, say, load(), you can either create it empty, or with some data and optionally, some ttypes.

data = ... # a list, uxf.List, dict, uxf.Map, or uxf.Table
point_ttype = uxf.TClass('point', (uxf.Field('x', 'real'), uxf.Field('y', 'real')))
uxo = uxf.Uxf(data, tclasses={point_ttype.ttype: point_ttype})

.load(filename_or_filelike, *, on_error=on_error)

Convenience method that wraps the module-level load() function.

.loads(uxt, filename='-', *, on_error=on_error)

Convenience method that wraps the module-level loads() function.

.dump(filename_or_filelike, *, on_error=on_error)

Convenience method that wraps the module-level dump() function.

.dumps(*, on_error=on_error)

Convenience method that wraps the module-level dumps() function.

List

This class is used to represent a UXF list. It is a collections.UserList subclass that also has .comment and .vtype attributes.

See Python UXF Types for more about vtypes.

Map

This class is used to represent a UXF map. It is a collections.UserDict subclass that also has .comment, .ktype, and .vtype attributes.

See Python UXF Types for more about ktypes and vtypes.

Table

This class is used to store UXF Tables. A Table has a TClass and a .records list which is a list of lists of value with each sublist having the same number of items as the number of TClass fields. It also has .comment, .ttype (a convenience for .tclass.ttype), and .tclass attributes.

See Python UXF Types for more about ktypes and ttypes.

Table's API is very similar to the list API, only it works in terms of whole records rather than individual values.

See also the table() convenience function.

TClass

This class is used to store a Table's ttype (i.e., its name) and fields.

The .ttype attribute holds the TClass's name (equivalent to a vtype or ktype name); it may not be the same as a built-in type name or constant. The .fields attribute holds a list of fields of type Field. The .comment holds an optional str.

Field

This class is used to store a Table's fields. The .name must start with a letter and be followed by 0-uxf.MAX_IDENTIFIER_LEN-1 letters, digits, or underscores; it may not be the same as a built-in type name or constant. A vtype must be one of these strs: bool, int, real, date, datetime, str, bytes, or None (which means accept any valid type), or a ttype name.

Error

This class is used to propagate errors.

(If this module produces an exception that isn't a uxf.Error or IOError subclass then it is probably a bug that should be reported.)

Functions

The functions are documented in importance order. Here are alphabetically ordered links: append_to_parent(), canonicalize(), dump(), dumps(), is_scalar(), load(), loads(), naturalize(), on_error(), table().

load(filename_or_filelike, *, on_error=on_error)

Returns a Uxf object.

filename_or_filelike is sys.stdin or a filename (str or pathlib.Path) or an open readable file (text mode UTF-8 encoded, optionally gzipped).

on_error is a custom error handling function that defaults to on_error.

loads(uxt, filename='-', *, on_error=on_error)

Returns a Uxf object.

uxt must be a str of UXF data, e.g., uxf 1.0\n[1 2 3].

If given, the filename is used for error messages.

For more on the other argument see load().

dump(filename_or_filelike, data, *, on_error=on_error)

filename_or_filelike is sys.stdout or a filename or an open writable file (text mode UTF-8 encoded). If filename_or_filelike is a filename with a .gz suffix then the output will be gzip-compressed.

data is a Uxf object, or a list, List, dict, Map, or Table, that this function will write to the filename_or_filelike in UXF format.

If the data contains values of types that aren't supported by UXF, they could either be transformed in advance (e.g., to a custom table type, a ttype).

dumps(data, *, on_error=on_error)

data is a Uxf object, or a list, List, dict, Map, or Table that this function will write to a str in UXF format which will then be returned.

For more on the other arguments see dump().

table(ttype, fields, *, comment=None)

Convenience function for creating empty tables with a new TClass.

See also the Table __init__() method.

naturalize(s)

Given str s returns True if the str is 't', 'true', 'y', 'yes', or False if the str is 'f', 'false', 'n', 'no' (case-insensitive); or returns an int if s holds a parsable int, or a float if s holds a parsable float, or a datetime.datetime if s holds a parsable ISO8601 datetime str, or a datetime.date if s holds a parsable ISO8601 date str, or failing these returns the original str s unchanged.

This is provided as a helper function (e.g., it is used by uxfconvert.py).

on_error(lino, code, message, *, filename, fail=False, verbose=True)

This is the default error handler which you can replace by passing a custom one to load(), loads(), dump(), or dumps().

This is called with the line number (lino), error code, error message, and filename. The filename may be '-' or empty if the UXF is created in memory rather than loaded from a file. If fail is True it means the error is unrecoverable, so the normal action would be to raise. If verbose is True the normal action is to print a textual version of the error data to stderr.

For examples of custom on_error() functions, see t/test_errors.py, t/test_imports.py t/test_include.py, t/test_merge.py, and t/test_sqlite.py.

is_scalar(x)

Returns True if x is None or a bool, int, float, datetime.date, datetime.datetime, str, bytes, or bytearray; otherwise returns False.

canonicalize(name, is_table_name)

Utility for UXF processors. Given a name, returns a name that is a valid table or field name. See uxfconvert.py for uses.

append_to_parent(parent, value)

Utility for UXF processors; see uxf.py and uxfconvert.py for examples of use.

Constants

Constant	Description
VERSION	The UXF file format version
RESERVED_WORDS	A set of names that cannot be used for table or field names

Command Line Usage

The uxf module can be used as an executable. To see the command line help run:

python3 -m uxf -h

or

path/to/uxf.py -h

---