TSVZ

tsvz.org: the format, its specification and its implementations.

TSVZ is a small, dependency-free library and CLI that keeps an ordered key-value store in a delimiter-separated text file. The first column of each row is the key; the file behaves like an ordered dictionary that is persisted to disk as you change it.

Version 4.4, Python 3.6+, one self-contained module (TSVZ.py): drop it next to your code and import TSVZ, or install it.

pip install tsvz          # from PyPI
pip install -e .          # editable, from this directory

Two file dialects, one API

The delimiter follows the extension: tab, comma, NUL, pipe. A trailing compression suffix (.gz, .bz2, .xz, .zst) is allowed on both; .zst needs Python 3.14. For important or sensitive data, keep the part you write to uncompressed and compress parts once they no longer change: a damaged compressed member can lose every record in it (spec §16.4.2).

TSVZed, TSVZedLite, readTabularFile, appendTabularFile, appendLinesTabularFile, clearTabularFile, scrubTabularFile (and the *TSV aliases) work on both dialects. t.dialect tells you which one a store uses.

Quick start

import TSVZ

t = TSVZ.TSVZed('people.tsvz', header='id\tname\tscore')
t['alice'] = ['alice', 'Alice', '10']
t['bob'] = 'bob\tBob\t20'
t['alice'] = ['alice', 'Alice', '11']   # last write wins
del t['bob']                            # appends a tombstone
print(t['alice'])                       # ['alice', 'Alice', '11']
t.close()

tsvz set people.tsvz carol Carol 5        # add or update a row (append is the same)
tsvz get people.tsvz carol                # one row; exit 3 if carol is missing
tsvz read people.tsvz                     # every row: a table on a terminal, records in a pipe
tsvz delete people.tsvz carol             # tombstone
tsvz read people.tsvz | tsvz set copy.tsvz -   # copy every row
tsvz clear people.tsvz                    # empty it, keeping the header and markers
tsvz scrub people.tsvz                    # compact it (for .tsvz: archival maintenance)
tsvz verify people.tsvz                   # check #_checksum_*_# segments; exit 4 on a mismatch
tsvz parts events.tsvz                    # list the parts of a multi-part store
tsvz has people.tsvz carol                # exit 0 if carol is live, 3 if not
tsvz pop people.tsvz carol                # print carol's row and delete it
tsvz serve people.tsvz &                  # keep the store loaded; tsvz commands then use it
tsvz stop people.tsvz
tsvz people.tsvz append carol Carol 5     # the 3.39 form still works
tsvz -V

What TSVZ 4 does with .tsvz files

• Reads everything a conformant writer can produce: markers (#_defaults_#, #_strip_trailing_whites_#, #_fill_empty_with_default_#, #_return_defaults_when_missing_#, ...), escapes (<sep>, <LF>, <lt>, <#>), tombstones, #_checksum_<algo>_# segments (crc32 and every fixed-length hashlib algorithm), compressed parts, and multi-part stores (store.tsvz.<hex>, part 0 first, .rotated parts skipped).
• Writes one file: the named file, or the highest-numbered part of a multi-part store. Rows are written as given (no padding); a lone key is a tombstone; a row of empty cells (k\t\t) is a live row.
• Compaction is manual. scrubTabularFile / tsvz scrub f.tsvz rewrites a single-part store in place (same inode). If another writer commits while it runs, it reads the store again; after three tries it writes nothing, with a warning (clearTabularFile does the same). The automatic rewrite options of 3.39 (rewrite_on_load, rewrite_on_exit, rewrite_interval, rewrite(), mapToFile(), hardMapToFile()) are ignored for .tsvz, with a warning.
• Missing keys: t[key] returns the defaults row while #_return_defaults_when_missing_# is true (its default); in, get, setdefault and pop behave like a dict.
• # keys are stored (<#>key). Keys of the form #_name_# are marker writes: t['#_defaults_#'] = [...] sets the defaults, and so does setDefaults([...]) once the object is constructed.

Command line

tsvz follows tsvz-spec-v1 §20: tsvz [OPTION ...] OPERATION STORE [ARG ...], with options anywhere before a lone --. The 3.39 form tsvz STORE [OPERATION] [ARG ...] works too; a store named like an operation then needs a path prefix (./read).

• stdout carries only rows. It gets a table on a terminal and records otherwise; --format table|records overrides. Records use the store's delimiter and escaping, so tsvz read A | tsvz set B - copies a store of the same variant.
• stderr carries every message. -q prints errors only; -v adds detail.
• Exit status:
  • 0: done.
  • 1: failed or refused. Also a missing store for read, get, scrub, verify and parts, a store that set, append or delete could not create, and a .tsvz part that could not be read (the rows of the other parts are still printed).
  • 2: usage error.
  • 3: get found a missing key.
  • 4: verify found a checksum mismatch.
• Arguments are literal. tsvz set s.tsvz k 'a<b' stores a<b. Values may begin with - (-5); after -- every argument is positional. set KEY alone deletes KEY, as in 3.39. set KEY '' stores an empty cell in a .tsvz store; in a .tsv file it deletes KEY, as in 3.39.
• Bulk input. set STORE - and delete STORE - read records or keys from stdin and append them in one write. An unterminated last line is ignored, with a warning.
• TSVZ extensions:
  • --x-header H and --x-defaults KEY<d>D1<d>... decode backslash escapes such as \t;
  • --x-strict / --x-force turn 3.39's column and header checks on or off.

Serving a store

tsvz serve STORE keeps a store loaded in one process and answers requests on a local socket (tsvz-spec-v1 §21). tsvz commands and TSVZ.TSVZClient find it through the pointer file STORE.serve and use it instead of re-reading the store. Their output and exit status stay the same.

• One appender. The server batches the writes it receives into single appends. Other writers may still append to the files; the server reads what they add before it answers.
• Acknowledgement. A write is acknowledged once it is queued (#_write_ack_# memory, the default), or after fsync (#_write_ack_# disk, or TSVZClient(..., sync=True)). Reads always see acknowledged writes.
• Access. Only the user who started the server can connect (socket mode 0600). --x-group G or --x-mode 0660 opens it on purpose. The server listens on a Unix-domain socket only; reaching it from other hosts is left to a separate adapter (for example TCP with TLS) that would relay to the socket.
• Trust. Clients use STORE.serve only when it is a regular file owned by them or by the store's owner, not writable by anyone else, naming a socket that belongs to that owner. Otherwise they warn and use the files. serve never writes through a symbolic link or into a file it did not create.
• Lifetime. It runs in the foreground until tsvz stop STORE, SIGINT or SIGTERM, or --x-idle-timeout S seconds without connections. It then writes and fsyncs everything and removes STORE.serve.
• Bypassing it. --x-direct makes a tsvz command use the files. Command lines with --x-header, --x-defaults or --x-strict, or another delimiter than the server's, use the files too.

import TSVZ

with TSVZ.TSVZClient('people.tsvz') as people:  # served, or in this process when no server runs
    people['dave'] = ['dave', 'Dave', '41']
    print(people['dave'], len(people), 'alice' in people)
    print(people.pop('dave'))

TSVZClient has TSVZed's semantics for reading and setting keys, del, in, len, iteration, pop, popitem, setdefault, update, clear and setDefaults. Its writes return once they are written (raising OSError if they cannot be), one client may be shared between threads, and without a server it warns once and works on the files in-process, finishing its writes when the program exits.

Fault tolerance

TSVZ keeps your data and keeps going when a file is not what it expects, and says so on stderr (or through teeLogger) with one summarised line per problem:

TSVZ warning: data.tsvz: invalid UTF-8 replaced with U+FFFD (3 occurrences, first at line 17)

Examples: bytes after the last newline (ignored on read; truncated before the next append, and quoted), invalid UTF-8, a byte order mark, a newer #_version_#, a marker with a bad value, a checksum mismatch, a damaged compressed member or a zero-filled line (a member cut short by a crash keeps the whole lines it decoded; a corrupt member or a zero-filled line is recovered, skipped or stops the read, as the store's #_on_corruption_# says, recover by default, spec §4.9, §4.10, §16.6; a scrub that rewrites such a part keeps the original as <part>.damaged-<time>), an unreadable part, and a delimiter or encoding argument that conflicts with the extension.

Breaking changes from 3.39

.tsv, .csv, .nsv, .psv: bug fixes only

.tsvz, .csvz, .nsvz, .psvz: now tsvz-spec-v1

The tsvz command

3.39 command lines still parse, including abbreviated long options (--delim) and -d=,, except the ones C5 and C9 list.

Spec deviations and interpretations

None of these fails a conformance case: conformance/reference-baseline.json, which lists the cases this implementation does not pass, is empty (see Conformance testing).

• §17.5: a numbered part opened directly is read alone, with a warning.
• §14.5: the defaults-on-missing read applies to t[key] only.
• §18.6: locking is one exclusive whole-file lock per write plus an in-process lock; readers do not lock. On Linux it is an open file description lock, which still excludes 3.39's lockf; elsewhere it is lockf.
• §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
• §20.7: -q also silences informational messages ("Created …"); only errors remain.
• §20.2: scrub of a multi-part store, and clear of one part of one, are refused with exit 1; read, get and parts of a numbered part read that part alone, with a warning.
• §20.2.4: on .tsv-family files get prints nothing for a missing key, verify checks nothing, and parts lists the file.
• §20.3: a bulk input line with an empty key is skipped, with a warning.
• §20.6: a write whose store cannot be created, and a read, get or verify that cannot read every part, exit 1 with an error that -q keeps.
• §21: a served .tsv file with a header line shows that line as a row, as tsvz read does (TSVZed hides it when given the header).
• §21.8: the server notices changes with stat(). It re-reads the part list while the store's directory changed less than 2 s ago, and reloads a part whose size stayed the same but whose inode or modification time changed. A rewrite in place that keeps a part's size and modification time is not seen until the part changes again.
• §21.9: tsvz commands and TSVZClient ask the server to answer writes once they are written (--x-written), so a write the server cannot make fails as it would on the files. While writes fail, every answer carries a warning.
• §21.3: tsvz serve needs Unix-domain sockets. Where Python has none (Windows), it exits 1, and tsvz commands and TSVZClient work on the files.
• §20.7: on .tsv-family files a -d longer than one character is used, as in 3.39, with a warning; a -d that does not decode is a usage error.

Website

website/ holds the tsvz.org site. website/tsvz_site.py (standard library only, Python 3.8+) renders website/index.md, website/spec-glance.md and tsvz-spec-v1.md, and serves HTML to browsers and markdown to agents.

python3 website/tsvz_site.py serve -p 8765     # http://127.0.0.1:8765/
python3 website/tsvz_site.py build /tmp/site   # static files for any host

deploy/tsvz-site.service runs it under systemd; restart it to publish an edit.

Conformance testing

tsvzconform.py tests how closely an implementation of tsvz-spec-v1, in any language, follows the spec. Its cases are JSON files in conformance/cases/, each naming the § clauses it tests, and it drives an implementation only through the spec's command line (§20) and handler socket (§21).

python3 tsvzconform.py --impl ./tsvz --feature crc32,sha256,handler
python3 tsvzconform.py --impl ./tsvz --class reader --only 13

Each failure is printed with its input, what was expected, what came back and a command line that reruns it alone; the summary gives a verdict for each conformance class (reader, writer, cli, handler). conformance/README.md explains the options and the features to claim, and how to load the cases into another language's tests.

Benchmarking implementations

tsvzbench.py compares implementations of tsvz-spec-v1, in any language, side by side on one machine. It drives each one only through the spec's command line (§20) and, when it has one, its handler socket (§21), so any conformant tsvz tool can be measured as it is.

python3 tsvzbench.py --impl ref='python3 TSVZ.py' --impl new='../tsvz-rs/target/release/tsvz' \
                     --suite quick --work-dir /mnt/fast-disk/tsvzbench

• --impl NAME=COMMAND, repeatable: the first is the baseline, and every other implementation gets a × ratio against it (below 1 is faster). A relative COMMAND runs from the directory you start tsvzbench in.
• --suite smoke|quick|standard|large: smoke checks a setup in seconds, quick takes about a minute, standard (the default, up to 1M rows) about five, and large (up to 10M rows) twenty or more and ~5 GB of memory for the harness itself.
• --only cold,bulk,perop,maint picks families: one-shot replays, set - and delete - batches, single operations (CLI and handler), and scrub/clear.
• --work-dir DIR keeps the generated fixtures between runs. Put it on the disk you mean to measure: not a tmpfs, and not a FUSE file system such as mergerfs (the harness warns about those). The default is a tsvzbench-* directory here.
• --json PATH also writes every raw sample.

The harness writes its own input stores, so every implementation reads the same bytes, and checks every result against its own replay model: ✓ right, ✗ wrong output or files, ! N an unexpected exit status, ⏱ timed out, refused an allowed refusal. It exits 1 when any cell failed. Each command runs under a small launcher that measures it with wait4; a peak RSS shown as ≤8.0 MiB is the launcher's own size, so the true peak is no higher. Compare an implementation with itself to see the noise: a few percent per cell, but the socket rows, whose latencies are tens of microseconds, can move 10–20% between runs, because a single run can land in a slower mode as a whole.

Tests

python3 -m pytest TSVZ_test.py -q                         # everything, incl. 3.39 differential tests
python3 -m pytest website/tsvz_site_test.py -q            # the tsvz.org site
python3 -m pytest tsvzbench_test.py -q                    # the benchmark harness
python3 -m pytest tsvzconform_test.py -q                  # the conformance runner and cases, against the reference
python3 benchTSVZ.py /tmp/b.tsv -n 100000 --compare-old    # benchmark (not a test)

TSVZ_old.py is the frozen 3.39 module the differential tests compare against; it is not installed.

License

GPL-3.0-or-later © Yufei Pan (pan@zopyr.us)