TSVZ

tsvz.org: the format, its specification and its implementations.

TSVZ is a small, dependency-free library and CLI that keeps an ordered key-value store in a delimiter-separated text file. The first column of each row is the key; the file behaves like an ordered dictionary that is persisted to disk as you change it.

Version 5.1, Python 3.6+, one self-contained module (TSVZ.py): drop it next to your code and import TSVZ, or install it.

pip install tsvz          # from PyPI
pip install -e .          # editable, from this directory

Two file dialects, one API

The delimiter follows the extension: tab, comma, NUL, pipe. A trailing compression suffix (.gz, .bz2, .xz, .zst) is allowed on both; .zst needs Python 3.14. For important or sensitive data, keep the part you write to uncompressed and compress parts once they no longer change: a damaged compressed member can lose every record in it (spec §16.4.2).

TSVZed, TSVZedLite, readTabularFile, appendTabularFile, appendLinesTabularFile, clearTabularFile, scrubTabularFile (and the *TSV aliases) work on both dialects. t.dialect tells you which one a store uses.

Quick start

import TSVZ

t = TSVZ.TSVZed('people.tsvz', header='id\tname\tscore')
t['alice'] = ['alice', 'Alice', '10']
t['bob'] = 'bob\tBob\t20'
t['alice'] = ['alice', 'Alice', '11']   # last write wins
del t['bob']                            # appends a tombstone
print(t['alice'])                       # ['alice', 'Alice', '11']
t.close()

tsvz set people.tsvz carol Carol 5        # add or update a row (append is the same)
tsvz get people.tsvz carol                # one row; exit 3 if carol is missing
tsvz read people.tsvz                     # every row: a table on a terminal, records in a pipe
tsvz delete people.tsvz carol             # tombstone
tsvz read people.tsvz | tsvz set copy.tsvz -   # copy every row
tsvz clear people.tsvz                    # empty it, keeping the header and markers
tsvz scrub people.tsvz                    # compact it (for .tsvz: archival maintenance)
tsvz verify people.tsvz                   # check #_checksum_*_# segments; exit 4 on a mismatch
tsvz parts events.tsvz                    # list the parts of a multi-part store
tsvz has people.tsvz carol                # exit 0 if carol is live, 3 if not
tsvz pop people.tsvz carol                # print carol's row and delete it
tsvz serve people.tsvz &                  # keep the store loaded; tsvz commands then use it
tsvz stop people.tsvz
tsvz people.tsvz append carol Carol 5     # the 3.39 form still works
tsvz -V

What TSVZ does with .tsvz files

• Reads everything a conformant writer can produce: markers (#_defaults_#, #_strip_trailing_whites_#, #_fill_empty_with_default_#, #_return_defaults_when_missing_#, ...), escapes (<sep>, <LF>, <lt>, <#>), tombstones, #_checksum_<algo>_# segments (crc32 and every fixed-length hashlib algorithm), compressed parts, and multi-part stores (store.tsvz.<hex>, part 0 first, .rotated parts skipped). A read starts at the last snapshot part (one that begins with #_snapshot_#, spec §19.10, and holds the store's whole state) and skips the history before it; verify and parts still read every part. A store whose parts change while they are read (another tool's in-place scrub) is read again, up to three times; a read still changing then is incomplete, as an unreadable part is.
• Writes one file: the named file, or the highest-numbered part of a multi-part store. Rows are written as given (no padding); a lone key is a tombstone; a row of empty cells (k\t\t) is a live row.
• Compaction is manual. scrubTabularFile / tsvz scrub f.tsvz rewrites a single-part store in place (same inode). If another writer commits while it runs, it reads the store again; after three tries it writes nothing, with a warning (clearTabularFile does the same). The automatic rewrite options of 3.39 (rewrite_on_load, rewrite_on_exit, rewrite_interval, rewrite(), mapToFile(), hardMapToFile()) are ignored for .tsvz, with a warning.
• Missing keys: t[key] returns the defaults row while #_return_defaults_when_missing_# is true (its default); in, get, setdefault and pop behave like a dict.
• # keys are stored (<#>key). Keys of the form #_name_# are marker writes: t['#_defaults_#'] = [...] sets the defaults, and so does setDefaults([...]) once the object is constructed. #_snapshot_# is refused (spec §19.10.2).
• Other writers: a TSVZed notices another process's change by the active part's inode, size and times and by the list of parts, and reloads. Rows it has queued but not yet written stay in memory across that reload, and a live key is never missing while it runs.

Command line

tsvz follows tsvz-spec-v1 §20: tsvz [OPTION ...] OPERATION STORE [ARG ...], with options anywhere before a lone --. The 3.39 form tsvz STORE [OPERATION] [ARG ...] works too; a store named like an operation then needs a path prefix (./read).

• One engine. Every operation runs through the engine tsvz serve uses, in its own process when no server runs, so a command prints the same with or without a server.
• stdout carries only rows. It gets a table on a terminal and records otherwise; --format table|records overrides. Records use the store's delimiter and escaping, so tsvz read A | tsvz set B - copies a store of the same variant.
• stderr carries every message. -q prints errors only; -v adds detail.
• Exit status:
  • 0: done.
  • 1: failed or refused. Also a missing store for read, get, scrub, verify and parts, a store that set, append or delete could not create, and a .tsvz part that could not be read (the rows of the other parts are still printed).
  • 2: usage error.
  • 3: get found a missing key.
  • 4: verify found a checksum mismatch.
• Arguments are literal. tsvz set s.tsvz k 'a<b' stores a<b. Values may begin with - (-5); after -- every argument is positional. set KEY alone deletes KEY, as in 3.39. set KEY '' stores an empty cell in a .tsvz store; in a .tsv file it deletes KEY, as in 3.39.
• Bulk input. set STORE - and delete STORE - read records or keys from stdin and append them in one write. An unterminated last line is ignored, with a warning.
• TSVZ extensions:
  • --x-header H and --x-defaults KEY<d>D1<d>... decode backslash escapes such as \t;
  • --x-header H makes line 1 of a .tsv-family file its header for every operation when line 1 is H: the same number of cells, each equal to H's once the whitespace around it is taken off (a trailing empty cell counts: id\tv\t is not id\tv; a UTF-8 byte order mark before line 1, as Excel writes one, is skipped). It is then never a key, row or count, scrub and clear keep it, and read --format table shows it above the rows. When line 1 differs, the file has no header and line 1 is a row, with 3.39's "Header mismatch" warning (--x-strict: an error, exit 1, nothing written). A missing or empty file gets H first. Without --x-header, line 1 is a row, as in 3.39. (The library's own header check, a header= given to readTabularFile, appendTabularFile, clearTabularFile, scrubTabularFile or TSVZed, is 3.39's, looser one: each cell of H within the cell of line 1 under it, so idaho\tvalue passes for id\tv.) A rewrite without --x-header can put a row equal to H on line 1 (limitations).
  • --x-strict / --x-force turn 3.39's column and header checks on or off.

Serving a store

tsvz serve STORE keeps a store loaded in one process and answers requests on a local socket (tsvz-spec-v1 §21). tsvz commands and TSVZ.TSVZClient find it through the pointer file STORE.serve and use it instead of re-reading the store. Their output and exit status stay the same.

• One appender. The server batches the writes it receives into single appends. Other writers may still append to the files; the server reads what they add before it answers.
• Acknowledgement. A write is acknowledged once it is queued (#_write_ack_# memory, the default), or after fsync (#_write_ack_# disk, or TSVZClient(..., sync=True)). Reads always see acknowledged writes.
• Access. Only the user who started the server can connect (socket mode 0600). --x-group G or --x-mode 0660 opens it on purpose. The server listens on a Unix-domain socket only; reaching it from other hosts is left to a separate adapter (for example TCP with TLS) that would relay to the socket.
• Trust. Clients use STORE.serve only when it is a regular file owned by them or by the store's owner, not writable by anyone else, naming a socket that belongs to that owner. Otherwise they warn and use the files. serve never writes through a symbolic link or into a file it did not create.
• Lifetime. It runs in the foreground until tsvz stop STORE, SIGINT or SIGTERM, or --x-idle-timeout S seconds without connections. It then writes and fsyncs everything and removes STORE.serve.
• Bypassing it. --x-direct makes a tsvz command use the files. Command lines with --x-header, --x-defaults or --x-strict, or another delimiter than the server's, use the files too. A server started with --x-header H answers every request by the --x-header rule above, and one started with --x-strict too answers as a direct --x-strict command does: a line 1 that is not H fails the request (status 1, nothing written), and reads and scrubs drop the rows of another width. It does not start when line 1 is not H then. The table of a routed read --format table shows no H line: its first row is set as the table's header, as with no --x-header.

import TSVZ

with TSVZ.TSVZClient('people.tsvz') as people:  # served, or in this process when no server runs
    people['dave'] = ['dave', 'Dave', '41']
    print(people['dave'], len(people), 'alice' in people)
    print(people.pop('dave'))

TSVZClient has TSVZed's semantics for reading and setting keys, del, in, len, iteration, pop, popitem, setdefault, update, clear and setDefaults. Its writes return once they are written (raising OSError if they cannot be), one client may be shared between threads, and without a server it warns once and works on the files in-process, finishing its writes when the program exits. A server of a .tsv-family file names its delimiter in STORE.serve, and TSVZClient splits string values on it.

Fault tolerance

TSVZ keeps your data and keeps going when a file is not what it expects, and says so on stderr (or through teeLogger) with one summarised line per problem:

TSVZ warning: data.tsvz: invalid UTF-8 replaced with U+FFFD (3 occurrences, first at line 17)

Examples: bytes after the last newline (ignored on read; truncated before the next append, and quoted), invalid UTF-8, a byte order mark, a newer #_version_#, a marker with a bad value, a checksum mismatch, a damaged compressed member or a zero-filled line (a member cut short by a crash keeps the whole lines it decoded; a corrupt member or a zero-filled line is recovered, skipped or stops the read, as the store's #_on_corruption_# says, recover by default, spec §4.9, §4.10, §16.6; a scrub that rewrites such a part keeps the original as <part>.damaged-<time>), an unreadable part, and a delimiter or encoding argument that conflicts with the extension.

Writing is stricter than reading: a key, value, header or marker that cannot be encoded as UTF-8, such as the lone surrogate os.fsdecode gives for a non-UTF-8 file name, raises a ValueError (TSVZEncodeError) that names it and writes nothing, instead of storing ?. Pass errors='replace' (or any codec error handler) to the writing functions or to TSVZed, TSVZedLite and TSVZClient to get the old ?. TSVZed and TSVZedLite refuse such a row when it is set. A rewrite that meets such text leaves the file as it is: mapToFile() and hardMapToFile() raise the error, and the rewrites TSVZed makes itself report it once. With a non-UTF-8 encoding=, U+FFFD (what the reader puts for a byte it could not decode) is text that encoding may lack, and it is refused like any other: such a store is not rewritten (rewrite_on_load leaves the file as it is, without failing), appends of rows that can be written still work, and the delete or update of a key that holds U+FFFD raises. Such a key stays until the file is edited, the store cleared, or the store opened with the encoding it was written in; errors='replace' would write ? for U+FFFD, losing the bytes the file holds there. The tsvz command exits 1 with the message.

A TSVZed writes a .tsv, .csv, .nsv or .psv file (compressed or not) from its background thread only: rewrite(), mapToFile(), hardMapToFile(), commitAppendToFile(), clear(), clear_file(), reload(), close() and the constructor's load hand it the job and wait, so a Ctrl-C stops the wait, never the write (the call still raises KeyboardInterrupt). At exit a write in progress is finished: a Ctrl-C then says so and keeps waiting, and a second one abandons it. A closed store is freed. (get_file_obj() and release_file_obj(), kept from 3.39, hand the locked file to the calling thread.) A file another process removed is made again from memory by the store's next write, an append included (a removal in the microseconds as a write starts is followed as a rotation instead); while that rewrite fails or is refused, the rows are appended to a new file there, and memory keeps all of them for the rewrite that gets through. A compressed file another writer left damaged is made again too (L5: every read stops at the damage, so rows appended after it would be read by no one): what the read did not get goes, the bytes after the damage and also the lines of the damaged member before it that the codec never handed over (Python's gzip gives none of a first member cut short, where zcat shows them). While that rewrite fails, the rows are appended after the damage and memory keeps them for the next rewrite that gets through; while it is refused (a row changed in place to text the encoding cannot hold), they are appended after it as in 4.9, and the next read drops them. Either way close() tries the rewrite once more, whatever rewrite_on_exit says. Appends that do not read the store first (tsvz set, appendTabularFile, TSVZedLite, and a TSVZed with monitor_external_changes=False for damage that lands after it loaded the file) still land after it, and so do a TSVZed's rows when a writer that takes no lock leaves the damage between the store's read and its append (docs/limitations.md).

It knows the size and SHA-256 of the bytes it last read or wrote. When another process has changed the file, and it is the same file (device and inode) and still begins with those bytes, nothing this store set is still waiting to be written, and setDefaults() has not changed its defaults since, only what follows is read and applied to memory, as a read of the file applies it: memory keeps an order set by move_to_end, and its # keys. Any other change reloads the file, changing memory in place, so a live key is never missing meanwhile.

A rewrite writes over the file and then cuts it to its new length, so a reader never meets it emptied. One that fails part way (a full disk, an I/O error), or that cannot make again a file that was removed, leaves the file damaged: the new bytes it wrote, then the old ones after them. Memory then holds the only good copy, as it does when the store appends after damage another writer left that it cannot make the file again over yet (that rewrite failed, or waits out a failed one's back-off). Until a rewrite gets through, the file is not reloaded over memory; the background writer still appends the rows set meanwhile, and the next rewrite (the background writer's, rewrite(), close()) makes the file again from memory. A change another process made meanwhile is merged into memory first, as above, when the file is still the one the damage was left in (for a file that was removed, any file made there since) and still begins with exactly the bytes it left; after a damage that left the file empty (or removed it), what follows must also begin with this store's header. Otherwise that change is overwritten, with the warning Overwriting external changes. What cannot be told from an append is merged as one: another process's rewrite of the same file in place that begins with exactly the bytes the damage left, and, for a store with no header (or verifyHeader=False), anything written to a file the damage left empty; the keys it sets or deletes then take its values, and the keys it left out keep memory's. An explicit reload() while the file is damaged replaces memory with the damaged file, as asked, and warns.

Upgrading to 5.2 from 5.1

The file formats are unchanged, and a 3.39 host still reads every .tsv-family file 5.2 writes.

• A scrub of a .tsv whose rows were all deleted shrinks it. scrubTabularFile, tsvz scrub (also through a tsvz serve handler) of a .tsv-family file (.tsv, .csv, .nsv, .psv and their compressed forms) in which every key has been deleted leaves the file as an empty store: the header line it has, and its #_defaults_# lines as a scrub of a file with rows keeps them (an empty one after the header, the last non-empty one after it, both as wide as the header, or as the column count the read took, which caller defaults set when given). A file with no header line keeps its empty first line, or becomes 0 bytes when its first line was a row; a first line that is no row (a comment, say) becomes an empty line. A defaults line it keeps follows an empty first line, which the scrub then writes. Comment lines go, as a scrub of a file with rows already drops them. A scrub that is given a header (and checks it, the default) keeps the file's first line as its header, an empty one included, with a "Header mismatch" warning when it differs; the header it is given is written only with verifyHeader=False, or when the file's first line cannot be decoded (plain text named .gz); tsvz scrub --x-header follows the --x-header rule below instead. A scrub of a path that is not there makes no file and returns {}, as in 5.1, unless createIfNotExist=True. A compressed file whose read decodes no row (plain text named .gz, a first member cut short) is made again as an empty store, as a scrub of a damaged file with rows already was in 5.1: what the read did not get goes, the bytes after the damage and also the lines of the damaged member before it that the codec never handed over (Python's gzip gives none of a first member cut short, where zcat shows them) (see the damaged-stream notes above), and tsvz clear --x-header H --x-strict (clearTabularFile(verifyHeader=True)) now clears such a file, where 5.1 exited 1 with the codec's error. 5.1 and 5.0 left such a file as it was, deletion lines included, so a writer that appended keys, deleted them all and scrubbed, over and over, saw the file grow without limit (205, 405, 605 bytes, ... for 20 keys a cycle); now it is the same size after every scrub. A file with no byte (or a compressed one with only empty streams) is still left alone, and so is a file whose strict scrub (strict=True, --x-strict) read dropped a line for its width and found no row: that line is not a deletion, as 5.1 left it. A scrub that finds the file changed since its read still reads again and writes nothing if it keeps changing, as for a file with rows.
• get_file_obj('ab') on a compressed store writes its trailer before the lock is let go. A program that calls it on a .tsv.gz, .bz2, .xz or .zst store and writes through the file now has all of it, the gzip trailer and the bz2, xz and zstd data included, in the file when release_file_obj() unlocks it. 5.1 and earlier wrote them after the unlock, where another writer could append in between and corrupt the stream.
• A TSVZed no longer lets a file grow with every session of sets and deletes. It rewrites a .tsv-family file at open when the file holds a key's row and the deletion of it, as 3.39 did only for a key with two values, so a program that sets keys and deletes them in each session (default settings, or rewrite_interval) keeps the file the same size (205, 405, 605 bytes, ... for 20 keys a session before). Every session that opens such a file now rewrites it once. With rewrite_on_load=False the open does not, and the next exit or periodic rewrite does it (rewrite_on_exit=True, rewrite_interval). What such a rewrite does, it now does at a plain open:
  • it cuts a row wider than the header, and the cut stays (limitations): id\tv\na\t1\t2\nb\t1\nb\t\n becomes id\tv\na\t1 \n, and id\tv\na\t\t7\nb\t1\nb\t\n becomes id\tv\na\t \n, the 7 lost;
  • it writes over the file in place and pads with spaces (a\t1 above): TSVZ readers strip them, tools that read the raw bytes (awk, cut) see them;
  • on a file it cannot write (read-only), it prints Failed to open file … Permission denied twice and leaves the file as it is, as 5.1 did for a key with two values.
• --x-header H makes line 1 the header for every tsvz operation. On a .tsv-family file whose line 1 is H (an exact match: the same cells, each without the whitespace around it, a trailing empty cell included), line 1 is the header: keys, len, read, get, has, pop, popitem and delete never list, count, print, pop or delete it, and scrub and clear keep it. When line 1 differs, the file has no header and line 1 is a row, with 3.39's "Header mismatch" warning, for the reads and the scrub alike, so a scrub keeps the rows a read shows (but for the column-width cases below), and in such a file it never writes a row that is H as line 1: it puts an empty line first (a\t1\nid\tv\na\t\n becomes \nid\tv\n), so the reads still take that row for a row. With --x-strict a line 1 that is not H is an error (exit 1, nothing written) for every operation, through a handler too. A missing or empty file gets H first at its first write, as in 5.1. A tsvz serve STORE --x-header H handler applies the rule to every request it answers, routed tsvz commands included; with --x-strict it is strict for them as above (5.1 ignored --x-strict at serve): such a handler refuses to start on a file whose line 1 is not H, its routed .tsv reads and scrubs drop lines of another column count, and its .tsvz reads are strict, as for a direct --x-strict command. The library's own header check (header= to readTabularFile, appendTabularFile, clearTabularFile, scrubTabularFile, TSVZed) is unchanged, 3.39's containment check. The CLI's writes check line 1 more strictly than 5.1's did: 5.1's set, append, delete and clear with --x-header silently took a line 1 that merely held H's cells (id\tvalue for id\tv) for the header and fitted rows to H's width; now they warn and write the row at its own width, and with --x-strict exit 1. A rewrite without --x-header (a scrub without it, scrubTabularFile, a TSVZed rewrite) can still put a row equal to H on line 1 of a header-less file (limitations). In 5.1 only set, append, delete and clear used --x-header: the reads and scrub took line 1 as a row (tsvz keys printed id a b for two keys, len said 3, popitem first popped the header), and so did every request to a handler. Without --x-header nothing changed: line 1 is a row, as in 5.1 and 3.39. In a .tsvz store pop, popitem and setdefault now check --x-header as set does; nothing else changed there.
• Known, mostly as in 5.1: a scrub, a rewrite or a read takes one column count for every row, and can lose cells or keys. A scrub given defaults narrower than the rows, a row wider than the header, and a .tsv that begins with a bare #_defaults_# line (read without defaults) are examples; the entry lists each, and which is new in 5.2 (limitations). A strict scrub of a file whose read dropped a line for its width and found no row leaves it, so it grows again over append, delete-all and scrub cycles (limitations). Both wait for the column-width design after 5.2.

Upgrading to 5.1 from 5.0

The file formats are unchanged, and a 3.39 host still reads every .tsv-family file 5.1 writes. Most of the changes are in scrub and clear.

• Scrub and clear of a .tsv write over the file. scrubTabularFile, clearTabularFile, tsvz scrub and tsvz clear (on the command line or through a tsvz serve handler), TSVZClient.clear() and TSVZedLite.clear_file() of a .tsv-family file write the new content over the file and then cut it, as TSVZed's rewrites do, so a reader that takes no lock never meets the file emptied. They take the file's lock, so they wait while another writer holds it. 5.0 emptied the file first, with no lock (TSVZedLite truncated its own descriptor).
• A scrub keeps another process's rows. A scrub of a .tsv writes only if the file is still what its read saw. Otherwise it reads again, up to three times, and if the file kept changing it writes nothing and warns: tsvz scrub exits 1 (scrub refused; nothing was written), as for a .tsvz (C6), and a handler answers refused. scrubTabularFile still returns the rows it read. 5.0 wrote over a row appended after its read.
• scrubTabularFile(lastLineOnly=True) is a read. It returns the last row and writes nothing, as for a .tsvz. 5.0 and 3.39 wrote that row back as deletion lines (id\tv\na\t\n3\t\n), and the store was gone.
• A scrub keeps the file's #_defaults_# lines. The defaults in effect at the end of the read are written after the rows: the last non-empty defaults line, its empty cells filled from the line before it as a reader fills them. An empty defaults line that was read (it turns defaults off) is written before the rows, padded to their width. Every row reads as it did to a reader given the scrub's defaults and strict. Defaults a caller passes still fill the rows' empty cells, as in 5.0, but are not written as a defaults line, and the caller's list is not changed. 5.0 and 3.39 dropped the lines, so rows appended later got no defaults.
• The header stays on line 1. A scrub given no header (tsvz scrub of a .tsv, scrubTabularFile(path)) and a TSVZed given none, of a file that has a header line, keep that line first, with a TSVZed's defaults line right after it. 5.0 and 3.39 put an empty line in front of it, so a reader given the header read the header as a row. A header-less file keeps the empty first line it begins with, and new files and clears given no header still get one, as in 5.0.
• Interrupts. A scrub or clear of either dialect keeps a copy of only the bytes it overwrites, to put back if it is interrupted before the file is cut to its new length; once the cut is made the rewrite is done and the new content stays. 5.0 kept a copy of the whole .tsvz part and put it all back. (A scrub still reads the whole store, and a .tsvz clear reads it too, to keep its markers.)
• A compressed store cleared to nothing is one empty stream. A .tsvz.gz (or .bz2, .xz, .zst) cleared or scrubbed to nothing holds one empty compressed stream, which bzcat, xzcat and Python's bz2 and lzma can read; 5.0 left 0 bytes.
• A header-checking clear of an empty file writes the header it was given. A clear that checks the header (verifyHeader=True, tsvz clear --x-header H --x-strict) of a 0-byte .tsv, or of a compressed file with no byte or only empty streams (another tool's), takes it for an empty store with that header, as every reader does (L9), and writes H without a "Header mismatch" warning. 5.0 wrote an empty line and warned of a mismatch, so a strict TSVZed given H then refused the file, and raised on a 0-byte .bz2, .xz or .zst.
• tsvz --help says what clear keeps: a .tsvz its header and markers, a .tsv the --x-header it is given. (With --x-strict too, a .tsv clear keeps the file's own header line.) TSVZed's message for a refused rewrite lists hardMapToFile() among the rewrites a program asks for.
• Known in 5.1, fixed in 5.2: a scrub of a .tsv whose rows were all deleted did not compact it when the scrub was given the file's header (scrubTabularFile(path, header=H)) or the file had no header line: it left the file as it was, deletion lines included; reads were right, as in 3.39 and 5.0. (tsvz scrub of a headed file read the header line as a row, so it did compact it.)
• Known: a scrub run with other defaults than the store's program can change how early rows read. In a .tsv with an empty #_defaults_# line and rows with empty cells before any defaults line, the reset line the scrub writes before the rows keeps those cells empty for a program that opens the store with its own defaults; run the scrub with the defaults that program uses (limitations).
• Running programs keep 5.0 until they are restarted: long-running programs that import TSVZ, and running tsvz serve handlers.

Upgrading to 5.0 from 4.9

• Writes refuse text they cannot encode. A key, value, header, defaults line or marker the file's encoding cannot hold (UTF-8 for .tsvz; encoding= for a .tsv), such as the lone surrogate os.fsdecode gives for a non-UTF-8 file name, raises TSVZEncodeError (a ValueError) and writes nothing; 4.9 stored ? (its tsvz stored U+FFFD for a non-UTF-8 argument, and its .tsvz file functions raised UnicodeEncodeError). errors='replace' brings ? back. Under a non-UTF-8 encoding=, a key read as U+FFFD cannot be deleted or updated. tsvz exits 1 with the message. See Fault tolerance.
• A Ctrl-C no longer cuts a .tsv write. Every write of a .tsv-family file runs on the store's background thread; a call waits for it, a Ctrl-C raises in the caller and the write still finishes, and at exit a write in progress is finished (a second Ctrl-C abandons it). In 4.9 a Ctrl-C while TSVZed() opened a compressed store (its rewrite on load) could leave the file with no readable row. A closed TSVZed or TSVZedLite is freed; 4.9 kept every one until exit.
• Another process's change is taken in, not reloaded over memory. An append to a .tsv-family file is read from where the store left off, so an order set by move_to_end and # keys survive it; any reload (both dialects) changes memory in place, so another thread never finds a live key missing. After a rewrite of the store's own fails part way, memory is kept and another process's appends are merged into it (Fault tolerance).
• .tsvz stores see another process's append made right after their load, a commit or a clear (4.9 missed it until the next change), keep the rows set while they reload, and raise a strict header mismatch at the first read.
• .tsv files: an append to an existing empty file writes the header first, and reading one gives an empty store; see L8 to L14 for the rest.
• tsvz runs every operation through the engine tsvz serve uses, with the same output. A server of a .tsv (or other tab- or NUL-delimited) file is now found by tsvz commands, TSVZClient and tsvz stop; 4.9's clients rejected its pointer file. The 3.39 warning that a file name does not end in .tsv is printed only when a command creates such a file, and a write that cannot be made reports the operating system's error.
• Running programs keep 4.9. Long-running programs that import TSVZ, and running tsvz serve handlers, keep 4.9 until they are restarted. 5.0's tsvz stop cannot stop a 4.9 handler of a .tsv (or other tab- or NUL-delimited) file: 4.9 wrote the delimiter into the pointer file's x-delimiter as <sep> (a NUL as <NUL>), which 5.0 decodes to the TAB or NUL itself and refuses as a control character. Send that handler SIGTERM instead: its pid is on the pid line of its pointer file (STORE.serve), and it finishes its writes and exits.
• macOS and other systems that lock with lockf: as in 4.9, a read in another thread of the process can still drop a write's lock (docs/limitations.md).

Breaking changes from 3.39

.tsv, .csv, .nsv, .psv: the format is unchanged

.tsvz, .csvz, .nsvz, .psvz: now tsvz-spec-v1

The tsvz command

3.39 command lines still parse, including abbreviated long options (--delim) and -d=,, except the ones C5 and C9 list.

Spec deviations and interpretations

None of these fails a conformance case: conformance/reference-baseline.json, which lists the cases this implementation does not pass, is empty (see Conformance testing).

• §17.5: a numbered part opened directly is read alone, with a warning.
• §14.5: the defaults-on-missing read applies to t[key] only.
• §18.6: locking is one exclusive whole-file lock per write plus an in-process lock; readers do not lock. On Linux it is an open file description lock, which still excludes 3.39's lockf; elsewhere (macOS and other systems, or a Linux file system that refuses those locks) it is lockf, which closing any descriptor of the file in the process drops: there a read in another thread of the same process, such as a handler's or a TSVZed's, can drop a write's lock while it is held. A known limitation, deferred: see docs/limitations.md.
• §19.4: a scrub whose rows would read differently under the final markers writes them with stripping and fill-empty off, then restores the final markers after the rows.
• §20.7: -q also silences informational messages ("Created …"); only errors remain.
• §20.2: scrub of a multi-part store, and clear of one part of one, are refused with exit 1; read, get and parts of a numbered part read that part alone, with a warning.
• §20.2.4: on .tsv-family files get prints nothing for a missing key, verify checks nothing, and parts lists the file.
• §20.3: a bulk input line with an empty key is skipped, with a warning.
• §20.6: a write whose store cannot be created, and a read, get or verify that cannot read every part, or whose parts kept changing through three reads, exit 1 with an error that -q keeps.
• §21: a served .tsv file with a header line shows that line as a row, as tsvz read does (TSVZed hides it when given the header).
• §21.8: the server notices changes with stat(). It re-reads the part list while the store's directory changed less than 2 s ago, and reloads a part whose size stayed the same but whose inode, modification time or change time changed. The change time catches a rewrite whose modification time was set back (cp -p, rsync --inplace -a); a rewrite in place that keeps a part's size within one clock tick is not seen until the part changes again.
• §21.9: tsvz commands and TSVZClient ask the server to answer writes once they are written (--x-written), so a write the server cannot make fails as it would on the files. While writes fail, every answer carries a warning.
• §21.3: tsvz serve needs Unix-domain sockets. Where Python has none (Windows), it exits 1, and tsvz commands and TSVZClient work on the files.
• §20.7: on .tsv-family files a -d longer than one character is used, as in 3.39, with a warning; a -d that does not decode is a usage error.

Website

The tsvz.org site lives in tsvz-org/tsvz.org. It renders tsvz-spec-v1.md from a checkout of this repository next to its own, and its tests also check the spec: every § reference names a real section, and the spec uses only markdown the site renders. Run them after editing the spec (see Tests); CI runs them on every push here, and the site publishes main's spec each morning.

Conformance testing

tsvzconform.py tests how closely an implementation of tsvz-spec-v1, in any language, follows the spec. Its cases are JSON files in conformance/cases/, each naming the § clauses it tests, and it drives an implementation only through the spec's command line (§20) and handler socket (§21).

python3 tsvzconform.py --impl ./tsvz --feature crc32,sha256,handler
python3 tsvzconform.py --impl ./tsvz --class reader --only 13

Each failure is printed with its input, what was expected, what came back and a command line that reruns it alone; the summary gives a verdict for each conformance class (reader, writer, cli, handler). conformance/README.md explains the options and the features to claim, and how to load the cases into another language's tests.

Benchmarking implementations

tsvzbench.py compares implementations of tsvz-spec-v1, in any language, side by side on one machine. It drives each one only through the spec's command line (§20) and, when it has one, its handler socket (§21), so any conformant tsvz tool can be measured as it is.

python3 tsvzbench.py --impl ref='python3 TSVZ.py' --impl new='../tsvz-rs/target/release/tsvz' \
                     --suite quick --work-dir /mnt/fast-disk/tsvzbench

• --impl NAME=COMMAND, repeatable: the first is the baseline, and every other implementation gets a × ratio against it (below 1 is faster). A relative COMMAND runs from the directory you start tsvzbench in.
• --suite smoke|quick|standard|large: smoke checks a setup in seconds, quick takes about a minute, standard (the default, up to 1M rows) about five, and large (up to 10M rows) twenty or more and ~5 GB of memory for the harness itself.
• --only cold,bulk,perop,maint picks families: one-shot replays, set - and delete - batches, single operations (CLI and handler), and scrub/clear.
• --work-dir DIR keeps the generated fixtures between runs. Put it on the disk you mean to measure: not a tmpfs, and not a FUSE file system such as mergerfs (the harness warns about those). The default is a tsvzbench-* directory here.
• --json PATH also writes every raw sample.

The harness writes its own input stores, so every implementation reads the same bytes, and checks every result against its own replay model: ✓ right, ✗ wrong output or files, ! N an unexpected exit status, ⏱ timed out, refused an allowed refusal. It exits 1 when any cell failed. Each command runs under a small launcher that measures it with wait4; a peak RSS shown as ≤8.0 MiB is the launcher's own size, so the true peak is no higher. Compare an implementation with itself to see the noise: a few percent per cell, but the socket rows, whose latencies are tens of microseconds, can move 10–20% between runs, because a single run can land in a slower mode as a whole.

Tests

python3 -m pytest TSVZ_test.py -q                         # everything, incl. 3.39 differential tests
python3 -m pytest ../tsvz.org/tsvz_site_test.py -q        # the spec as tsvz.org renders it (needs the tsvz.org checkout next to this one)
python3 -m pytest tsvzbench_test.py -q                    # the benchmark harness
python3 -m pytest tsvzconform_test.py -q                  # the conformance runner and cases, against the reference
python3 benchTSVZ.py /tmp/b.tsv -n 100000 --compare-old    # benchmark (not a test)

TSVZ_old.py is the frozen 3.39 module the differential tests compare against; it is not installed. CI (.github/workflows/test.yml) runs the three test files on Python 3.6, 3.9, 3.12 and 3.14. Accepted limitations are listed in docs/limitations.md.

License

GPL-3.0-or-later © Yufei Pan (pan@zopyr.us)