PortableTab

PortableTab is a Python library that allows for serialization of typed tables into a set of files, as well as deserialization of specific rows extracted from the files.

Features

The serialized files are independent of OS and CPU architecture, so it can be used to create portable table which is useful when working with large datasets that need to be shared between different systems or environments.

It also allows fast deserialization of only specified rows without loading the entire table into memory, so it does not take time to load and deserialize the table on the first access, nor consume memory during execution.

• Capn' Proto is used for serialization, making the file format portable.
• Since PortableTab uses mmap for file access, it does not consume much memory even when handling large tables.
• Indexes on strings are created using Marisa-trie, the output files are also portable and accessible using mmap.

Limitations

The tables are serialized into compact files so they cannot be dynamically modified.

• Rows can only be retrieved at their specified position. If you want to access by an attribute such as id, you must create an index on that attribute.
• Updating records in serialized files is possible but very slow.
• It is not possible to insert rows in the middle of a serialized file. If you want to insert rows in the middle, the only way is to deserialize the entire table and recreate another table.

How to use

Please refer to the documentation at PortableTab Document.

Development status

Unstable alpha version.

License

This package is available according to the MIT license.

Author

Takeshi SAGARA sagara@info-proto.com