Dead-simple and easy-to-hack RecordIO-like data storing. Inspired by MXNet's RecordIO and Protobuf.
Project description
Dead Simple Records
Easy to hack and dead simple RecordIO-like data storing. Inspired by MXNet's RecordIO and Protobuf.
(planned) features
- Binary-based data format with index file
- Easy custom serialization schemes
- Common serialization schemes (more TBA)
- Documentation
- Append, update, delete data sample
- Pack/unpack form
- Schema serialization
Quick start
See the overview section in the documentations.
Also, check out this notebook for a quick example, this notebook has not been updated in a while though.
How does it work?
Saving
- Open an empty file, this is the data file.
- Take whatever data you got, serialize it in a way (for example, write to PNG buffer)
- Write the serialized data to the said file buffer, store the offset and size of the data
- Repeat until all data are written
- Write all the offsets and sizes to another file (index file)
Loading
- Load the data file and the index file.
- When given an index, use the index file to get the offset and size of the data.
- Seek the file pointer to the offset, read the exact size and deserialize the data.
Prebuilt format
Customized serialization format
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
dsrecords-0.4.18.tar.gz
(10.1 kB
view hashes)