A utility for storing and reading files for LM training.
Project description
LM_Dataformat
![Coverage Status](https://pypi-camo.freetls.fastly.net/4fed0e15f5f7c84df38d8dc3a0f297e288a808f3/68747470733a2f2f636f766572616c6c732e696f2f7265706f732f6769746875622f6c656f67616f322f6c6d5f64617461666f726d61742f62616467652e7376673f6272616e63683d6d6173746572)
Utilities for storing data for LM training.
Basic Usage
To write:
ar = Archive('output_dir')
for x in something():
# do other stuff
ar.add_data(somedocument, meta={
'example': stuff,
'someothermetadata': [othermetadata, otherrandomstuff],
'otherotherstuff': True
})
# remember to commit at the end!
ar.commit()
To read:
rdr = Reader('input_dir_or_file')
for doc in rdr.stream_data():
# do something with the document
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
lm_dataformat-0.0.20.tar.gz
(5.7 kB
view hashes)
Built Distribution
Close
Hashes for lm_dataformat-0.0.20-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 247468181c9c2fea33a663cdb2f6fea489ddf6741d216fe6b466e60f002705af |
|
MD5 | a0283cd76814e6c47d6c92947a4f7a26 |
|
BLAKE2b-256 | 7c5bddcd67b81e0a41c367c1e52e761fd6071407985cc67f621d5e16c5209ff0 |