A utility for storing and reading files for LM training.
Project description
LM_Dataformat
Utilities for storing data for LM training.
Basic Usage
To write:
ar = Archive('output_dir')
for x in something():
# do other stuff
ar.add_data(somedocument, meta={
'example': stuff,
'someothermetadata': [othermetadata, otherrandomstuff],
'otherotherstuff': True
})
# remember to commit at the end!
ar.commit()
To read:
rdr = Reader('input_dir_or_file')
for doc in rdr.stream_data():
# do something with the document
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
lm_dataformat-0.0.20.tar.gz
(5.7 kB
view hashes)
Built Distribution
Close
Hashes for lm_dataformat-0.0.20-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 247468181c9c2fea33a663cdb2f6fea489ddf6741d216fe6b466e60f002705af |
|
MD5 | a0283cd76814e6c47d6c92947a4f7a26 |
|
BLAKE2b-256 | 7c5bddcd67b81e0a41c367c1e52e761fd6071407985cc67f621d5e16c5209ff0 |