alea-preprocess
Description
Efficient, accessible preprocessing routines for pretrain, SFT, and DPO training data preparation.
This library is part of ALEA's open source large language model training pipeline, used in the research and development of the KL3M project.
Installation
Note that this project is a work-in-progress and relies on compiled Rust code. As such, it is recommended to install the package from GitHub source until a stable release is available.
You can install the latest release from PyPI using pip:
pip install alea-preprocess
You can install a development version of the package by running the following command:
poetry run maturin develop
Examples
Example use cases are currently available under the tests/ directory.
Additional documentation and examples will be provided in the future.
License
This ALEA project is released under the MIT License. See the LICENSE file for details.
Support
If you encounter any issues or have questions about using this ALEA project, please open an issue on GitHub.
Learn More
To learn more about ALEA and its software and research projects like KL3M, visit the ALEA website.
1 maintainer
Metadata
1 maintainer
Release files for alea-preprocess 0.1.12
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| alea_preprocess-0.1.12.tar.gz | 82.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| alea_preprocess-0.1.12-cp312-cp312-manylinux_2_34_x86_64.whl | CPython 3.12 | CPython 3.12 | Linux glibc 2.34+ x86-64 | Details |
Total release size: 9.3 MB
Release files / alea_preprocess-0.1.12.tar.gz
| Download URL | alea_preprocess-0.1.12.tar.gz |
|---|---|
| Size | 82.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
be4c6f0b8dc81bcf27eadee4bdad825cd27f8a8c40e73789fec6b82ef921124a
|
|
BLAKE2b-256 checksum How to use checksums |
3d653ca921037b1f7d14cef1aa7dd04d3ef5ead63e53b2ecb434e12cfcef6cd7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.7.4
|
Release files / alea_preprocess-0.1.12-cp312-cp312-manylinux_2_34_x86_64.whl
| Download URL | alea_preprocess-0.1.12-cp312-cp312-manylinux_2_34_x86_64.whl |
|---|---|
| Size | 9.2 MB |
| Tags | CPython 3.12 Linux glibc 2.34+ x86-64 |
|
SHA-256 checksum How to use checksums |
cd12af2ad853b5e91b83676157fda337cb35c9db95bdf871cae1a50b45620519
|
|
BLAKE2b-256 checksum How to use checksums |
045b0e9a0d630fbbe6cddac68a76dac8638467fc1f78b4758370f8b580c4231f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.7.4
|
