dolma 0.6.0

Data filters

dolma

Data to feed OLMo's Appetite

Data and tools for generating and inspecting OLMo pre-training data.

Setup

Install Rust

curl https://sh.rustup.rs -sSf | sh

Install CMake

• On Mac OSX with brew install cmake
• On Linux with apt-get install cmake

Install OpenSSL

• On Mac OSX with brew install openssl re2
• On Linux with apt-get install openssl

Install Protobuf

• On Mac OSX with brew install protobuf
• On Linux with apt-get install protobuf-compiler

Setting up Python

conda create -n dolma python=3.10

Install Maturin

pip install maturin
maturin develop

Installing this repository

cd dolma
pip install -e .

Citation

If you use this repository, please cite it as:

@software{dolma,
    author = {{Soldaini, Luca and Lo, Kyle and Kinney, Rodney and Naik, Aakanksha and Ravichander, Abhilasha and Bhagia, Akshita and Groeneveld, Dirk and Schwenk, Dustin and Magnusson, Ian and Chandu, Khyathi}},
    license = {{Apache-2.0}},
    title = {{DOLMa}},
    url = {https://github.com/allenai/dolma}
}