LLM Foundry

This repository contains code for training, finetuning, evaluating, and deploying LLMs for inference with Composer and the MosaicML platform. Designed to be easy-to-use, efficient and flexible, this codebase is designed to enable rapid experimentation with the latest techniques.

You'll find in this repo:

llmfoundry/ - source code for models, datasets, callbacks, utilities, etc.
scripts/ - scripts to run LLM workloads
- data_prep/ - convert text data from original sources to StreamingDataset format
- train/ - train or finetune HuggingFace and MPT models from 125M - 70B parameters
  - train/benchmarking - profile training throughput and MFU
- inference/ - convert models to HuggingFace or ONNX format, and generate responses
  - inference/benchmarking - profile inference latency and throughput
- eval/ - evaluate LLMs on academic (or custom) in-context-learning tasks
mcli/ - launch any of these workloads using MCLI and the MosaicML platform

MPT

MPT-7B is a GPT-style model, and the first in the MosaicML Foundation Series of models. Trained on 1T tokens of a MosaicML-curated dataset, MPT-7B is open-source, commercially usable, and equivalent to LLaMa 7B on evaluation metrics. The MPT architecture contains all the latest techniques on LLM modeling -- Flash Attention for efficiency, Alibi for context length extrapolation, and stability improvements to mitigate loss spikes. The base model and several variants, including a 64K context length fine-tuned model (!!) are all available:

Model	Context Length	Download	Demo	Commercial use?
MPT-7B	2048	https://huggingface.co/mosaicml/mpt-7b		Yes
MPT-7B-Instruct	2048	https://huggingface.co/mosaicml/mpt-7b-instruct	Demo	Yes
MPT-7B-Chat	2048	https://huggingface.co/mosaicml/mpt-7b-chat	Demo	No
MPT-7B-StoryWriter	65536	https://huggingface.co/mosaicml/mpt-7b-storywriter		Yes

To try out these models locally, follow the instructions in scripts/inference/README.md to prompt HF models using our hf_generate.py or hf_chat.py scripts.

Latest News

Prerequisites

Here's what you need to get started with our LLM stack:

Use a Docker image with PyTorch 1.13+, e.g. MosaicML's PyTorch base image
- Recommended tag: mosaicml/pytorch:1.13.1_cu117-python3.10-ubuntu20.04
- This image comes pre-configured with the following dependencies:
  - PyTorch Version: 1.13.1
  - CUDA Version: 11.7
  - Python Version: 3.10
  - Ubuntu Version: 20.04
  - FlashAttention kernels from HazyResearch
Use a system with NVIDIA GPUs

Installation

To get started, clone this repo and install the requirements:

git clone https://github.com/mosaicml/llm-foundry.git
cd llm-foundry

# Optional: we highly recommend creating and using a virtual environment
python -m venv llmfoundry-venv
source llmfoundry-venv/bin/activate

pip install -e ".[gpu]"  # or pip install -e . if no NVIDIA GPU

Quickstart

Here is an end-to-end workflow for preparing a subset of the C4 dataset, training an MPT-125M model for 10 batches, converting the model to HuggingFace format, evaluating the model on the Winograd challenge, and generating responses to prompts.

If you have a write-enabled HuggingFace auth token, you can optionally upload your model to the Hub! Just export your token like this:

export HUGGING_FACE_HUB_TOKEN=your-auth-token

and uncomment the line containing --hf_repo_for_upload ....

(Remember this is a quickstart just to demonstrate the tools -- To get good quality, the LLM must be trained for longer than 10 batches 😄)

cd scripts

# Convert C4 dataset to StreamingDataset format
python data_prep/convert_dataset_hf.py \
  --dataset c4 --data_subset en \
  --out_root my-copy-c4 --splits train_small val_small \
  --concat_tokens 2048 --tokenizer EleutherAI/gpt-neox-20b --eos_text '<|endoftext|>'

# Train an MPT-125m model for 10 batches
composer train/train.py \
  train/yamls/mpt/125m.yaml \
  data_local=my-copy-c4 \
  train_loader.dataset.split=train_small \
  eval_loader.dataset.split=val_small \
  max_duration=10ba \
  eval_interval=0 \
  save_folder=mpt-125m

# Convert the model to HuggingFace format
python inference/convert_composer_to_hf.py \
  --composer_path mpt-125m/ep0-ba10-rank0.pt \
  --hf_output_path mpt-125m-hf \
  --output_precision bf16 \
  # --hf_repo_for_upload user-org/repo-name

# Evaluate the model on Winograd
python eval/eval.py \
  eval/yamls/hf_eval.yaml \
  icl_tasks=eval/yamls/winograd.yaml \
  model_name_or_path=mpt-125m-hf

# Generate responses to prompts
python inference/hf_generate.py \
  --name_or_path mpt-125m-hf \
  --max_new_tokens 256 \
  --prompts \
    "The answer to life, the universe, and happiness is" \
    "Here's a quick recipe for baking chocolate chip cookies: Start by"

Contact Us

If you run into any problems with the code, please file Github issues directly to this repo.

If you want to train LLMs on the MosaicML platform, reach out to us at demo@mosaicml.com!

Metadata

Release files for llm-foundry 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-foundry 0.1.0
File	Size	Uploaded	Provenance
llm-foundry-0.1.0.tar.gz	106.3 kB	May 10, 2023	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for llm-foundry 0.1.0
File	Interpreter	ABI	Platform	Reset
llm_foundry-0.1.0-py3-none-any.whl 106.0 kB May 10, 2023 Without provenance	Python 3	none	any	Details

Total release size: 212.3 kB

Release files / llm-foundry-0.1.0.tar.gz

Download URL	llm-foundry-0.1.0.tar.gz
Size	106.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`7ff9e195d7a8a4d609a87272dda48eca1a952397240808b26cc3f8f7a8dba090`
BLAKE2b-256 checksum How to use checksums	`3aae57c3346bd35154ba2f506fc13cb07b38cefc4f5511cb34705c1c34c568f1`
Upload date	May 10, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.11.3`

Release files / llm_foundry-0.1.0-py3-none-any.whl

Download URL	llm_foundry-0.1.0-py3-none-any.whl
Size	106.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`c94955cbdb02dc3df25d862a35fd07f12fc7be8107a30d9ee87c88e43a1ef787`
BLAKE2b-256 checksum How to use checksums	`d732b1afffb2d76c0be72fcda3184050bc4ebc32695a645ef95bdb35103007ad`
Upload date	May 10, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.1 CPython/3.11.3`

Security for llm-foundry 0.1.0

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.