bentoml-unsloth 0.1.2

BentoML: The easiest way to serve AI apps and models

BentoML Unsloth integrations

Installation

pip install "bentoml[unsloth]"

Examples.

See train.py

API

To use this integration, one can use bentoml.unsloth.build_bento:

bentoml.unsloth.build_bento(model, tokenizer)

If you model is continued froma fine-tuned checkpoint, then model_name must be passed as well:

bentoml.unsloth.build_bento(model, tokenizer, model_name="llama-3-continued-from-checkpoint")

[!important]

Make sure to save the chat templates to tokenizer instance to make sure generations are correct based on how you setup your data pipeline. See example and documentation for more information.