textslinger 0.2.6

TextSlinger: Fast and Accurate Text Predictions in Python

This is a Python library for making text predictions using different types of language models. Current features:

• Predict the distribution over the next character given the previous text.
• Predict the most likely next words given the previous text and prefix of current word.
• Supports:
  • N-gram language models via KenLM.
  • Subword tokenized large language models (LLMs) via Hugging Face.
  • Byte tokenized LLMs via Hugging Face and Byte Latent Transformer.

Developer setup

Our code style is whatever the Black formatter says it should be. You should configure your IDE to format using Black when you save.

Windows

Windows is not supported by the library dependencies of textslinger. To use our library on Windows, you will need to use Windows Subsystem for Linux (WSL). Please follow the installation directions and use the Linux commands below. You will also need to install pip to install Python libraries. Follow the pip installation directions before proceeding.

Setting up a Python environment

If you don't have Miniforge installed in your user account you'll first need to do that.

To install Miniforge on MacOS using Apple Silicon:

curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
zsh Miniforge3-MacOSX-arm64.sh
~/miniforge3/bin/conda init zsh

To install Miniforge on Linux and WSL:

curl -LO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh

When installing on WSL, enter "yes" when asked if you want to automatically initialize conda. This sets up easier access to conda in your WSL environment. You can disable the automatic start of the base environment using the following conda command after restarting your terminal:

conda config --set auto_activate_base false

After installing Miniforge, be sure to close your terminal and start a new one. Create an environment as follows:

conda config --remove-key channels
conda config --add channels conda-forge
conda config --set channel_priority strict
conda create -n textslinger python=3.10 -y
conda activate textslinger

Installation of PyTorch

MacOS using Apple Silicon:

pip install torch torchvision torchaudio

Linux with CUDA support (GPU driver must support installed library version or greater. Run nvidia-smi to check driver support):

# CUDA 11.8 
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# CUDA 12.1
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Linux without CUDA support:

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

WSL with CUDA support (GPU driver must support installed library version or greater. Run nvidia-smi to check driver support). Due to a security vulnerability in earlier versions, WSL requires at least torch 2.6:

# CUDA 12.6
python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

WSL without CUDA support:

python -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

Test if the PyTorch installation worked:

python - <<'EOF'
import torch
print("Torch version:", torch.__version__)
print("MPS available:", torch.backends.mps.is_available())
print("MPS built:", torch.backends.mps.is_built())
print("CUDA available:", torch.cuda.is_available())
print("CUDA version:", torch.version.cuda)
print("CUDA device count:", torch.cuda.device_count())

if torch.backends.mps.is_available():
    x = torch.randn(2, 2, device="mps")
    print("Tensor device:", x.device)
elif torch.cuda.is_available():
    x = torch.randn(2, 2, device="cuda")
    print("Tensor device:", x.device)
else:
    x = torch.randn(2, 2)
    print("Tensor device:", x.device)
EOF

Installation of libraries

Install transformers (5.2.0 or greater required).

MacOS and Linux:

pip install transformers

WSL:

python -m pip install transformers

Check transformers version and model support:

python - <<'EOF'
import transformers
from transformers import __version__
from transformers.utils import is_torch_available

print("Version:", __version__)
print("File:", transformers.__file__)

# Check for BLT symbols that do NOT exist in stable 5.0.0
try:
    from transformers.models.blt.modeling_blt import BltModel
    print("BLT model available ✅")
except Exception as e:
    print("BLT model missing ❌", e)
EOF

Hugging Face Authentication

You will need to log in to Hugging Face using the CLI. This CLI is installed with the transformers library. Please see the Hugging Face CLI Login direction for how to do this. We recommend using a token so you do not need to manually enter your credentials.

hf auth login

Install other dependencies on MacOS and Linux:

pip install pytest scipy peft psutil datasets

# NOTE: increase MAX_ORDER if you plan to load n-gram models with longer context 
MAX_ORDER=12 pip install https://github.com/kpu/kenlm/archive/master.zip

Install other dependencies on WSL:

python -m pip install pytest scipy peft psutil datasets

If using the default WSL Ubuntu installation, the build-essential package is needed to build the KenLM library. You also need the unzip package to run the download.sh script later in this guide.

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install build-essential unzip
# NOTE: increase MAX_ORDER if you plan to load n-gram models with longer context 
export MAX_ORDER=12
python -m pip install https://github.com/kpu/kenlm/archive/master.zip

Fix harmless warning message on MacOS and Linux:

pip install --upgrade --force-reinstall setuptools

Fix harmless warning message on WSL:

CUDA 12.6 does not support setuptools versions 82.0.0+.

python -m pip install --upgrade --force-reinstall setuptools==81.0.0

Install TextSlinger itself so scripts can import the textslinger package from any working directory. Run this from the repository root on all platforms, the directory containing pyproject.toml:

python -m pip install --no-deps -e .

Testing installation

Download assets needed by the test suite and then run it. If you are on WSL, check if the assets directory is owned by root using ls -l. If so, restart your WSL session and rercheck the ownsership.

cd textslinger/assets
./download.sh
cd ..
pytest -v -rs

If you encounter an error with line endings (e.g., "\r: no such file or directory") when running download.sh, you can change the line endings using:

vim download.sh -c "set ff=unix" -c ":wq"

---

This material is based upon work supported by the NSF under Grant No. IIS-1909089 and IIS-2402876.