AI Handler: An engine which wraps certain huggingface models
Project description
AI Handler
This is a simple framework for running AI models. It makes use of the huggingface API which gives you a queue, threading, a simple API, and the ability to run Stable Diffusion and LLMs seamlessly from your local hardware.
This is not intended to be used as a standalone application.
It can easily be extended and used to power interfaces or it can be run from the command line.
AI Handler is a work in progress. It powers two projects at the moment, but may not be ready for general use.
Installation
Pre-requisites
System requirements
- Python 3.10.8
- pip 23.0.1
- CUDA toolkit 11.7
- CUDNN 8.6.0.163
- Cuda capable GPU
- 16gb+ ram
Create a venv and activate it:
Ubuntu
python -m venv venv
source venv/bin/activate
Windows
python -m venv venv
venv\Scripts\activate
Upgrade pip to latest
python.exe -m pip install --upgrade pip
Diffusers and transformers
Install modified versions of diffusers and transformers from these forks:
pip install git+https://github.com/w4ffl35/diffusers.git@ckpt_fix
pip install git+https://github.com/w4ffl35/transformers.git@tensor_fix
The author of aihandler
is also the author of those changes.
Ubuntu 20.04+
pip install aihandler
Windows 10+
First install torch
pip install torch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 --index-url https://download.pytorch.org/whl/cu117
Now install the rest
Install this repo pip install aihandler
Currently bitsandbytes on windows is bitsandbroken. Here's how you can hack around it:
git clone https://github.com/DeXtmL/bitsandbytes-win-prebuilt
git clone https://github.com/james-things/bitsandbytes-prebuilt-all_arch
copy bitsandbytes-win-prebuilt/*.dll <your_venv_path>/site-packages/bitsandbytes
copy bitsandbytes-prebuilt-all_arch/*.dll <your_venv_path>/site-packages/bitsandbytes
- Edit
main.py
in<your_venv_path>/site-packages/bitsandbytes/cuda_setup
- Find and replace all
ct.cdll.LoadLibrary(binary_path)
withct.cdll.LoadLibrary(str(binary_path))
- Find and replace all
if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None
withif torch.cuda.is_available(): return 'libbitsandbytes_cudaall.dll', None, None, None, None
Optional
These are optional instructions for installing TensorRT and Deepspeed for Windows
Install Tensor RT:
- Download TensorRT-8.4.3.1.Windows10.x86_64.cuda-11.6.cudnn8.4
- Git clone TensorRT 8.4.3.1
- Follow their instructions to build TensorRT-8.4.3.1 python wheel
- Install TensorRT
pip install tensorrt-*.whl
Install Deepspeed:
- Git clone Deepspeed 0.8.1
- Follow their instructions to build Deepspeed python wheel
- Install Deepspeed `pip install deepspeed-*.whl
Environment variables
AIRUNNER_ENVIRONMENT
-dev
orprod
. Defaults todev
. This controls the LOG_LEVELLOG_LEVEL
-FATAL
for production,DEBUG
for development. Override this to force a log level
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aihandler-1.8.14-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b63d457181fc650a7ff0c54b1327603232b95f8fc4534d5f1e10a2a83762166a |
|
MD5 | 0c2d0fee7e15882203bcab16ac6ff749 |
|
BLAKE2b-256 | dba2d009a92da9dded4f776f13515fa26aaa1af045e310904a85b84842656081 |