A Jupyter kernel using LLM models from Huggingface
Project description
chatbot_kernel
A Jupyter kernel to use Jupyter as a chat window, running downloaded LLMs locally.
Installation
This kernel package can be install from PyPI
$ pip install chatbot_kernel
Once the package installed, the kernel spec can be installed to home directory by command:
$ python -m chatbot_kernel install --user
If you are using virtualenv, do
$ python -m chatbot_kernel install --sys-prefix
instead.
If you install the package to a virtual environment, you may need to set up JUPYTER_PATH=/path/to/venv/share/jupyter:$JUPYTER_PATH
so that jupyter can find the kernel
Usage
A few magics are available in the kernel:
%help
: Print help messages%config
: Set advanced configuration. See more in%config help
%load
: Load a pretrained LLM%hf_home
: Set the path to find downloaded LLMs, similar to setHF_HOME
environment variable%model_list
: Show the available LLMs%new_chat
: Clean up the chat history
Before start chatting, you need to at least download a model from HuggingFace. For example, huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct
.
Once models are downloaded, launch a jupyter notebook/lab and execute %load <model>
and start chatting. Here is an example:
%load meta-llama/Meta-Llama-3-8B-Instruct
hi
who are you
Caveat
Currently, the kernel use the AutoModelForCausalLM
and it is not supported by all models.
A few models have been tested:
meta-llama/Meta-Llama-3-8B-Instruct
mistralai/Mistral-7B-Instruct-v0.3
: needssentencepiece
dependencyunsloth/llama-3-8b-Instruct-bnb-4bit
: needsbitsandbytes
dependency
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for chatbot_kernel-0.3.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f7e26dcf9acfaa3b61ed68ea12eeb7984211f31268a6d55ba345fbb7571399b |
|
MD5 | 6d3ca017677b482e4a589e65c67487bd |
|
BLAKE2b-256 | 8b44e93dad51183b2f0e7eb600123bd8b0a462a2c0a7a5a2f79598c483374744 |