inincompatibility 0.0.8

Usage Guidelines

Installation:

To install the inincompatibility package, run:

pip install inincompatibility

Example: Making Your LLMs Callable Like an API:

First, make your LLMs (e.g., Llama-3.2-1B) callable functions:

# your_llm.py
import torch
from transformers import pipeline as pl

m = "meta-llama/Llama-3.2-1B"
p = pl("text-generation", model=m, device="cuda")

def llm_qa(msg: list) -> dict:
    return p(msg)[0]["generated_text"]

Next, create another Python file (e.g., to_import_ori.py) to import the LLM function:

# to_import_ori.py
from your_llm import llm_qa

Then, use the inincompatibility CLI to generate the necessary importable code and then run the LLM in its (Python / conda) environment:

conda activate llama
pip install inincompatibility
python -m inincompatibility -i to_import_ori.py -o to_import.py

A file named to_import.py (specified by the -o argument) will be generated as follows:

# Generated by `inincompatibility`.
# **Not** depend on `incompatibility`. :)
# You can directly `import` this file in another environment.
import socket
import pickle

BUFFER_SIZE = 65536

client = socket.socket(2, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 57849))

def _func_eval(func, args, kwargs):
    data = pickle.dumps((func, args, kwargs))
    client.sendall(data)
    return pickle.loads(client.recv(BUFFER_SIZE))

def _inincompatibility_remote_eval(*args, **kwargs):
    return _func_eval('_inincompatibility_remote_eval', args, kwargs)

def _inincompatibility_remote_exec(*args, **kwargs):
    return _func_eval('_inincompatibility_remote_exec', args, kwargs)

def llm_qa(*args, **kwargs):
    return _func_eval('llm_qa', args, kwargs)

Now, you can directly import the generated code in another (Python / conda) environment:

# main.py
from to_import import llm_qa
print(llm_qa('The key to life is'))

Run your main script (e.g., main.py) in the target environment:

conda activate blackbox
python main.py

For more details, check out the sample-llama directory on GitHub.

Example: Additional Samples:

For more usage examples, visit the sample-local directory on GitHub.