A multi-threaded explainability framework using Shapley values for LLM-based outputs.

Getting Started

Install the llmshap package (with all optional dependencies):

pip install "llmshap[all]"

Install in editable mode with all optional dependencies (after cloning the repository):

pip install -e ".[all]"

Documentation is available at llmSHAP Docs and a hands-on tutorial can be found here.

• Full documentation
• Tutorial

Example Usage

from llmSHAP import DataHandler, BasicPromptCodec, ShapleyAttribution
from llmSHAP.llm import OpenAIInterface

data = "In what city is the Eiffel Tower?"
handler = DataHandler(data, permanent_keys={0,3,4})
result = ShapleyAttribution(model=OpenAIInterface(model_name="gpt-4o-mini"),
                            data_handler=handler,
                            prompt_codec=BasicPromptCodec(system="Answer the question briefly."),
                            use_cache=True,
                            num_threads=16,
                            ).attribution()

print("\n\n### OUTPUT ###")
print(result.output)

print("\n\n### ATTRIBUTION ###")
print(result.attribution)

print("\n\n### HEATMAP ###")
print(result.render())

Multimodal Example with Image:

The following example shows llmSHAP with images.

from llmSHAP import DataHandler, BasicPromptCodec, ShapleyAttribution, Image
from llmSHAP.llm import OpenAIInterface

data = {
    "question": "Has our stockprice increased or decreased since the beginning?",
    "Num employees"        : "The company has about 450 employees.",
    "[IMAGE] Stock chart"  : Image(image_path="./docs/_static/demo-stock-price.png"),
    "Report release date"  : "Quarterly reports are released on the 15th.",
    "Headquarter Location" : "The headquarters is located in a mid-sized city.",
    "Num countries"        : "It has offices in three countries."
}

result = ShapleyAttribution(model=OpenAIInterface(model_name="gpt-5-mini", reasoning="low"),
                            data_handler=DataHandler(data, permanent_keys={"question"}),
                            prompt_codec=BasicPromptCodec(system="Answer the question briefly."),
                            use_cache=True,
                            num_threads=35,
                            ).attribution()

print("\n\n### OUTPUT ###")
print(result.output)

print("\n\n### HEATMAP ###")
print(result.render(abs_values=True, render_labels=True))

Embedding-Based Output Scoring

EmbeddingCosineSimilarity measures semantic similarity between outputs using embeddings. It supports two backends:

• API — any OpenAI-compatible embeddings endpoint via api_url_endpoint.
• Local — a sentence-transformers model downloaded on first use.

For the local backend, install the embeddings extra:

pip install "llmshap[embeddings]"

The example below uses the OpenAI API backend, which is already included in [all].

from llmSHAP import DataHandler, BasicPromptCodec, ShapleyAttribution, EmbeddingCosineSimilarity
from llmSHAP.llm import OpenAIInterface

data = "In what city is the Eiffel Tower?"
handler = DataHandler(data)
result = ShapleyAttribution(model=OpenAIInterface(model_name="gpt-4o-mini"),
                            data_handler=handler,
                            prompt_codec=BasicPromptCodec(system="Answer the question briefly."),
                            use_cache=True,
                            num_threads=16,
                            value_function=EmbeddingCosineSimilarity(
                                model_name = "text-embedding-3-small",
                                api_url_endpoint = "https://api.openai.com/v1")
                            ).attribution()

print("\n\n### OUTPUT ###")
print(result.output)

print("\n\n### HEATMAP ###")
print(result.render(abs_values=True, render_labels=True))

Example data

You can pass either a string or a dictionary:

from llmSHAP import DataHandler

# String input
data = "The quick brown fox jumps over the lazy dog"
handler = DataHandler(data)

# Dictionary input
data = {"a": "The", "b": "quick", "c": "brown", "d": "fox"}
handler = DataHandler(data)

To exclude certain keys from the computations, use permanent_keys:

from llmSHAP import DataHandler

data = {"a": "The", "b": "quick", "c": "brown", "d": "fox"}
handler = DataHandler(data, permanent_keys={"a", "d"})

# Get data with index 1 WITHOUT the permanent features.
print(handler.get_data({1}, exclude_permanent_keys=True, mask=False))
# Output: {'b': 'quick'}

# Get data with index 1 AND the permanent features.
print(handler.get_data({1}, exclude_permanent_keys=False, mask=False))
# Output: {'a': 'The', 'b': 'quick', 'd': 'fox'}

Comparison with TokenSHAP

Stars ⭐️