LLM Summary

Use an LLM to summarize text.

The purpose of the library is to summarize large texts, where core information is captured in the summary

Input: Paragraphs of text

Output: Summary

Install and use library

pip install llm_summary

from llm_summary.inference_api.summary_inference 
import SummaryInferenceProvider

text = "us business leaders ...."

inference_provider = SummaryInferenceProvider(model="llama3.2") 
summary = inference_provider.summarize(text)

print (summary)

Output:

Summary inference completed in 13.31 seconds

summary="Business leaders are donating large sums to Donald Trump's second inaugural fund, with predicted total donations exceeding $107m"

Environment setup

Local LLM - Ollama

Run a local LLM using OLLAMA.

• Download and install OLLAMA: https://ollama.com/download

• Download the LLM Model

ollama run gemma3:12b

Example

Model: gemma3:27b

import logging
import os
from llm_summary.inference_api.summary_inference import SummaryInferenceProvider
from llm_summary.inference_api.mlflow_config import MlFlowConfig

summarizer = SummaryInferenceProvider(
            model="gemma3:27b",
            strategy="ollama"
            ollama_host="http://localhost:11434"
        )

summarizer.summarize(text=text)
                            summary_text = result.summary

Prompt management

Prompts can vary between models/providers. Version controlling the prompts and annotating the prompt helps with faster evaluation.

MlFlow provides a prompt repository and is used to get the user prompt and system prompt.

E.g. prompts for news article summaries

-e MLFLOW_SYSTEM_PROMPT_ID=SUMMARY_System/@article \
-e MLFLOW_USER_PROMPT_ID=SUMMARY_User/@article \

Data pipelines - Consumers

Two data pipeline examples show how existing mongodb documents can be decorated with a summary:

• Decorate text with a summary

E.g.

Mongo DB Summary Decorator

In this example text is read from a MongoDb Collection and the a summary is generated. The entry in mongodb is updated with the summary.

The container would read a collection in batches and generates summaries.

docker run -e OLLAMA_MODEL=gemma3:12b \
           -e OLLAMA_HOST=http://ollama:11434 \
           -e MONGODB_CONNECTION_STRING=mongodb://mongodb:27017 \
           -e MLFLOW_SYSTEM_PROMPT_ID=SUMMARY_System/@article \
           -e MLFLOW_USER_PROMPT_ID=SUMMARY_User/@article \
           -e MLFLOW_TRACKING_HOST=http://mlflow:5000
           --network development_network \
           --rm \
           --name summarizer \
           --hostname summarizer \
           mongodb-summarizer

Dependencies

Ollama should be running in a docker container; or a network route should be available.

Run the following for a CPU-Only ollama instance:

docker run -d --rm 
    -v /usr/share/ollama/.ollama:/root/.ollama \
    -p 11434:11434 \
    --network development_network \
    --name ollama \
    --hostname ollama  \
    ollama/ollama

Note: replace the path /usr/share/ollama/.ollama to the host's ollama model path

Evaluation

Due to the sheer amount of model options and inference providers; it is necessary to evaluate.

Evaluation allows methodical selection of hyper parameters:

• User Prompt (based on model, text source and model provider)
• System Prompt (based on model, text source and model provider)
• Model

Evaluation criteria

The summary should contain 'key information' extracted from the larger text; the information can be subjective due to the following:

• The context (i.e. what is the subject matter, and how the summary is used)
• Definition 'key information'
• Writing style of the summary

LLM as a judge

Create an LLM Judge, the judge would judge summaries.

Evaluate the LLM as a judge

The LLM judge is key during evaluation of models/prompts.

Therefore the LLM judge should have a strong Score. The LLM judge will be evaluated using the following metrics:

LLM Judge F1-Score:

• True positives (TP): the summary is "similar" to the original
• False positives (FP): the summary contains additional irrelevant information
• True negatives (TN): the summary does not include trivial information
• False negatives (FN): the summary is missing significant information

Precision

Proportion of correctly summarized text (accuracy).

High precision means that when the model summarizes, it is likely to be correct.

$$ \begin{aligned} \text{Precision} &= \frac{TP}{TP + FP} \end{aligned} $$

Where:

• ( TP ) = True Positives
• ( FP ) = False Positives

Recall

Ability to summarize text.

High recall means the model has summarized significant information from the text.

$$ \begin{aligned} \text{Recall} &= \frac{TP}{TP + FN} \end{aligned} $$

Where:

• ( TP ) = True Positives
• ( FN ) = False Negatives

F1 Score

$$ \begin{aligned} F1 &= 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \end{aligned} $$

Experiments

Publish the evaluation metrics for each "experiment".

ML-Flow is a good tool for analysis.

Metrics

Below are some evaluations, the evaluations are based on the following metrics:

• LLM Model
• Quality of summarization
• Local vs. Cloud
• Estimate costs

Conclude the evaluation

Pick the metric most significant, then short-list the models, and do a cost analysis.

To evaluate several weights can be given to each of the metric.

E.g. load your metrics and use a ranker to evaluate:

(e.g. Accuracy: 1, Percentage:2, Distance: 3 and Latency: 1)

License

Copyright (C) 2026 Paul Eger

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.