dygest 0.4

DYGEST: Document Insights Generator

🌞 dygest: Document Insights Generator

[!NOTE] dygest is a text analysis tool that extracts insights from .txt files, generating summaries, keywords, TOCs, and performing Named Entity Recognition (NER).

Info

dygest was created to gain fast insights into longer transcripts of audio and video content by retrieving relevant topics and providing an easy to use HTML interface with short cuts from summaries to corresponding text chunks. NER processing further enhances those insights by identifying names of individuals, organisations, locations etc.

Features 🧩

• Text insights
  Generate concise insights for your text files using various LLM services by creating summaries, keywords, table of contents (TOC) and named entities (NER).

• Unified LLM Interface
  dygest uses litellm and provides integration for various LLM service providers: OpenAI, Anthropic, HuggingFace, Groq, Ollama etc. Check the complete provider list for all available services.

• Token Friendly
  dygest performs token-heavy text analysis and summarization tasks. Therefore, the underlying LLM pipeline can be tailored to your needs and specific rate limits using a mixed experts approach.

• Mixed Experts Approach
  dygest utilizes two fully customizable LLMs to handle different processing tasks. The first, referred to as the light_model, is designed for lighter tasks such as summarization and keyword extraction. The second, called the expert_model, is optimized for more complex tasks like constructing Tables of Contents (TOCs).

  This flexibility allows for various pipeline configurations. For example, the light_model can run locally using Ollama, while the expert_model can leverage an external API service like OpenAI or Groq. This approach ensures efficiency and adaptability based on specific requirements.

[!TIP] As the expert_model is dealing with a lot of input content it is recommended to use a larger LLM (>=32B) for this task. Smaller LLMs (3B to 7B) perform well as light_model.

• Named Entity Recognition (NER)
  Named Entity Recognition via fast and reliable flair framework (identifies persons, organisations, locations etc.).

• User-friendly HTML Editor
  By default dygest will create a .html file that can be viewed in standard browsers and combines summaries, keywords, TOC and NER for your text. It features a text editor for you to make further changes.

• Export Formats: .json .csv .html

Requirements

• 🐍 Python >=3.10
• 🔑 API keys for LLM services like OpenAI, Anthropic and Groq and / or a running Ollama instance

[!NOTE] API Keys have to be stored in your environment (e.g. export $OPENAI_API_KEY=skj....)

Installation

Install with pip

Create a Python virtual environment

python3 -m venv venv

Activate the environment

source venv/bin/activate

Install dygest

pip install dygest

Install from source

Clone this repository

git clone https://github.com/tsmdt/dygest.git
cd dygest

Create a Python virtual environment

python3 -m venv venv

Activate the environment

source venv/bin/activate

Install dygest

pip install .

Usage

Configuration

Customize the dygest LLM pipeline by running the dygest config command:

 Usage: dygest config [OPTIONS]

 Configure LLMs, Embeddings and Named Entity Recognition.

╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --light_model      -l                 TEXT     LLM model name for lighter tasks (summarization, keywords) [default: None]                │
│ --expert_model     -x                 TEXT     LLM model name for heavier tasks (TOCs). [default: None]                                  │
│ --embedding_model  -e                 TEXT     Embedding model name. [default: None]                                                     │
│ --temperature      -t                 FLOAT    Temperature of LLM. [default: None]                                                       │
│ --sleep            -s                 FLOAT    Pause LLM requests to prevent rate limit errors (in seconds). [default: None]             │
│ --chunk_size       -c                 INTEGER  Maximum number of tokens per chunk. [default: None]                                       │
│ --ner                     --no-ner             Enable Named Entity Recognition (NER). Defaults to False. [default: no-ner]               │
│ --precise                 --fast               Enable precise mode for NER. Defaults to fast mode. [default: fast]                       │
│ --lang             -lang              TEXT     Language of file(s) for NER. Defaults to auto-detection. [default: None]                  │
│ --api_base         -api               TEXT     Set custom API base url for providers like Ollama and Hugginface. [default: None]         │
│ --view_config      -v                          View loaded config parameters.                                                            │
│ --help                                         Show this message and exit.                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

The configuration is saved as dygest_config.yaml in the project directory. The .yaml config looks like this:

light_model: ollama/mistral:latest
expert_model: groq/llama-3.3-70b-versatile
embedding_model: ollama/nomic-embed-text:latest
temperature: 0.4
chunk_size: 1000
ner: true
language: auto
precise: false
api_base: null
sleep: 0

Processing

Run the dygest LLM pipeline with the dygest run command:

 Usage: dygest run [OPTIONS]

 Create insights for your documents (summaries, keywords, TOCs).

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --files            -f         TEXT                 Path to the input folder or .txt file. [default: None]                       │
│ --output_dir       -o         TEXT                 If not provided, outputs will be saved in the input folder. [default: None]  │
│ --export_format    -ex        [all|json|csv|html]  Set the data format for exporting. [default: html]                           │
│ --toc              -t                              Create a Table of Contents (TOC) for the text. Defaults to False.            │
│ --summarize        -s                              Include a short summary for the text. Defaults to False.                     │
│ --keywords         -k                              Create descriptive keywords for the text. Defaults to False.                 │
│ --sim_threshold    -sim       FLOAT                Similarity threshold for removing duplicate topics. [default: 0.85]          │
│ --verbose          -v                              Enable verbose output. Defaults to False.                                    │
│ --export_metadata  -meta                           Enable exporting metadata to output file(s). Defaults to False.              │
│ --help                                             Show this message and exit.                                                  │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Export formats

Example for JSON output

{
    "filename": "Embeddings_What_they_are_and_why_they_matter_en",
    "output_filepath": "output/Embeddings_What_they_are_and_why_they_matter_en.json",
    "input_text": "So Simon Wilson is going to be our last speaker before our break. So let's delve into ...",  # Full input text is abbreviated in this example
    "language": "en",
    "chunk_size": 1000,
    "token_count": 8705,
    "light_model": "ollama/qwen2.5:3b",
    "expert_model": "groq/llama-3.3-70b-versatile",
    "summary": "Embeddings, a technology used in large language models, represent text as numerical vectors, allowing for the understanding of concepts and relationships between them. This technology is utilized in various applications, including semantic searches, content recommendation, and text analysis. Researchers and developers, such as Simon Wilson, have demonstrated the potential of embeddings in creating systems that serve up related content and identify similar articles based on their similarity scores. The use of embeddings has also been explored in other areas, including geospatial SQL queries, image embedding, and audio incorporation. Overall, embeddings have become a fundamental component in the development of language models and AI frameworks, enabling advanced capabilities such as retrieval augmented generation and dimension reduction.",
    "keywords": [
        "large language models",
        "GeoPoly",
        "SQLite",
        "text representation",
        "Embedding vectors",
        "Similarity scores",
        "clustering",
        "geospatial SQL",
        "text clustering",
        "Serverless hosting",
        "text search",
        "OpenAI API",
        "Word2Vec",
        "semantic search",
        "browser compatibility",
        "related content",
        "retrieval augmented generation",
        "vibebased search",
        "code functions",
        "ImageBind",
        "function lookup",
        "Faucet Finder",
        "Clip",
        "vector databases",
        "LangChain",
        "PCA",
        "blog question answering",
        "concrete vs abstract",
        "TIL blog",
        "LLM",
        "GeoPackage",
        "GitHub Actions",
        "multimodal"
    ],
    "toc": [
        {
            "headline": "Embeddings and Similarity",
            "topics": [
                {
                    "summary": "Understanding Embeddings in Data Exploration",
                    "location": "S4"
                },
                {
                    "summary": "Using Embeddings for Similarity Analysis",
                    "location": "S85"
                },
                {
                    "summary": "OpenAI API for Embeddings",
                    "location": "S86"
                },
                {
                    "summary": "Analyzing Text for Clusters and Embeddings",
                    "location": "S316"
                },
                {
                    "summary": "Flexible Embedding Tools",
                    "location": "S407"
                }
            ]
        },
        {
            "headline": "Data Journalism and Tools",
            "topics": [
                {
                    "summary": "Simon Wilson's Background and Achievements",
                    "location": "S20"
                }
            ]
        },
        {
            "headline": "Geospatial and SQL",
            "topics": [
                {
                    "summary": "Geospatial SQL Queries Overview",
                    "location": "S43"
                }
            ]
        },
        {
            "headline": "Serverless and Hosting",
            "topics": [
                {
                    "summary": "Bake-to-Data Architecture Pattern",
                    "location": "S122"
                }
            ]
        },
        {
            "headline": "Language Models and Search",
            "topics": [
                {
                    "summary": "LLM Command-Line Utility",
                    "location": "S123"
                },
                {
                    "summary": "Vibe-Based Semantic Search for Readmes",
                    "location": "S222"
                },
                {
                    "summary": "Image-Text Similarity in Browser",
                    "location": "S223"
                },
                {
                    "summary": "Vibes-based search for faucets and other items",
                    "location": "S265"
                }
            ]
        },
        {
            "headline": "Databases and Indexing",
            "topics": [
                {
                    "summary": "Specialized Indexing Solutions",
                    "location": "S355"
                }
            ]
        },
        {
            "headline": "Multimodal and Browser",
            "topics": [
                {
                    "summary": "Fascinating Multimodal Space",
                    "location": "S408"
                },
                {
                    "summary": "Smaller and More Accessible",
                    "location": "S426"
                }
            ]
        }
    ]
}

Acknowledgments

dygest uses great python packages:

• litellm: https://github.com/BerriAI/litellm
• flair: https://github.com/flairNLP/flair
• typer: https://github.com/fastapi/typer
• json_repair: https://github.com/mangiucugna/json_repair