Skip to main content

DISK

Incremental conStruction of Domain Knowledge graph.

Overview

DISK is a comprehensive toolkit for extracting knowledge from PDF documents and building domain knowledge graphs through text distillation, entity/relation extraction, and semantic merging. The system provides a modular pipeline that transforms unstructured PDF documents into structured knowledge representations.

Core Capabilities

  • Document Distillation: Extract and validate text blocks, tables, and images from PDF documents
  • Entity Extraction: Identify and extract domain entities with semantic embeddings
  • Relation Extraction: Discover relationships between entities with contextual understanding
  • Knowledge Graph Construction: Build and manage knowledge graphs with incremental updates
  • Semantic Merging: Intelligently merge similar entities and relations using cosine similarity

Architecture

System Architecture

graph TB
    subgraph "Input Layer"
        PDF[PDF Document]
    end

    subgraph "Distillation Layer"
        Distiller[PDF Distiller]
        TextBlocks[Validated Text Blocks]
    end

    subgraph "Extraction Layer"
        EntExtractor[Entity Extractor]
        RelExtractor[Relation Extractor]
        UnifiedExtractor[Unified Extractor]
        Entities[(Entities + Embeddings)]
        Relations[(Relations + Embeddings)]
    end

    subgraph "Processing Layer"
        Merger[Semantic Merger]
        Manager[KG Manager]
    end

    subgraph "Output Layer"
        KG[Knowledge Graph]
        Logs[Logs & Results]
    end

    subgraph "Configuration"
        Config[LLM Config]
        Embed[Embeddings Model]
    end

    PDF --> Distiller
    Distiller --> TextBlocks

    TextBlocks --> EntExtractor
    TextBlocks --> RelExtractor
    TextBlocks --> UnifiedExtractor

    EntExtractor --> Entities
    RelExtractor --> Relations
    UnifiedExtractor --> Entities
    UnifiedExtractor --> Relations

    Entities --> Merger
    Relations --> Merger
    Merger --> Manager

    Manager --> KG
    Manager --> Logs

    Config --> EntExtractor
    Config --> RelExtractor
    Config --> UnifiedExtractor
    Embed --> EntExtractor
    Embed --> RelExtractor
    Embed --> UnifiedExtractor
    Embed --> Merger

    style PDF fill:#e1f5fe
    style KG fill:#c8e6c9
    style Distiller fill:#fff3e0
    style Merger fill:#f3e5f5
    style Manager fill:#e8f5e9

Module Structure

graph LR
    subgraph DISK
        DiskMain[disk.py<br/>Main Entry Point]

        subgraph Core
            Distiller[distiller/<br/>PDF Distillation]
            Extractor[extractor/<br/>Information Extraction]
            MergerMod[merger/<br/>Knowledge Merging]
            ManagerMod[manager/<br/>KG Management]
        end

        subgraph Support
            Models[models/<br/>Data Models]
            Utils[utils/<br/>Utilities]
            ConfigMod[config/<br/>Configuration]
        end
    end

    DiskMain --> Distiller
    DiskMain --> Extractor
    DiskMain --> MergerMod
    DiskMain --> ManagerMod

    Extractor --> Models
    MergerMod --> Models
    ManagerMod --> Models

    Distiller --> Utils
    Extractor --> Utils
    ManagerMod --> Utils

    DiskMain --> ConfigMod

    style DiskMain fill:#1976d2,color:#fff
    style Distiller fill:#ffa726
    style Extractor fill:#42a5f5
    style MergerMod fill:#ab47bc
    style ManagerMod fill:#66bb6a

Data Flow

sequenceDiagram
    participant User
    participant DISK
    participant Distiller
    participant Extractor
    participant Merger
    participant Manager
    participant KG

    User->>DISK: build_knowledge_graph(pdf_path)
    DISK->>Distiller: extract_text_blocks(pdf)
    Distiller-->>DISK: validated_text_blocks

    loop For each text block
        DISK->>Extractor: extract_entities(text)
        Extractor-->>DISK: entities + embeddings

        DISK->>Extractor: extract_relations(text)
        Extractor-->>DISK: relations + embeddings

        DISK->>Merger: merge(new, existing)
        Merger-->>DISK: merged entities/relations
    end

    DISK->>Manager: add_entities(entities)
    DISK->>Manager: add_relations(relations)
    Manager->>KG: update_knowledge_graph
    DISK-->>User: Knowledge Graph

Modules

Distillation Module (distiller/)

  • pdf_distiller
    • extract paragraphs with intelligent validation
    • extract tables(to be improved)
    • extract imgs(to be improved)
    • filter out low-quality text blocks (references, incomplete sentences)

Extraction Module (extractor/)

  • entities_extractor

    • extract domain entities with labels and descriptions
    • generate semantic embeddings for each entity
  • relations_extractor

    • extract relationships between entities
    • generate semantic embeddings for each relation
  • extractor (unified)

    • extract both entities and relations in a single pass
    • optimized for incremental processing

Processing Modules

  • extract entities
  • extract relationships
  • semantic merging (merger/)
    • merge similar entities using cosine similarity
    • update relations after entity merging
    • configurable threshold (default: 0.8)
  • construct knowledge graph (manager/)
    • incremental knowledge graph construction
    • deduplication of entities and relations

Config

env

# use uv to manage the environment
uv venv
uv sync

LLM Configuration

  1. Copy the example configuration file:

    cp config.example.toml config.toml
    
  2. Edit config.toml to set your API keys and preferences:

[disk]
llm = "openai"  # Choose provider: openai, qwen, ollama, etc.

[disk.embeddings]
model = "text-embedding-3-small"
api_key = "ai-..."
api_url = "https://api.openai.com/v1"

[model.openai]
api_url = "https://api.openai.com/v1"
api_key = "ai-..."
model = "gpt-4o"

[model.other]
api_url = "https://api.otherprovider.com/v1"
api_key = "sk-..."
model = "gpt-4o"
  1. Supported providers:
    • OpenAI (default)
    • Qwen (DashScope)
    • Kimi (Moonshot)
    • Ollama (Local)
    • All other providers that support OpenAI-compatible APIs

You can switch providers by changing the llm field in [disk] or using the runtime switch() function.

Contrast

merge

  • itext2kg

    [INFO] Wohoo! Entity was matched --- [poor deep semantic understanding in traditional ie models:Limitation] --merged--> [cosine similarity ignores deep semantic differences:Limitation]
    

Metadata

Release files for disk-kg 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for disk-kg 1.2.0
File Size Uploaded Provenance Actions
disk_kg-1.2.0.tar.gz 1.9 MB Without provenance Details for disk_kg-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for disk-kg 1.2.0
File Interpreter ABI Platform Actions
disk_kg-1.2.0-py3-none-any.whl Python 3 none any Details for disk_kg-1.2.0-py3-none-any.whl

Total release size: 2.0 MB

Release files / disk_kg-1.2.0.tar.gz

Download URL disk_kg-1.2.0.tar.gz
Size 1.9 MB
Tags Source
SHA-256 checksum
How to use checksums
b340d0c6602f02a61c62d92435c628cea7c345f17d300c3c31892b61585e4971
BLAKE2b-256 checksum
How to use checksums
2fe61ce7b17bab47281bee87080457b9ea5258ebb1894f444dd9c7b9211d6ce5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.11 {"installer":{"name":"uv","version":"0.9.11"},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / disk_kg-1.2.0-py3-none-any.whl

Download URL disk_kg-1.2.0-py3-none-any.whl
Size 60.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f8d0a43115ec91d0c065fba75782be2edffcd9f9536b3347756061b24971d82a
BLAKE2b-256 checksum
How to use checksums
b12688fd915fd00ae04094ec8c1c7fc483690fb3994f6a4a19cd92bc1f445856
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.11 {"installer":{"name":"uv","version":"0.9.11"},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for disk-kg 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page