DISK
Incremental conStruction of Domain Knowledge graph.
Overview
DISK is a comprehensive toolkit for extracting knowledge from PDF documents and building domain knowledge graphs through text distillation, entity/relation extraction, and semantic merging. The system provides a modular pipeline that transforms unstructured PDF documents into structured knowledge representations.
Core Capabilities
- Document Distillation: Extract and validate text blocks, tables, and images from PDF documents
- Entity Extraction: Identify and extract domain entities with semantic embeddings
- Relation Extraction: Discover relationships between entities with contextual understanding
- Knowledge Graph Construction: Build and manage knowledge graphs with incremental updates
- Semantic Merging: Intelligently merge similar entities and relations using cosine similarity
Architecture
System Architecture
graph TB
subgraph "Input Layer"
PDF[PDF Document]
end
subgraph "Distillation Layer"
Distiller[PDF Distiller]
TextBlocks[Validated Text Blocks]
end
subgraph "Extraction Layer"
EntExtractor[Entity Extractor]
RelExtractor[Relation Extractor]
UnifiedExtractor[Unified Extractor]
Entities[(Entities + Embeddings)]
Relations[(Relations + Embeddings)]
end
subgraph "Processing Layer"
Merger[Semantic Merger]
Manager[KG Manager]
end
subgraph "Output Layer"
KG[Knowledge Graph]
Logs[Logs & Results]
end
subgraph "Configuration"
Config[LLM Config]
Embed[Embeddings Model]
end
PDF --> Distiller
Distiller --> TextBlocks
TextBlocks --> EntExtractor
TextBlocks --> RelExtractor
TextBlocks --> UnifiedExtractor
EntExtractor --> Entities
RelExtractor --> Relations
UnifiedExtractor --> Entities
UnifiedExtractor --> Relations
Entities --> Merger
Relations --> Merger
Merger --> Manager
Manager --> KG
Manager --> Logs
Config --> EntExtractor
Config --> RelExtractor
Config --> UnifiedExtractor
Embed --> EntExtractor
Embed --> RelExtractor
Embed --> UnifiedExtractor
Embed --> Merger
style PDF fill:#e1f5fe
style KG fill:#c8e6c9
style Distiller fill:#fff3e0
style Merger fill:#f3e5f5
style Manager fill:#e8f5e9
Module Structure
graph LR
subgraph DISK
DiskMain[disk.py<br/>Main Entry Point]
subgraph Core
Distiller[distiller/<br/>PDF Distillation]
Extractor[extractor/<br/>Information Extraction]
MergerMod[merger/<br/>Knowledge Merging]
ManagerMod[manager/<br/>KG Management]
end
subgraph Support
Models[models/<br/>Data Models]
Utils[utils/<br/>Utilities]
ConfigMod[config/<br/>Configuration]
end
end
DiskMain --> Distiller
DiskMain --> Extractor
DiskMain --> MergerMod
DiskMain --> ManagerMod
Extractor --> Models
MergerMod --> Models
ManagerMod --> Models
Distiller --> Utils
Extractor --> Utils
ManagerMod --> Utils
DiskMain --> ConfigMod
style DiskMain fill:#1976d2,color:#fff
style Distiller fill:#ffa726
style Extractor fill:#42a5f5
style MergerMod fill:#ab47bc
style ManagerMod fill:#66bb6a
Data Flow
sequenceDiagram
participant User
participant DISK
participant Distiller
participant Extractor
participant Merger
participant Manager
participant KG
User->>DISK: build_knowledge_graph(pdf_path)
DISK->>Distiller: extract_text_blocks(pdf)
Distiller-->>DISK: validated_text_blocks
loop For each text block
DISK->>Extractor: extract_entities(text)
Extractor-->>DISK: entities + embeddings
DISK->>Extractor: extract_relations(text)
Extractor-->>DISK: relations + embeddings
DISK->>Merger: merge(new, existing)
Merger-->>DISK: merged entities/relations
end
DISK->>Manager: add_entities(entities)
DISK->>Manager: add_relations(relations)
Manager->>KG: update_knowledge_graph
DISK-->>User: Knowledge Graph
Modules
Distillation Module (distiller/)
- pdf_distiller
- extract paragraphs with intelligent validation
- extract tables(to be improved)
- extract imgs(to be improved)
- filter out low-quality text blocks (references, incomplete sentences)
Extraction Module (extractor/)
-
entities_extractor
- extract domain entities with labels and descriptions
- generate semantic embeddings for each entity
-
relations_extractor
- extract relationships between entities
- generate semantic embeddings for each relation
-
extractor (unified)
- extract both entities and relations in a single pass
- optimized for incremental processing
Processing Modules
- extract entities
- extract relationships
- semantic merging (merger/)
- merge similar entities using cosine similarity
- update relations after entity merging
- configurable threshold (default: 0.8)
- construct knowledge graph (manager/)
- incremental knowledge graph construction
- deduplication of entities and relations
Config
env
# use uv to manage the environment
uv venv
uv sync
LLM Configuration
-
Copy the example configuration file:
cp config.example.toml config.toml
-
Edit
config.tomlto set your API keys and preferences:
[disk]
llm = "openai" # Choose provider: openai, qwen, ollama, etc.
[disk.embeddings]
model = "text-embedding-3-small"
api_key = "ai-..."
api_url = "https://api.openai.com/v1"
[model.openai]
api_url = "https://api.openai.com/v1"
api_key = "ai-..."
model = "gpt-4o"
[model.other]
api_url = "https://api.otherprovider.com/v1"
api_key = "sk-..."
model = "gpt-4o"
- Supported providers:
- OpenAI (default)
- Qwen (DashScope)
- Kimi (Moonshot)
- Ollama (Local)
- All other providers that support OpenAI-compatible APIs
You can switch providers by changing the llm field in [disk] or using the runtime switch() function.
Contrast
merge
-
itext2kg
[INFO] Wohoo! Entity was matched --- [poor deep semantic understanding in traditional ie models:Limitation] --merged--> [cosine similarity ignores deep semantic differences:Limitation]
Metadata
Release files for disk-kg 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| disk_kg-1.2.0.tar.gz | 1.9 MB | Without provenance | Details for disk_kg-1.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| disk_kg-1.2.0-py3-none-any.whl | Python 3 | none | any | Details for disk_kg-1.2.0-py3-none-any.whl |
Total release size: 2.0 MB
Release files / disk_kg-1.2.0.tar.gz
| Download URL | disk_kg-1.2.0.tar.gz |
|---|---|
| Size | 1.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b340d0c6602f02a61c62d92435c628cea7c345f17d300c3c31892b61585e4971
|
|
BLAKE2b-256 checksum How to use checksums |
2fe61ce7b17bab47281bee87080457b9ea5258ebb1894f444dd9c7b9211d6ce5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.11 {"installer":{"name":"uv","version":"0.9.11"},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / disk_kg-1.2.0-py3-none-any.whl
| Download URL | disk_kg-1.2.0-py3-none-any.whl |
|---|---|
| Size | 60.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f8d0a43115ec91d0c065fba75782be2edffcd9f9536b3347756061b24971d82a
|
|
BLAKE2b-256 checksum How to use checksums |
b12688fd915fd00ae04094ec8c1c7fc483690fb3994f6a4a19cd92bc1f445856
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.11 {"installer":{"name":"uv","version":"0.9.11"},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for disk-kg 1.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.