athenaeum-kb 0.1.0

Tools for intelligent interaction with knowledge bases

Athenaeum

Project Scope and Goals

The goal of this project is to build a Python library that equips AI agents with a robust set of tools for intelligent interaction with knowledge bases. The library focuses on document ingestion, semantic search, and structured access to content, making it suitable for agent-based systems, RAG pipelines, and automation workflows.

Once the module is fully tested and validated, the intended outcome is to package and publish it as a reusable Python library on PyPI.

Tools

load_doc

Load a document into the knowledge base, automatically extracting content, metadata, and embeddings.

load_doc(path: str) -> str

Parameters:

• path: Path to the document file

Supported formats: PDF, PPTX, DOCX, XLSX, JSON, CSV, TXT, MD, HTML, XML, RTF, EPUB

Returns: A document identifier (doc_id) that can be used for subsequent operations.

list_docs

List all documents currently stored in the knowledge base.

list_docs() -> list[DocSearchHit]

Returns: A list of documents, including metadata (id, name, format, etc.) and, when available, a table of contents.

search_docs

Search across all documents in the knowledge base.

search_docs(
    query: str,
    top_k: int = 10,
    scope: Literal["names", "contents"] = "contents",
    strategy: Literal["hybrid", "bm25", "vector"] = "hybrid"
) -> list[DocSearchHit]

Parameters:

• query: Search query text
• top_k: Maximum number of results (default: 10)
• scope: Where to search
  • "contents": Search within document contents (default)
  • "names": Search only document names
• strategy: Search strategy (only applies when scope is "contents")
  • "hybrid": Combines vector and BM25 search (default)
  • "bm25": Keyword-based search only
  • "vector": Semantic similarity search only

Returns: A ranked list of documents matching the query.

search_doc_contents

Search within a specific document.

search_doc_contents(
    doc_id: str,
    query: str,
    top_k: int = 5,
    strategy: Literal["hybrid", "bm25", "vector"] = "hybrid"
) -> list[ContentSearchHit]

Parameters:

• doc_id: Document identifier
• query: Search query text
• top_k: Maximum number of results (default: 5)
• strategy: Search strategy
  • "hybrid": Combines vector and BM25 search (default)
  • "bm25": Keyword-based search only
  • "vector": Semantic similarity search only

Returns: A list of matching content fragments with relevance scores.

read_doc

Read a specific range of lines from a document.

read_doc(
    doc_id: str,
    start_line: int = 1,
    end_line: int = 100
) -> Excerpt

Parameters:

• doc_id: Document identifier
• start_line: Starting line number (1-indexed, default: 1)
• end_line: Ending line number (1-indexed, inclusive, default: 100)

Returns: A document excerpt containing the requested lines.

Search Strategies

• Hybrid Search (Default): Combines vector similarity search with BM25 keyword search using Reciprocal Rank Fusion (RRF).
• Vector Search: Uses embedding models for semantic similarity search.
• BM25 Search: Traditional keyword-based search using the BM25 algorithm.

Document Ingestion Workflow

The ingestion pipeline is triggered by the load_doc(path) function and follows these steps:

1. Validation
   • Verify that the file exists.
   • Confirm that the file format is supported.
2. Content Extraction
   • Run an OCR or parsing pipeline to convert the raw file into Markdown.
   • If the document contains images:
     • Replace them with placeholders in the Markdown.
     • Store image references for later retrieval.
3. Pre-processing
   • Generate document metadata (e.g., id, name, format).
   • Build a table of contents (TOC) from Markdown headings when possible.
4. Indexing
   • Generate vector embeddings using the configured embedding model.
   • Store embeddings in the vector database for semantic retrieval.

Data Models

A preliminary set of domain models is defined in models.py. These classes are exploratory and serve as a conceptual starting point for the project.

They are not considered final and may be refactored, renamed, or removed as the overall architecture of Athenaeum evolves and solidifies.