aidkits 0.2.0

A tool buildg AI docs assistans. Parsing Git repositories and markdown files. Work with vector db

AI (Documentation) Kit

AIDkit is a Python tools collection designed to help create AI documentation assistants. It includes utilities for parsing Git repositories or local directories containing Markdown files, extracting content, organizing it into structured JSON, and saving the output. These tools are particularly useful for processing documentation repositories for AI-powered assistance. aidkit

---

Features

• Clone remote Git repositories or work with local directories
• Extract and split Markdown content into chunks based on headers (#, ##, ###)
• Save parsed Markdown data in JSON format
• Index and retrieve chunks using OpenSearch
• Split large JSON files into multiple smaller files based on a grouping field
• Modular and extensible design
• Available as command-line utilities: mdcrawler and jsonsplitter

---

Installation

Make sure you are using Python 3.9 or newer and have pip installed.

pip install aidkit

Once installed, the command-line utility giga_crawler will be available.

---

Usage

The tool works by taking a Git repository URL (or a local directory) and outputs a JSON file containing structured data extracted from Markdown files.

Command-line Arguments

Argument	Type	Description
--uri	String	URL of a remote Git repository to be cloned, or path to a local directory with Markdown files.
--output_path	String	Path to save the output JSON file. (Default: output.json)
--directory	String	Optional. Path with docs source if used remote repo with clone.
--multy_process	Boolean	Spawn multiple processes to speed up the process. (Default: False)

Examples

1. Clone a remote repository and parse Markdown files:

aidkit parse --uri https://github.com/example/repo.git

or

from aidkits import MarkdownCrawler

MarkdownCrawler("repo/path").work()

This will:

• Clone the repository into a temporary directory
• Parse all Markdown files in the repository
• Save the JSON output to output.json

2. Parse a local directory and save output to a custom JSON file:

aidkit parse --uri ./local_directory --output_path result.json

This will:

• Use the specified local directory ./local_directory
• Parse all Markdown files in the directory
• Save the JSON output to result.json

JSON Output Format

The JSON file saves structured data with the following format:

[
  {
    "title": "example.md",
    "chunks": [
      {
        "title": "Header 1",
        "content": "Content under Header 1",
        "length": 120,
        "chunk_num": 1,
        "chunk_amount": 2
      },
      {
        "title": "Header 2",
        "content": "Content under Header 2",
        "length": 240,
        "chunk_num": 2,
        "chunk_amount": 2
      }
    ]
  }
]

---

Advanced Usage

OpenSearchRetriever

The OpenSearchRetriever class provides advanced vector search capabilities using OpenSearch. It allows you to:

• Search for documents based on semantic similarity
• Create and manage collections in OpenSearch
• Upload documents and libraries to OpenSearch

Example Usage

from opensearchpy import OpenSearch
from sentence_transformers import SentenceTransformer
from aidkits.storage.opensearch_retriever import OpenSearchRetriever
from aidkits.models import LibrarySource

# Initialize the OpenSearch client
client = OpenSearch(
    hosts=[{"host": "localhost", "port": 9200}],
    http_auth=("admin", "admin"),
    use_ssl=False,
    verify_certs=False,
)

# Initialize the encoder
encoder = SentenceTransformer("all-MiniLM-L6-v2")

# Create the retriever
retriever = OpenSearchRetriever(client, encoder)

# Create a collection
retriever.create_collection("documentation")

# Upload a library
library = LibrarySource.from_json("path/to/library.json")
retriever.upload_library(library)

# Search for documents
results = retriever.search(
    question="How do I use the API?",
    collection_name="documentation",
    top_k=5
)

# Print the results
for result in results:
    print(result.markdown)

DocumentationTool

The DocumentationTool class provides a high-level interface for answering questions using documentation stored in OpenSearch. It uses the OpenSearchRetriever to find relevant documentation and a language model to generate answers.

Example Usage

from langchain_core.language_models import ChatOpenAI
from aidkits.documentation_tool import DocumentationTool
from aidkits.storage.opensearch_retriever import OpenSearchRetriever

# Initialize the language model
llm = ChatOpenAI(model="gpt-3.5-turbo")

# Initialize the retriever (as shown above)
# ...

# Create the documentation tool
doc_tool = DocumentationTool(
    llm=llm,
    retriever=retriever,
    collection_name="documentation",
    top_k=5
)

# Answer a question
answer = doc_tool.invoke({"question": "How do I use the API?"})
print(answer)

JsonSplitter

The JsonSplitter class provides functionality for splitting a large JSON file into multiple smaller files based on a grouping field. It can be used to organize JSON data by a common field, making it easier to work with large datasets.

Example Usage

from aidkits.json_splitter import JsonSplitter

# Create a JsonSplitter instance
splitter = JsonSplitter(output_dir="output_directory")

# Split a JSON file
grouped_data = splitter.split_json_file(
    input_file="large_file.json",
    group_by_field="title",
    encoding="utf-8"
)

# Print information about the created files
print(f"Total files created: {len(grouped_data)}")

# You can also split JSON data directly
data = [
    {"title": "Document 1", "content": "Content 1"},
    {"title": "Document 2", "content": "Content 2"},
    {"title": "Document 1", "content": "More content for Document 1"}
]

grouped_data = splitter.split_json_data(
    data=data,
    group_by_field="title"
)

# This will create two files:
# - output_directory/Document_1.json (containing 2 items)
# - output_directory/Document_2.json (containing 1 item)

---

Contributing

We welcome contributions! Here's how you can help:

1. Fork the repository and create a branch for your feature or bug fix.
2. Write clear, concise code and include comments where necessary.
3. Submit a pull request with a detailed explanation of your changes.

---

License

This project is licensed under the MIT License. See the LICENSE file for more details.