Skip to main content

Chunktopus

A powerful document processing and chunking API by Unsiloed AI, featuring multithreaded OCR and intelligent document chunking using OpenAI's Vision models.

Installation

pip install chunktopus

Features

  • Process PDF, DOCX, and PPTX documents
  • Multiple chunking strategies:
    • Fixed size chunks
    • Semantic chunks (using OpenAI)
    • Page-based chunks
    • Paragraph-based chunks
    • Heading-based chunks
  • Fast parallel processing
  • RESTful API with FastAPI
  • Simple JavaScript-like API

Configuration

OpenAI API Key

Important: This package requires your own OpenAI API key. You must provide your API key in one of the following ways:

  1. Set it as an environment variable:
export OPENAI_API_KEY="your-api-key"
  1. Create a .env file in your project directory:
OPENAI_API_KEY=your-api-key
  1. Set it programmatically:
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
  1. Provide it directly in the API call (recommended for the simple API)

Usage

import chunktopus

# Process a document with your own OpenAI API key
result = chunktopus.process_sync({
    "filePath": "https://example.com/document.pdf",
    "credentials": {
        "apiKey": "your-openai-api-key"
    },
    # Optional parameters
    "strategy": "semantic",  # or "fixed", "page", "paragraph", "heading"
    "chunkSize": 1000,
    "overlap": 100
})

# Print the results
print(f"Number of chunks: {result['total_chunks']}")
for i, chunk in enumerate(result['chunks']):
    print(f"Chunk {i+1}: {chunk['text'][:100]}...")

# Async version is also available
import asyncio

async def main():
    result = await chunktopus.process({
        "filePath": "path/to/document.pdf",
        "credentials": {
            "apiKey": "your-openai-api-key"
        }
    })
    print(f"Number of chunks: {result['total_chunks']}")

asyncio.run(main())

Using as a Python Package (Legacy)

from chunktopus import chunktopus

# Process a document with your own OpenAI API key
result = chunktopus(
    file_path="path/to/document.pdf",
    credentials={"api_key": "your-openai-api-key"}
)

# Or process a document from a URL
result = chunktopus(
    file_path="https://example.com/document.pdf",
    credentials={"api_key": "your-openai-api-key"}
)

# Or process raw text
result = chunktopus(
    text="Your text content to chunk semantically...",
    credentials={"api_key": "your-openai-api-key"}
)

# Print the results
for i, chunk in enumerate(result):
    print(f"Chunk {i+1}: {chunk['text'][:100]}...")

Using from Command Line

After installation, you can use the chunktopus command:

# Process a file with your OpenAI API key
chunktopus path/to/document.pdf --api-key "your-openai-api-key"

# Output to a file
chunktopus path/to/document.pdf --api-key "your-openai-api-key" --output results.json

# Format as text instead of JSON
chunktopus path/to/document.pdf --api-key "your-openai-api-key" --format text

Running as a REST API

from chunktopus.server import run_server

# Start the server on port 8000
run_server(port=8000)

Using the document processing functions directly

from chunktopus.services import process_document_chunking
from chunktopus.utils.chunking import ChunkingStrategy

# Set your OpenAI API key
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"

# Process a document with semantic chunking
result = process_document_chunking(
    file_path="path/to/document.pdf",
    file_type="pdf",
    strategy=ChunkingStrategy.SEMANTIC,
)

# Print the results
print(f"Number of chunks: {result['total_chunks']}")
for i, chunk in enumerate(result['chunks']):
    print(f"Chunk {i+1}: {chunk['text'][:100]}...")

API Endpoints

/chunking

POST endpoint that processes a document and returns chunks.

Parameters:

  • document_file: The document file (PDF, DOCX, PPTX)
  • strategy: Chunking strategy (semantic, fixed, page, paragraph, heading)
  • chunk_size: Size of chunks for fixed strategy (default: 1000)
  • overlap: Overlap size for fixed strategy (default: 100)

License

MIT License

Metadata

Release files for chunktopus 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for chunktopus 0.1.2
File Size Uploaded Provenance Actions
chunktopus-0.1.2.tar.gz 13.9 kB Without provenance Details for chunktopus-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for chunktopus 0.1.2
File Interpreter ABI Platform Actions
chunktopus-0.1.2-py3-none-any.whl Python 3 none any Details for chunktopus-0.1.2-py3-none-any.whl

Total release size: 28.1 kB

Release files / chunktopus-0.1.2.tar.gz

Download URL chunktopus-0.1.2.tar.gz
Size 13.9 kB
Tags Source
SHA-256 checksum
How to use checksums
195193754dec9a9a7b8f77fe31f25f9cabe125ffafcece6a3e8ee509702e544a
BLAKE2b-256 checksum
How to use checksums
8156e40c5920b01ba1913b648af7c5da7a8bf25f88bb0a94f0a6e500a92fb619
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.6

Release files / chunktopus-0.1.2-py3-none-any.whl

Download URL chunktopus-0.1.2-py3-none-any.whl
Size 14.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
468bf12941e47e0018aa8d4229cfa2336f48f29b3e6766f7da943620bd64f269
BLAKE2b-256 checksum
How to use checksums
8d5f0c0af141654cd9b579ce6da2bde9e5a1ec4602f39034d8e70b93c949c73e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.9.6

Security for chunktopus 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page