Chunktopus
A powerful document processing and chunking API by Unsiloed AI, featuring multithreaded OCR and intelligent document chunking using OpenAI's Vision models.
Installation
pip install chunktopus
Features
- Process PDF, DOCX, and PPTX documents
- Multiple chunking strategies:
- Fixed size chunks
- Semantic chunks (using OpenAI)
- Page-based chunks
- Paragraph-based chunks
- Heading-based chunks
- Fast parallel processing
- RESTful API with FastAPI
- Simple JavaScript-like API
Configuration
OpenAI API Key
Important: This package requires your own OpenAI API key. You must provide your API key in one of the following ways:
- Set it as an environment variable:
export OPENAI_API_KEY="your-api-key"
- Create a .env file in your project directory:
OPENAI_API_KEY=your-api-key
- Set it programmatically:
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
- Provide it directly in the API call (recommended for the simple API)
Usage
Using the Simple API (Recommended)
import chunktopus
# Process a document with your own OpenAI API key
result = chunktopus.process_sync({
"filePath": "https://example.com/document.pdf",
"credentials": {
"apiKey": "your-openai-api-key"
},
# Optional parameters
"strategy": "semantic", # or "fixed", "page", "paragraph", "heading"
"chunkSize": 1000,
"overlap": 100
})
# Print the results
print(f"Number of chunks: {result['total_chunks']}")
for i, chunk in enumerate(result['chunks']):
print(f"Chunk {i+1}: {chunk['text'][:100]}...")
# Async version is also available
import asyncio
async def main():
result = await chunktopus.process({
"filePath": "path/to/document.pdf",
"credentials": {
"apiKey": "your-openai-api-key"
}
})
print(f"Number of chunks: {result['total_chunks']}")
asyncio.run(main())
Using as a Python Package (Legacy)
from chunktopus import chunktopus
# Process a document with your own OpenAI API key
result = chunktopus(
file_path="path/to/document.pdf",
credentials={"api_key": "your-openai-api-key"}
)
# Or process a document from a URL
result = chunktopus(
file_path="https://example.com/document.pdf",
credentials={"api_key": "your-openai-api-key"}
)
# Or process raw text
result = chunktopus(
text="Your text content to chunk semantically...",
credentials={"api_key": "your-openai-api-key"}
)
# Print the results
for i, chunk in enumerate(result):
print(f"Chunk {i+1}: {chunk['text'][:100]}...")
Using from Command Line
After installation, you can use the chunktopus command:
# Process a file with your OpenAI API key
chunktopus path/to/document.pdf --api-key "your-openai-api-key"
# Output to a file
chunktopus path/to/document.pdf --api-key "your-openai-api-key" --output results.json
# Format as text instead of JSON
chunktopus path/to/document.pdf --api-key "your-openai-api-key" --format text
Running as a REST API
from chunktopus.server import run_server
# Start the server on port 8000
run_server(port=8000)
Using the document processing functions directly
from chunktopus.services import process_document_chunking
from chunktopus.utils.chunking import ChunkingStrategy
# Set your OpenAI API key
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
# Process a document with semantic chunking
result = process_document_chunking(
file_path="path/to/document.pdf",
file_type="pdf",
strategy=ChunkingStrategy.SEMANTIC,
)
# Print the results
print(f"Number of chunks: {result['total_chunks']}")
for i, chunk in enumerate(result['chunks']):
print(f"Chunk {i+1}: {chunk['text'][:100]}...")
API Endpoints
/chunking
POST endpoint that processes a document and returns chunks.
Parameters:
document_file: The document file (PDF, DOCX, PPTX)strategy: Chunking strategy (semantic, fixed, page, paragraph, heading)chunk_size: Size of chunks for fixed strategy (default: 1000)overlap: Overlap size for fixed strategy (default: 100)
License
MIT License
Metadata
Release files for chunktopus 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| chunktopus-0.1.2.tar.gz | 13.9 kB | Without provenance | Details for chunktopus-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| chunktopus-0.1.2-py3-none-any.whl | Python 3 | none | any | Details for chunktopus-0.1.2-py3-none-any.whl |
Total release size: 28.1 kB
Release files / chunktopus-0.1.2.tar.gz
| Download URL | chunktopus-0.1.2.tar.gz |
|---|---|
| Size | 13.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
195193754dec9a9a7b8f77fe31f25f9cabe125ffafcece6a3e8ee509702e544a
|
|
BLAKE2b-256 checksum How to use checksums |
8156e40c5920b01ba1913b648af7c5da7a8bf25f88bb0a94f0a6e500a92fb619
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.6
|
Release files / chunktopus-0.1.2-py3-none-any.whl
| Download URL | chunktopus-0.1.2-py3-none-any.whl |
|---|---|
| Size | 14.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
468bf12941e47e0018aa8d4229cfa2336f48f29b3e6766f7da943620bd64f269
|
|
BLAKE2b-256 checksum How to use checksums |
8d5f0c0af141654cd9b579ce6da2bde9e5a1ec4602f39034d8e70b93c949c73e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.6
|
Security for chunktopus 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.