chunkops 0.1.0

CLI tool for detecting duplicate content in PDF documents

ChunkOps CLI

A command-line tool for detecting duplicate and near-duplicate content in PDF documents.

Installation

From PyPI (when published)

pip install chunkops

From Source (Development)

cd chunkops-cli
python3 -m pip install -e .

Note: Use python3 -m pip if pip command is not found in your PATH.

Important: Make sure your virtual environment is activated:

source /path/to/venv/bin/activate  # Activate your venv
chunkops --version                  # Should work now

Or use the full path:

/path/to/venv/bin/chunkops --version

Usage

Scan a folder of PDFs and generate a JSON report of duplicates:

chunkops scan /path/to/pdf/folder --output report.json

Options

• --output, -o: Path to output JSON file (default: duplicates_report.json)
• --threshold: Similarity threshold for near-duplicates (0.0-1.0, default: 0.90)
• --exact-threshold: Threshold for exact duplicates (default: 1.0)
• --verbose, -v: Enable verbose output

Example

chunkops scan ./documents --output duplicates.json --threshold 0.95

Output Format

The JSON report contains:

{
  "scan_date": "2024-01-15T10:30:00Z",
  "total_files": 10,
  "total_chunks": 245,
  "duplicates": [
    {
      "type": "EXACT_DUPLICATE",
      "file_a": "document1.pdf",
      "file_b": "document2.pdf",
      "similarity": 1.0,
      "chunk_a_preview": "...",
      "chunk_b_preview": "..."
    },
    {
      "type": "NEAR_DUPLICATE",
      "file_a": "document3.pdf",
      "file_b": "document4.pdf",
      "similarity": 0.95,
      "chunk_a_preview": "...",
      "chunk_b_preview": "..."
    }
  ],
  "summary": {
    "exact_duplicates": 2,
    "near_duplicates": 5,
    "unique_content": 238
  }
}

Limitations

• Local only: No cloud sync, runs entirely on your machine
• No UI: Command-line interface only
• PDF only: Currently supports PDF files only
• Single scan: Each scan is independent (no persistent history)

License

MIT

Support

Visit chunkops.ai for more information.