Code Duplicates CLI — Version 3
Multi-language code duplicate detector using semantic embeddings, Tree-Sitter parsing, parallelism, minimum size filtering, configurable exclusions, and HTML reports with syntax highlighting and JSON output.
✨ Features
- Multi-language support:
py, js, ts, tsx, jsx, go, rs (rust), c, cpp, java(extensible) - AST extraction with
tree-sitter-languages(functions, methods, classes) - Semantic embeddings with
SentenceTransformers(default:sentence-transformers/all-mpnet-base-v2) - Fast similarity search with
faiss(Inner Product + L2 normalization) - Parallel processing for snippet extraction and embedding generation
- Smart caching of embeddings by file content hash
- Configurable exclusions for folders and files (node_modules, .git, etc.)
- Minimum block size filtering (by lines)
- Rich HTML reports with Pygments syntax highlighting + JSON export
- Configurable similarity threshold and top-k results per snippet
- Verbose mode and dry-run capabilities
🚀 Installation
python -m venv .venv
source .venv/bin/activate # (Windows: .venv\Scripts\activate)
pip install -r requirements.txt
Note:
tree-sitter-languagesincludes pre-compiled grammars; no manual compilation required.
📖 Usage
Basic Usage
# Simple scan with defaults
python cli.py ./path/to/project
# Scan current directory
python cli.py .
Advanced Usage
python cli.py ./path/to/project \
--languages py,ts,tsx,js,go \
--threshold 0.85 \
--min-lines 5 \
--top-k 10 \
--report-html reports/duplicates.html \
--report-json reports/duplicates.json \
--model sentence-transformers/all-mpnet-base-v2 \
--cache-dir .cache_embeddings \
--exclusions-file .codeduplicates-ignore \
--verbose
Command Line Options
Options:
--languages TEXT Comma-separated list of file extensions (default: py,js,ts,tsx,jsx,go,rs,c,cpp,java)
--threshold FLOAT Similarity threshold (0.0-1.0, default: 0.85)
--min-lines INTEGER Minimum lines per code block (default: 3)
--top-k INTEGER Max similar snippets per block (default: 5)
--report-html PATH HTML report output path
--report-json PATH JSON report output path
--model TEXT Embedding model name (default: sentence-transformers/all-mpnet-base-v2)
--cache-dir PATH Directory for embedding cache
--exclusions-file PATH Custom exclusions file (default: .codeduplicates-ignore)
--verbose Enable verbose output
--dry-run Show what would be processed without running
--help Show this message and exit
🚫 Exclusions
The CLI includes default exclusions for common folders like node_modules, .git, __pycache__, etc.
Create a custom exclusions file:
# .codeduplicates-ignore
node_modules
.git
dist
build
*.min.js
my-generated-folder
test-data
venv
.venv
__pycache__
.pytest_cache
coverage
.coverage
Supports glob patterns (*, ?, []) and comments with #.
📊 Examples
Language-Specific Scans
# Scan TypeScript/JavaScript with high threshold
python cli.py . --languages ts,tsx,js --threshold 0.92 --report-html reports/js-duplicates.html
# Python and Go only, minimum 5 lines, top-3 results
python cli.py src --languages py,go --min-lines 5 --top-k 3 --report-json reports/py-go-duplicates.json
# Rust projects with custom exclusions
python cli.py . --languages rs --exclusions-file rust-exclusions.txt --verbose
Performance Optimization
# Large codebase with caching
python cli.py . --cache-dir .embeddings-cache --min-lines 10 --threshold 0.80
# Quick scan with dry-run
python cli.py . --dry-run --verbose
🏗️ Project Structure
code-duplicates-cli-v3/
├── cli.py # Main CLI entry point
├── core/
│ ├── config.py # Configuration constants
│ ├── utils.py # Utility functions
│ ├── embeddings.py # Embedding generation engine
│ ├── similarity.py # Similarity calculation engine
│ ├── report.py # HTML/JSON report generation
│ └── extractors/
│ ├── base_extractor.py # Base extractor interface
│ ├── parser_manager.py # Tree-sitter parser management
│ ├── code_extractor.py # AST-based code extraction
│ ├── regex_extractor.py # Regex-based extraction
│ └── simple_extractor.py # Simple text extraction
├── ARCHITECTURE.md # Detailed architecture documentation
├── MODEL_ANALYSIS.md # Embedding model analysis
└── requirements.txt # Python dependencies
🎯 Model Information
Current Default Model
- Model:
sentence-transformers/all-mpnet-base-v2 - Type: Semantic similarity model based on MPNet
- Dimensions: 768
- Strengths: Better semantic understanding, fewer false positives
- Use Case: General-purpose semantic similarity with good code understanding
Alternative Models
# Code-specific models
python cli.py . --model microsoft/codebert-base
python cli.py . --model microsoft/graphcodebert-base
# Faster alternatives
python cli.py . --model sentence-transformers/all-MiniLM-L6-v2
python cli.py . --model sentence-transformers/paraphrase-MiniLM-L6-v2
⚡ Performance Tips
For Large Repositories
- Use caching:
--cache-dir .embeddings-cacheto avoid recomputing embeddings - Increase minimum lines:
--min-lines 10to reduce noise from trivial blocks - Adjust threshold:
--threshold 0.80for broader matches,0.90+for strict matches - Limit top-k:
--top-k 3for faster processing
Memory Optimization
- Smaller models: Use
all-MiniLM-L6-v2for faster inference - Batch processing: The tool automatically handles large codebases efficiently
- Exclusions: Use comprehensive exclusion files to skip irrelevant directories
🐛 Troubleshooting
Common Issues
- High false positives: Increase
--thresholdto 0.90 or higher - Missing duplicates: Decrease
--thresholdto 0.70-0.80 - Too many results: Increase
--min-linesor use stricter exclusions - Slow performance: Use
--cache-dirand consider a smaller model
Debug Mode
# Verbose output for debugging
python cli.py . --verbose --dry-run
# Test specific file types
python cli.py test-project --languages py --verbose
📄 Output Formats
HTML Report
- Syntax highlighting with Pygments
- Side-by-side comparison of duplicate code blocks
- Similarity scores and file locations
- Interactive navigation between duplicates
JSON Report
- Structured data for programmatic processing
- Complete metadata including file paths, line numbers, similarity scores
- Easy integration with CI/CD pipelines and other tools
🤝 Contributing
- Fork the repository
- Create a feature branch
- Make your changes
- Add tests if applicable
- Submit a pull request
📝 License
This project is licensed under the MIT License.
Metadata
Release files for code-duplicates-cli 3.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| code_duplicates_cli-3.0.0.tar.gz | 18.5 kB | Without provenance | Details for code_duplicates_cli-3.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| code_duplicates_cli-3.0.0-py3-none-any.whl | Python 3 | none | any | Details for code_duplicates_cli-3.0.0-py3-none-any.whl |
Total release size: 36.8 kB
Release files / code_duplicates_cli-3.0.0.tar.gz
| Download URL | code_duplicates_cli-3.0.0.tar.gz |
|---|---|
| Size | 18.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
40f2dd933082b7e810a4cfa591c530e7be5927423d88ddd5a629cc3bae5ea65b
|
|
BLAKE2b-256 checksum How to use checksums |
b0e3a37fe79272268241544d6ecc52ad5d8ecaab2545612c0e6996573029679b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.9
|
Release files / code_duplicates_cli-3.0.0-py3-none-any.whl
| Download URL | code_duplicates_cli-3.0.0-py3-none-any.whl |
|---|---|
| Size | 18.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4d2a03f84f7bf93db1d9a7270dee23d65c47ef4a9b3262fc08c0c83964b25990
|
|
BLAKE2b-256 checksum How to use checksums |
7e25444c0bf302fc8fe5a59199752be334fd874c4a05148366aba009d19c4453
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.9
|
Security for code-duplicates-cli 3.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.