Duplifinder

The "Batteries Included" duplicate code detector. Detect and refactor duplicate Python classes, functions, and async defs—plus text and tokens across other languages—to keep your codebase lean and mean.

⚡ Quick Start (The "5-Minute Rule")

Prerequisites

• Python 3.12+
• pip (or uv/poetry)

Installation

pip install duplifinder

Usage Example

Get instant feedback on your current directory:

# Standard scan (AST + Token)
duplifinder .

# Watch mode for live feedback (Best for dev loop)
duplifinder . --watch --preview

# Scan with parallel processing and detailed audit logs
duplifinder src/ --parallel --audit --verbose

Pre-commit Hook

Add to your .pre-commit-config.yaml to block duplicates before they merge:

-   repo: https://github.com/dhruv13x/duplifinder
    rev: v11.0.0  # Use latest version
    hooks:
    -   id: duplifinder
        args: ["--fail", "--dup-threshold=0.05"]

✨ Features (The "Why")

Core Capabilities

• AST-Powered Detection: Precision finding for ClassDef, FunctionDef, and AsyncFunctionDef (Python). It sees through variable name changes.
• Multi-Language Support: Token and text-based similarity checks for Python, JavaScript, TypeScript, and Java.
• Smart Watch Mode: "Live" scanning that updates results instantly as you modify files.

Performance & Security

• Parallel Processing: Blazing fast scans using multi-threading or multi-processing (GIL-aware) with --parallel and --use-multiprocessing.
• Smart Caching: Skips unchanged files to dramatically speed up re-scans.
• Audit Logging: Enterprise-grade JSONL trails for file access and scan operations.

Developer Experience

• Automated Refactoring Suggestions: "God Level" advice—tells you how to fix the duplication (e.g., "Extract to shared utility").
• Rich Reporting: Beautiful console tables, JSON output for CI/CD, and formatted previews.

🛠️ Configuration (The "How")

Customize behavior via CLI flags or a .duplifinder.yaml file.

CLI Reference

Configuration File (.duplifinder.yaml)

You can also use .duplifinder.yaml. The CLI args override these settings.

Note: Environment variables are not currently supported for configuration to ensure reproducibility via code.

# Example .duplifinder.yaml
root: src
ignore: "tests,legacy"
exclude_patterns: "*/migrations/*"
token_mode: true
similarity_threshold: 0.85
audit: true
parallel: true

🏗️ Architecture

Duplifinder uses a Strategy pattern to dispatch scanners based on file type and mode.

Directory Tree

src/duplifinder/
├── application.py       # Workflow orchestration
├── cli.py               # Argument parsing
├── config.py            # Pydantic configuration & validation
├── finder.py            # Strategy Dispatcher
├── definition_finder.py # AST-based Logic (Python)
├── token_finder.py      # Token-based Similarity (Multi-lang)
├── text_finder.py       # Regex Pattern Matcher
├── refactoring.py       # Refactoring Suggestion Engine
├── processors.py        # File I/O & Parallel Processing
├── output.py            # Rich Console & JSON Renderers
├── utils.py             # File discovery & Audit logging
└── watcher.py           # Watchdog event handling

Data Flow

1. Discovery: utils.py recursively finds files, respecting .gitignore.
2. Dispatch: finder.py selects the right strategy (AST, Token, or Text) based on file extension.
3. Analysis: processors.py runs in parallel to extract definitions or tokens.
4. Comparison: Hashes or token vectors are compared to find duplicates.
5. Refactoring: refactoring.py analyzes results to generate actionable fixes.
6. Reporting: Results are streamed to Console (using Rich), JSON, or HTML.

🐞 Troubleshooting

Debug Mode: Run with --verbose to see detailed logs and performance metrics.

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for details on how to get started.

Dev Setup

1. Clone the repo.
2. Install dependencies: pip install -e ".[dev]"
3. Run tests: pytest
4. Linting: ruff check .

🗺️ Roadmap

See ROADMAP.md for the full vision.

• ✅ Foundation: AST Detection, Parallelism, Rich Output.
• ✅ Standard: Watch Mode, Refactoring Suggestions, Multi-language.
• 🚧 Ecosystem (Next): IDE Plugins, GitHub Action, Webhooks.
• 🔮 Vision: AI-Powered Refactoring, Cross-Repo Analysis.

Built with 💙 by Dhruv & the Open Source Community.