Skip to main content

FileSense

A local CLI-based file analysis and deduplication engine built in Python 3.x.


Requirements

  • Python 3.12+
  • Windows / Linux / macOS

Installation

pip install -r requirements.txt

Running Unit Tests

# Run all 55 tests
python -X utf8 -m pytest tests/ -v

# Run a specific test file
python -X utf8 -m pytest tests/test_scanner.py -v
python -X utf8 -m pytest tests/test_hasher.py -v
python -X utf8 -m pytest tests/test_duplicate.py -v
python -X utf8 -m pytest tests/test_storage.py -v

CLI Commands

Note: Once installed, you can run filesense from anywhere on your computer.

Understanding the Command Syntax

  • <path>: Required argument. The directory you want to scan (e.g., ., ./Downloads, "C:\My Files").
  • [...]: Optional flags. You do not type the brackets. These modify how the command runs.

1. scan — Discover and index files

Scans a directory, finds duplicates, and saves everything to a local SQLite cache.

filesense scan <path> [--workers N] [--follow-symlinks] [--verbose]

Options:

  • --workers N (or -w N): Set the number of concurrent hashing threads (default is 4). Use 8 for fast SSDs, or 1 to disable concurrency.
  • --follow-symlinks: By default, symbolic links are ignored. Add this flag to follow them.
  • --verbose (or -v): Prints detailed DEBUG logs to the terminal (e.g., showing exactly which files are being hashed).

Example:

filesense scan "C:\Users\Name\Downloads" --workers 8 --verbose

2. duplicates — View duplicate groups

Displays exactly which files are duplicates of each other and how much space they waste.

filesense duplicates <path> [--workers N]

3. analyze — Storage breakdown

Shows how your storage is being used, grouped by file category (Images, Videos, Code) and extension (.mp4, .pdf).

filesense analyze <path>

4. largest — Top space consumers

Lists the biggest individual files in the directory.

filesense largest <path> [--top N]

Options:

  • --top N: How many files to show (default is 10).

Example:

filesense largest . --top 20

5. report — Full combined output

Runs scan, duplicate detection, storage analysis, and largest files all in one massive report.

filesense report <path> [--workers N]

6. benchmark — Performance comparison

Compares the speed of a naive full-hash approach vs FileSense's multi-stage pipeline.

filesense benchmark <path> [--workers 1,2,4,8]

Options:

  • --workers <comma-list>: The thread configurations you want to test. Default is 1,2,4.

Example:

filesense benchmark ./data --workers 1,4,8,16

7. clean — Safely remove duplicates

Previews or performs actual deletion of duplicate files to recover space.

# Preview what WOULD be deleted (SAFE - Default behavior)
filesense clean <path> --dry-run

# Perform ACTUAL deletion (Requires typing 'y' at a prompt)
filesense clean <path> --no-dry-run

8. index status — Check database stats

Shows how many files are currently cached in your local ~/.filesense/filesense.db database.

filesense index status

Project Structure

FileSense/
├── filesense/
│   ├── metadata.py        # FileMetadata dataclass + extension classifier
│   ├── scanner.py         # FileScanner — recursive directory traversal
│   ├── hasher.py          # HashManager, PartialHashStrategy, FullHashStrategy
│   ├── index.py           # PersistentIndex — SQLite cache (~/.filesense/)
│   ├── duplicate.py       # DuplicateDetector — multi-stage pipeline
│   ├── storage.py         # StorageAnalyzer — usage breakdowns
│   ├── report.py          # ReportGenerator — Rich terminal output
│   ├── benchmark.py       # BenchmarkEngine — naive vs multi-stage
│   ├── logging_config.py  # Rotating log handler (logs/filesense.log)
│   └── cli.py             # Typer CLI — all 8 commands
├── tests/
│   ├── test_scanner.py
│   ├── test_hasher.py
│   ├── test_duplicate.py
│   └── test_storage.py
├── docs/
│   └── INTERVIEW_GUIDE.md # Deep-dive technical reference
├── logs/
│   └── filesense.log
├── main.py
├── requirements.txt
└── README.md

Pipeline

Files → Size Grouping → Partial Hash → Full SHA-256 → Duplicate Groups

Please do not try on C drives

Metadata

Release files for filesense-cli 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for filesense-cli 0.1.0
File Size Uploaded Provenance Actions
filesense_cli-0.1.0.tar.gz 35.5 kB With provenance Details for filesense_cli-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for filesense-cli 0.1.0
File Interpreter ABI Platform Actions
filesense_cli-0.1.0-py3-none-any.whl Python 3 none any Details for filesense_cli-0.1.0-py3-none-any.whl

Total release size: 59.2 kB

Release files / filesense_cli-0.1.0.tar.gz

Download URL filesense_cli-0.1.0.tar.gz
Size 35.5 kB
Tags Source
SHA-256 checksum
How to use checksums
e9f49c1f867cce881841ae4456350d5e8fd6a60fb4c6a19d3e09defbb6986f04
BLAKE2b-256 checksum
How to use checksums
852014ffa9b5f11dcfd06350a00a64b733e2a9eca63fb6d8c1f2cb181b0e91f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 17, 2026.

Transparency log

Release files / filesense_cli-0.1.0-py3-none-any.whl

Download URL filesense_cli-0.1.0-py3-none-any.whl
Size 23.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dc3ab5eab9e85e9ea5c4898a6fdcbcb54e9f21ee4207d9ce869c37ad1c63db8e
BLAKE2b-256 checksum
How to use checksums
0f9a34a259623ab57d7b6e560986b89de1365cc915d4dc47d62566790b35cdee
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 17, 2026.

Transparency log

Security for filesense-cli 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for filesense_cli-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page