dedup-file-tools 0.1.0

Non-Redundant Media File Copy Tool: resumable, auditable, cross-platform, CLI-based.

🚀 dedup-file-tools: The Ultimate Toolkit for Safe, Fast, and Auditable File Management

Two powerful CLI tools, one mission: never lose a file, never copy a duplicate, and always know what happened.

---

What Does Each Tool Do?

• dedup-file-copy-fs: Effortlessly copy files from one place to another—guaranteeing that only a single, non-duplicate copy of each file ever lands in your destination. No more wasted space or accidental double copies!

• dedup-file-move-dupes: Hunts down duplicate files in any folder or storage pool and safely moves all detected duplicates to a special .dupes directory. It’s the easiest way to clean up your drives and reclaim space—exciting, right?

Both tools are designed for safety, auditability, and total peace of mind!

Why dedup-file-tools?

• No more accidental duplicates: Move or copy files between drives, folders, or backup pools with confidence—deduplication is automatic.
• Resumable & robust: Interrupt a job? No problem. All state is tracked in a job database. Resume anytime, anywhere.
• Audit everything: Every action, every file, every error—fully logged and queryable.
• Cross-platform: Works on Windows, Linux, and with both fixed and removable drives.
• Portable & future-proof: Thanks to UidPath, your job and checksum cache remain valid even if drive letters or mount points change.
• Agent/AI ready: YAML config, one-shot workflows, and full audit trails make it perfect for automation and integration.

---

Two Tools, Two Workflows — Pick Your Power!

---

🚦 dedup-file-copy-fs: The "Copy Without Duplicates" Workflow

Want to copy your files from one drive or folder to another, but never want to see a duplicate again? This is your magic wand!

Step-by-Step Tutorial

1. Initialize your copy job:

   dedup-file-copy-fs init --job-dir ./copyjob --job-name copyjob
   

2. Add your source(s):

   dedup-file-copy-fs add-source --job-dir ./copyjob --job-name copyjob --src ./source
   

3. Analyze and checksum:

   dedup-file-copy-fs analyze --job-dir ./copyjob --job-name copyjob --src ./source --dst ./dest
   dedup-file-copy-fs checksum --job-dir ./copyjob --job-name copyjob --table source_files
   dedup-file-copy-fs checksum --job-dir ./copyjob --job-name copyjob --table destination_files
   

4. Copy, with deduplication magic:

   dedup-file-copy-fs copy --job-dir ./copyjob --job-name copyjob --src ./source --dst ./dest
   

   Voila! Only unique files are copied. No duplicates, ever.
5. Verify and audit:

   dedup-file-copy-fs verify --job-dir ./copyjob --job-name copyjob --stage deep
   dedup-file-copy-fs log --job-dir ./copyjob
   

Want it all in one go? Just run:

dedup-file-copy-fs one-shot --job-dir ./copyjob --job-name copyjob --src ./source --dst ./dest

And... Voila! Your files are safely, uniquely copied.

See the User Guide for advanced tricks and YAML config magic.

---

🧹 dedup-file-move-dupes: The "Find & Sweep Duplicates" Workflow

Ready to reclaim space and banish duplicate files from your drives? This tool is your digital broom!

Step-by-Step Tutorial

1. Initialize your dedupe job:

   dedup-file-move-dupes init --job-dir ./myjob --job-name myjob
   

2. Scan your pool for duplicates:

   dedup-file-move-dupes analyze --job-dir ./myjob --job-name myjob --lookup-pool ./data
   dedup-file-move-dupes checksum --job-dir ./myjob --job-name myjob --table lookup_files
   

3. Move all detected duplicates to .dupes:

   dedup-file-move-dupes move-dupes --job-dir ./myjob --job-name myjob --dupes-folder ./dupes
   

   Voila! All your duplicate files are safely moved to .dupes for review or deletion.
4. Summarize and audit:

   dedup-file-move-dupes summary --job-dir ./myjob --job-name myjob
   dedup-file-move-dupes log --job-dir ./myjob
   

Want it all in one go? Just run:

dedup-file-move-dupes one-shot --job-dir ./myjob --job-name myjob --lookup-pool ./data --dupes-folder ./dupes

And... Voila! Your drives are clean and duplicate-free.

See the User Guide for advanced workflows and YAML config power.

---

Quick Start (Unified)

1. Install:

   pip install .
   # or for isolated CLI usage:
   pipx install .
   

2. Generate a config file (recommended):

   dedup-file-copy-fs generate-config
   # or
   dedup-file-move-dupes --config config.yaml
   

3. Run a one-shot workflow:

   dedup-file-move-dupes one-shot --job-dir ./myjob --job-name myjob --lookup-pool ./data --dupes-folder ./dupes
   dedup-file-copy-fs one-shot --job-dir ./copyjob --job-name copyjob --src ./source --dst ./dest
   

4. Check logs, verify, and audit:

   dedup-file-move-dupes summary --job-dir ./myjob --job-name myjob
   dedup-file-copy-fs verify --job-dir ./copyjob --job-name copyjob --stage deep
   

---

The Magic of UidPath

Both tools use a unique, system-independent path abstraction called UidPath. This means:

• Your job and checksum cache remain valid even if drive letters or mount points change.
• You can move drives between systems, plug into different USB ports, and never lose track of your files.
• Deduplication and verification are robust and portable. See UidPath documentation for details.

---

Want More?

• Full CLI docs:
  • dedup_file_tools_dupes_move CLI Reference
  • dedup_file_tools_fs_copy CLI Reference
• User guides & tutorials:
  • dedup_file_tools_dupes_move User Guide
  • dedup_file_tools_fs_copy User Guide
• Agent/AI integration:
  • External AI Tool Integration
• Requirements & design:
  • Requirements & Design

---

License

MIT License