hydrusvideodeduplicator 0.2.2

Video deduplicator utility for Hydrus Network

Hydrus Video Deduplicator

Hydrus Video Deduplicator detects similar video files and marks them as potential duplicates through the Hydrus API

---

How It Works:

The deduplicator works by comparing videos by computing a perceptual hash.

A perceptual hash is a way to characterize videos in small chunks.

The perceptual hashes are stored in a database file in the running directory to avoid computing them every time.

Once all perceptual hashes for all the videos in your database are computed, they are compared against each other to detect if they're similar. If they are similar, they will be marked as potential duplicates in Hydrus.

The accuracy is extremely good because of vpdq. You can adjust the threshold of similarity using --threshold. The default is 75%.

For more information check out the wiki and the FAQ

---

Installation:

Dependencies:

• Python >=3.10
• FFmpeg

python3 -m pip install hydrusvideodeduplicator

---

Usage:

python3 -m hydrusvideodeduplicator --api-key="<your key>"

---

TODO:

• Option to rollback and remove potential duplicates
• OR predicates for --query
• Parallelize hashing and duplicate search
• Automatically generate access key with Hydrus API
• Docker container
• Upload Docker container to Docker Hub (GitHub Action)
• Pure Python port of vpdq
• Windows compatibility without WSL or Docker

Please create an issue on Github if you have any problems or questions! Pull requests are also welcome.

---

Credits:

Hydrus Network by dev

Hydrus API Library by Cryzed

pdq by Meta

vpdq by Meta, ported to Python by me.

Big Buck Bunny clips by Blender Foundation (CC BY 3.0)