hydrusvideodeduplicator 0.2.0

Video deduplicator utility for Hydrus Network

Hydrus Video Deduplicator

Hydrus Video Deduplicator detects similar video files and marks them as potential duplicates through the Hydrus API

---

How It Works:

The deduplicator works by comparing videos by computing a perceptual hash.

A perceptual hash is a way to characterize videos in small chunks.

The perceptual hashes are stored in a database file in the running directory to avoid computing them every time.

Once all perceptual hashes for all the videos in your database are computed, they are compared against each other to detect if they're similar. If they are similar, they will be marked as potential duplicates in Hydrus.

The accuracy is extremely good because of vpdq. You can adjust the threshold of similarity using --threshold. The default is 75%.

For more information check out the wiki and the FAQ

---

Installation:

Windows requires WSL

Linux:

Install dependencies

Then install with pip:

pip install hydrusvideodeduplicator

---

Usage:

python3 -m hydrusvideodeduplicator --api-key="<your key>"

---

TODO:

• Option to rollback and remove potential duplicates
• OR predicates for --query
• Parallelize hashing and duplicate search
• Automatically generate access key with Hydrus API
• Docker container
• Upload Docker container to Docker Hub (GitHub Action)
• Pure Python port of vpdq
• Windows compatibility without WSL or Docker

Please create an issue on Github if you have any problems or questions! Pull requests also welcome on this or my VideoHash fork.

There is a lot to improve and cleanup and I'm more experienced in C than Python, so fix stuff please.

---

Credits:

Hydrus Network by dev

Hydrus API Library by Cryzed

vpdq by Meta

various other files from threatexchange by Meta