whatsapp-dedup-guard
CLI tool that scans a WhatsApp bot inbox for duplicate message files and removes the noise.
Built because every WhatsApp message was being saved 2-3× — once per LLM responding in parallel (Gemini + GPT + local LLM). The inbox looked busy. It wasn't.
What it does
INBOX/whatsapp/
├── 20260527-041538-...-2ac234681b402aa8d891-mpnjwwxl.md ← ORIGINAL
├── 20260527-041539-...-2ac234681b402aa8d891-mpnjwxbu.md ← DUPLICATE
├── 20260527-041634-...-2a4e7a869b127b936cc6-mpnjy3l0.md ← ORIGINAL
└── 20260527-041635-...-2a4e7a869b127b936cc6-mpnjy4pg.md ← DUPLICATE
whatsapp_dedup_guard.py parses Message ID: from each file header, groups by ID, and identifies extras. Keeps the earliest copy. Flags or deletes the rest.
Quick start
# Scan for duplicates (read-only, no changes)
python3 whatsapp_dedup_guard.py scan /path/to/whatsapp/inbox/
# Full report with details
python3 whatsapp_dedup_guard.py report /path/to/whatsapp/inbox/
# Stats: total files, unique IDs, duplicate rate
python3 whatsapp_dedup_guard.py stats /path/to/whatsapp/inbox/
# Mark duplicates with Duplicate: true header (dry-run first)
python3 whatsapp_dedup_guard.py mark --dry-run /path/to/whatsapp/inbox/
python3 whatsapp_dedup_guard.py mark /path/to/whatsapp/inbox/
File structure
whatsapp-dedup-guard/
├── whatsapp_dedup_guard.py # CLI tool — stdlib only, no dependencies
└── README.md
Expected inbox file format
Files must have these headers in the first 15 lines:
# WhatsApp Task - DD/MM/YYYY, H:MM:SS
Status: new
Source: WhatsApp
Sender: 972XXXXXXXXX@c.us
Message ID: 2AC234681B402AA8D891
Agent Route: vision
Saved At: 2026-05-27T04:15:38.937Z
Exit codes
| Code | Meaning |
|---|---|
0 |
Clean — no duplicates found |
1 |
Duplicates detected |
2 |
Error (bad path, parse failure) |
Use in CI/cron: python3 whatsapp_dedup_guard.py scan $DIR || alert "duplicates in inbox"
Requirements
- Python 3.10+
- stdlib only (
argparse,os,sys,datetime) — no pip install needed
Why this exists
Built as part of RLASAF12's AI agent system. The WhatsApp bot was routing messages to multiple LLMs simultaneously, causing each message to be saved 2-3× under different filenames. At 15%+ duplicate rate across 152 files, the inbox was unreliable. This tool fixes that.
Built by Ben (nightly builder) · 2026-05-28
1 maintainer
Metadata
1 maintainer
Release files for whatsapp-dedup-guard 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| whatsapp_dedup_guard-0.1.0.tar.gz | 7.2 kB | With provenance | Details for whatsapp_dedup_guard-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| whatsapp_dedup_guard-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for whatsapp_dedup_guard-0.1.0-py3-none-any.whl |
Total release size: 15.0 kB
Release files / whatsapp_dedup_guard-0.1.0.tar.gz
| Download URL | whatsapp_dedup_guard-0.1.0.tar.gz |
|---|---|
| Size | 7.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
116d4d8f178b1fb17e53efa925d0da48c26f5b08a493b713a4db1218d8f05bae
|
|
BLAKE2b-256 checksum How to use checksums |
9d06b161e0d4054ac63d1ca39c69e0f9ca2a598ef20d40c333fbef22425c9cb5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 4, 2026.
Transparency logRelease files / whatsapp_dedup_guard-0.1.0-py3-none-any.whl
| Download URL | whatsapp_dedup_guard-0.1.0-py3-none-any.whl |
|---|---|
| Size | 7.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1b58ab85cf64043a7ca88ce4f12cf140757c47ab2b726a480433642a6cda5da1
|
|
BLAKE2b-256 checksum How to use checksums |
67dc8728e48d497bbad59fb018a53342b784ef20715e61c45abca6439cf65df4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 4, 2026.
Transparency logSecurity for whatsapp-dedup-guard 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for whatsapp_dedup_guard-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jun 4, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
