mirror-dedupe 1.2.0

Pool-based content-addressable APT repository mirror sync with global deduplication via hardlinks.

Features

• Pool-based content-addressable storage: every unique file is stored once by SHA-256 hash
• Hardlink deduplication across all mirrored repositories
• HTTPS-based sync with automatic upstream failover
• Auto-discovery of distributions, architectures, and components via HTTP probing
• Snapshot/restore system for safe repository migrations
• Glob pattern support for distributions (e.g. noble*, */mongodb-org/7.0)
• GPG key URL tracking in per-repo configuration
• Stale pool orphan detection and cleanup (--sweep-pool)

Installation

Option 1: Debian/Ubuntu Package (Recommended)

Install from the MungerWare apt repository:

curl -fsSL https://apt.mungerware.com/key.asc \
  | sudo gpg --dearmor -o /usr/share/keyrings/mungerware-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/mungerware-archive-keyring.gpg] https://apt.mungerware.com noble main" \
  | sudo tee /etc/apt/sources.list.d/mungerware.list

sudo apt update && sudo apt install mirror-dedupe

This includes systemd integration, man pages, nginx configuration, and proper package management.

Alternatively, download the latest .deb package from GitHub Releases:

curl -fsSL https://api.github.com/repos/munger/mirror-dedupe/releases/latest \
  | grep browser_download_url | grep all.deb | head -1 | cut -d'"' -f4 \
  | xargs wget
sudo dpkg -i mirror-dedupe_*.deb

Option 2: PyPI (All Linux Distributions)

pip install mirror-dedupe

Then install systemd files manually:

sudo ./install.sh --pip

Option 3: From Source

git clone https://github.com/munger/mirror-dedupe.git
cd mirror-dedupe
sudo ./install.sh

Configuration

Configuration files are located in /etc/mirror-dedupe/:

• mirror-dedupe.conf — Global settings (mirror_root, concurrency, etc.)
• repos-available/ — Available repository configurations
• repos-enabled/ — Enabled repositories (symlinks to repos-available)

mirror_root is the single root for all mirror data. Repositories are stored under <mirror_root>/repos and the deduplication pool under <mirror_root>/pool. Both always share the same logical volume, which is required for hardlinks.

Adding a Repository

Use --scan to auto-generate configuration for a repository:

mirror-dedupe --scan --name grafana --out /etc/mirror-dedupe/repos-available https://apt.grafana.com

See config/repos-available/README.md for the full scan reference and ready-made commands for all packaged example repositories.

Then test and enable it:

mirror-dedupe --test grafana
mirror-dedupe --activate grafana

Advanced: Alternative config directory

Override the default config directory with --config-dir:

mirror-dedupe --config-dir /tmp/mirror --test grafana

Pre-configured Repositories

The package includes pre-configured repository definitions in config/repos-available/.

Usage

# Sync all enabled mirrors
mirror-dedupe --sync

# Sync a specific mirror
mirror-dedupe --sync --mirror ubuntu

# Scan a repository and generate config (single release)
mirror-dedupe --scan --name ubuntu --release noble --out /etc/mirror-dedupe/repos-available http://archive.ubuntu.com/ubuntu

# Scan with all discovered architectures and distributions
mirror-dedupe --scan --name postgresql --no-filter --out /etc/mirror-dedupe/repos-available http://apt.postgresql.org/pub/repos/apt

# Scan without JSON snapshot
mirror-dedupe --scan --name grafana --out /etc/mirror-dedupe/repos-available https://apt.grafana.com

# Pool orphan sweep
mirror-dedupe --sweep-pool

# Create a hardlink snapshot of all repos
mirror-dedupe --snapshot

Systemd Integration

If installed via Debian package, systemd is already configured. Otherwise:

sudo systemctl enable --now mirror-dedupe.timer
sudo systemctl status mirror-dedupe.timer

View logs:

journalctl -u mirror-dedupe.service

Nginx Configuration

See nginx/mirror.conf for an example nginx configuration.

License

MIT License — see LICENSE file for details.

Author

Tim Hosking tim@mungerware.com

https://github.com/munger/mirror-dedupe