mirrorer 0.1.0

dist-mirror shard mirroring CLI

mirrorer

mirrorer is the shard-host CLI for dist-mirror. It runs repository sync containers, KESL antivirus scan, ZFS snapshots, snapshot replication and publication commands and repository catalog generation.

Repository catalog generation is bundled as reusable library code in mirrorer.services.catalog and is exposed through mirrorer catalog. There is no separate repo-catalog console script.

Install

Install on a shard host into the dist-mirror virtualenv:

python3.11 -m venv /opt/dist-mirror/venv
/opt/dist-mirror/venv/bin/pip install \
  --index-url "https://gitlab.example/api/v4/projects/<project-id>/packages/pypi/simple" \
  mirrorer

The installed CLI entrypoint is:

/opt/dist-mirror/venv/bin/mirrorer --help

Configuration

Runtime config is external to the package. The default path is:

/etc/dist-mirror/dist-mirror.yaml

Example:

log:
  log_level: INFO

mirror:
  workdir: /opt/dist-mirror
  syncer_dir: /etc/dist-mirror
  zfs_mirror_dataset: distpool/mirror
  zfs_mirror_dir: /distpool/mirror
  kesl_max_workers: 4

replication:
  replica_host: shard-replica01.example.org
  ssh_user: tech_dist-mirror
  skip_remote: false

antivirus:
  fail_on_threats: false

repo:
  paths:
    apt/mirror:
      deb.debian.org: debian
    dnf/repos:
      centos: centos

syncer:
  - type: apt
    name: debian
    prefix: debian
    config: syncer.yaml

  - type: rsync
    name: epel
    prefix: epel
    repos: syncer_repos.yaml

  - type: dnf
    name: rpm
    prefix: rpm
    config: syncer.yaml
    repos: syncer_repos.yaml
    dest: dnf/repos

  - type: rmt
    name: sles-x86_64
    prefix: sles-x86_64
    config: syncer.yaml
    repos: syncer_repos.yaml
    dest: rmt_x86_64/storage/public/repo

Syncer config files are expected under:

/etc/dist-mirror/

Do not commit secrets into config. Deploy /etc/dist-mirror/dist-mirror.yaml, /etc/dist-mirror/syncer.yaml and /etc/dist-mirror/syncer_repos.yaml with the infrastructure configuration system.

Commands

Common options:

mirrorer --config /etc/dist-mirror/dist-mirror.yaml --log-level INFO --no-json-logs <command>

Run all syncers:

mirrorer sync

Run one configured prefix:

mirrorer sync --syncer debian

Stage commands:

mirrorer sync
mirrorer av-scan
mirrorer snapshot
mirrorer replicate
mirrorer publish-snapshot
mirrorer publish-inventory
mirrorer publish-catalog
mirrorer catalog --root-dir /opt/dist-mirror/repo/snapshots/20260603-120000
mirrorer cleanup-state

Snapshot selection:

publish-snapshot                 # publishes state/current_snapshot.txt
publish-snapshot --snapshot NAME # publishes NAME
publish-snapshot --all           # rebuilds publication for all ZFS snapshots
publish-catalog                  # catalogs state/current_snapshot.txt
publish-catalog --snapshot NAME  # catalogs NAME
publish-catalog --rebuild        # catalogs all published snapshots

publish-snapshot builds the public repository tree from symlinks to ZFS snapshot contents and does not write metadata into the repository tree.

Metadata is stored outside the published symlink repository:

/opt/dist-mirror/metadata/
├── inventory.json
└── snapshots/
    └── <snapshot>/
        └── repo_catalog.json

The stable HTTP metadata contract is:

/metadata/inventory.json
/metadata/snapshots/<snapshot>/repo_catalog.json

The nginx mapping for /metadata/ is managed outside mirrorer, but the layout above is part of this package contract.

catalog is the direct standalone catalog builder and does not require a mirrorer config file or shard runtime binaries:

mirrorer catalog \
  --root-dir /opt/dist-mirror/repo/snapshots/20260603-120000 \
  --output /tmp/repo_catalog.json \
  --format apt \
  --format rpm \
  --summary-json

Logging

Plain text logs are the default and are intended for GitLab job logs. Use --json-logs when machine-readable logs are needed. Python tracebacks are visually marked in plain log mode.

Secrets must not be written into event fields or command arguments that are logged.

Build

Local build with uv:

uv sync --extra dev
uv run ruff check .
uv run pytest
uv build

mypy is included in the development dependencies, but the extracted codebase should be tightened incrementally before making type checks a required CI gate.

Publish

GitLab CI publishes tagged builds to the GitLab PyPI Package Registry:

uv publish \
  --publish-url "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi" \
  --username gitlab-ci-token \
  --password "${CI_JOB_TOKEN}" \
  dist/*