Conda OCI Mirror

Welcome to the Conda Forge Mirror python module! We use this module to retrieve and update conda mirror packages. You can see it being used in the wild here and brief usage is shown below.

Concepts

Mirror

A mirror means you will copy (mirror) all packages from a regular conda-channel to an OCI registry. In practice, this typically means all packages, however for the purposes of testing we allow selection of a subset. You must have control of the registry you intend to mirror to, meaning you can push and pull from it. When you do a mirror, the repodata.json is always pulled fresh, and any local changes you've made are over-written. We do this so the local cache is in sync with the remote. The downloaded repodata snapshot is published only after all scheduled package uploads succeed; an upload failure prevents repodata publication. Metadata is not downloaded again at publication time. Filtered mirrors still publish the full upstream repodata, not a filtered index.

Packages whose encoded version/build cannot form a valid OCI tag (for example, build strings containing spaces or *, or tags longer than 128 characters) are skipped before package downloads or registry lookups for that archive. Each skip is logged as a warning with the source archive and tag. Mirroring and cache operations continue with valid packages, and the CLI prints a summary warning but still exits with status 0. Python callers can inspect mirror.errors after each operation; PackageRepo.find_packages() also exposes its scan errors via repo.errors. This does not rename packages or remove entries from the upstream repodata snapshot: a run with skipped packages is incomplete, not a successful full mirror. No GitHub issues are created automatically.

OCI channel layout

Package archives are stored by package name and version/build tag. Repository metadata is stored at:

<registry>/<channel>/<subdir>/repodata.json:latest

That manifest contains repodata.json, repodata.json.zst, and, when the source channel provides it, repodata_shards.msgpack.zst as separate media-type layers. Content-addressed repodata shards are stored as:

<registry>/<channel>/<subdir>/repodata.json/shards:<sha256>

The shards repository is nested under repodata.json so it can never collide with a package repository, since conda package names cannot contain /. Clients map the index's relative ./shards/<sha256>.msgpack.zst URLs to this repository.

Shards are pushed before the index that references them. Each run compares the upstream index with the one published at repodata.json:latest and only downloads and pushes shards the published index does not already reference, so the registry's tags are not listed. Until the first index is published, the shard tags are listed once so that an interrupted first push resumes where it stopped. Shards are uploaded by --shard-workers threads (default 2); keep this low to avoid overloading the registry.

--repush-shards pushes every shard in the index again, not only new ones. Use it once after a change to the shard manifests, such as the switch of the shard media type from +zstd to +zst: shards that are already published are otherwise never pushed again and keep their old manifests. The blobs are unchanged, so this rewrites one manifest per shard. A run with this flag does not resume, so if it is cut off, run it again with the flag.

Shards are mirrored only for complete channel mirrors. A package-filtered mirror cannot publish the upstream shard index because it would reference packages that were not mirrored.

Pull Cache

A pull-cache can pull from a registry that you may not be able to write to, to your local cache. It selects the newest conda version and build number separately for each package and archive format; when builds tie, the first record in repodata wins. Versions are not ordered by upload date. Downloaded OCI layers are checked against their SHA-256 digests before replacing local files. Interrupted or invalid downloads leave an existing destination unchanged and remove temporary files. Missing metadata, missing requested package layers, and download failures cause the command to fail rather than silently succeed. Upstream package downloads also replace files atomically and verify checksums when repodata supplies them.

Push Cache

A push-cache can push your local cache to a registry you control. This means that we compare packages you've built against what are known in the repodata.json, and we push the ones that are not known to the repodata.json. A push cache with --push-all (also available as --all) will push all selected local packages to your registry, regardless of status. Cache pushes leave local archives and repodata unchanged, including during dry runs and failed uploads. They compare against the existing local repodata without running conda index.

Archive formats and concurrent publishers

A package tag can contain both .conda and .tar.bz2 archive layers. Uploading one format preserves the other; shared info layers describe the most recently uploaded archive. No new tag naming scheme is required. Uploads to the same tag are serialized within a run; independent mirror processes must not write the same tag concurrently.

Normal mirroring keeps the tag-only check for builds with a single upstream format. When upstream lists both formats, the mirror checks the manifest and repairs a missing layer. Known-new tags need no preservation lookup on their first upload; cache pushes and retries inspect existing manifests. None of these checks download package blobs. For a full layer-level audit, PackageRepo.get_existing_packages() verifies manifests by default; this is intentionally more expensive than normal mirror planning. Registry authorization and server errors are not treated as absent packages.

Usage

Install

Python 3.8 or newer is required. Conda version ordering uses py-rattler, not PEP 440 ordering.

Create a new environment:

$ python -m venv env
$ source env/bin/activate

And install:

$ pip install -e .

Authentication

For registries that require authentication, set ORAS_USER and ORAS_PASS in the environment. Use --dry-run to test without pushing. Missing credentials do not enable dry-run automatically; anonymous writes are attempted and succeed only if the registry permits them.

export ORAS_USER=myuser
export ORAS_PASS=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxx

Note that when you make the token, ensure the packages box (for read and write) is checked.

Each Mirror owns an independent client, including its transport and credentials. Worker processes retain that configuration but create their own HTTP sessions. For standalone reads, pass the original registry URL (including http:// for local registries), or pass client=mirror.client when constructing a PackageRepo from mirror.registry. The legacy global oras client remains available for direct callers but is not used by mirror operations.

Shared options

All commands accept repeatable --package names or glob patterns, such as --package 'numpy*'. Quote patterns to prevent shell expansion. --push-all includes already-indexed packages but still honors the package filter. --quiet, --debug, and --push-all are switches; do not append true or false. Workers must be positive. --upload-delay sets a nonnegative delay in milliseconds; --timeout remains an alias for it, not a network timeout. --quiet suppresses informational logs, not all progress output.

Mirror

The main functionality of the mirror is to create a copy of a channel and packages and push them to a mirror (or just stage for a dry run). Let's say that we want to mirror conda-forge and the package zlib We might first do a dry run:

$ conda-oci mirror --channel conda-forge --package zlib --dry-run

And then for realsies, here we want to push to our organization myorg

$ conda-oci mirror --channel conda-forge --package zlib --registry ghcr.io/myorg

For this command, we are pulling packages from a channel and mirroring to the registry defined under user.

Pull Cache

You can use pull-cache to pull the latest packages to a local cache. The same authentication is needed only if the registry packages are private or internal. Akin to the others, you can start with a --dry-run

$ conda-oci pull-cache --registry ghcr.io/researchapps --dry-run

Target a specific arch (subdir of the cache) and one package we know exist

$ conda-oci pull-cache --registry ghcr.io/researchapps --dry-run --subdir linux-64

If you want to preview what would be pulled, set --dry-run:

$ conda-oci pull-cache --registry ghcr.io/researchapps --subdir linux-64 --package zlib --dry-run

Downloading conda-forge/linux-64/repodata.json to /home/vanessa/Desktop/Code/conda_oci_mirror/cache/conda-forge/linux-64/conda-forge/linux-64/repodata.json
Would be pulling zlib, but dry-run is set.

For this command, we are pulling packages from our registry defined as user and mirroring to a local filesystem cache.

Push Cache

You can use push-cache to push the packages in your cache to your remote. This command will require authentication, and you are also encouraged to use --dry-run first.

$ conda-oci push-cache --registry ghcr.io/researchapps --dry-run --package zlib --subdir linux-64

Python API

Mirror

Let's say we want to do a simple mirror of a conda forge package to our own registry at http://127.0.0.1:5000. Note that cache_dir will default to "cache" in your present working directory.

from conda_oci_mirror.mirror import Mirror

mirror = Mirror(
    channel="conda-forge",
    packages=["redo"],
    # Push repodata and packages to this registry
    registry="http://127.0.0.1:5000/dinosaur",
    subdirs=["noarch"],
)

updates = mirror.update()

Pull Cache

A pull cache means that we start with a mirror, but then pull down packages from our registry that aren't in our local cache. The creation of the Mirror is the same, except we call a different function.

from conda_oci_mirror.mirror import Mirror

mirror = Mirror(
    channel="conda-forge",
    packages=["redo"],
    # Push repodata and packages to this registry
    registry="http://127.0.0.1:5000/dinosaur",
    subdirs=["noarch"],
)

latest_packages = mirror.pull_latest()

Push Cache

A push cache checks your local repodata.json and finds packages that exist that aren't yet added, and then updates and pushes them to your registry. You can also use push_all to push all local packages regardless of presence in the repodata.json. First, here is pushing new:

from conda_oci_mirror.mirror import Mirror

mirror = Mirror(
    channel="conda-forge",
    packages=["redo"],
    # Push repodata and packages to this registry
    registry="http://127.0.0.1:5000/dinosaur",
    subdirs=["noarch"],
)

pushed_packages = mirror.push_new()

And pushing all:

all_packages = mirror.push_all()

If you want to run in serial (for either of the above):

all_packages = mirror.push_all(serial=True)

Right now this is checking against the local repodata.json, and I'm not sure if this should be checking against the registry (intuitively it should if we expect a push-cache to push local cache entries that aren't in the remote to the remote).

PackageRepo

You can use the PackageRepo class to interact directly with a package in a registry. As an example, let's say we want to interact with a local registry http://127.0.0.1/dinosaur to look for the conda-forge channel, linux-64 subdirectory and package zlib. You might first mirror the package there as follows:

$ conda-oci mirror --registry http://127.0.0.1:5000/dinosaur --subdir linux-64 --package zlib

You could also do this first with the Python API to get explicitly back the list of tags mirrored. We would create a package repo as follows:

from conda_oci_mirror.repo import PackageRepo
import os

cache_dir = os.path.join(os.getcwd(), "cache")
repo = PackageRepo(
    "conda-forge", "linux-64", cache_dir, registry="http://127.0.0.1:5000/dinosaur"
)

Now let's retrieve the index.json. You need the exact tag you are interested in - there is no "latest."

# Should retrieve from
# http://127.0.0.1:5000/dinosaur/conda-forge/linux-64/zlib:1.2.11-0'
index_json = repo.get_index_json("zlib:1.2.11-0")

{'arch': 'x86_64',
 'build': '0',
 'build_number': 0,
 'depends': [],
 'license': 'zlib',
 'license_family': 'Other',
 'name': 'zlib',
 'platform': 'linux',
 'subdir': 'linux-64',
 'version': '1.2.11'}

Now we might want to get the package info:

info = repo.get_info("zlib:1.2.11-0")
# <tarfile.TarFile at 0x7f7b78a00e80>

This is technically a tarfile, so to iterate over members:

for member in info:
    print(member.name)

And finally, to get the full package archive:

pkg = repo.get_package("zlib:1.2.11-0")

Note that we first try to get the new format (.conda) and fall back to .tar.bz2.

Development

If you want to test pushing to packages, make sure to export your credentials first, as discussed above. Then ensure that --user is targeting your GitHub user or organizational account to push to:

$ conda-oci mirror --channel conda-forge --package zlib --registry ghcr.io/researchapps

For a package that includes the new format:

# Mirror zope.event from conda-forge to our local registry
$ conda-oci mirror --channel conda-forge --subdir noarch --package zope.event --registry http://127.0.0.0:5000/dinosaur

# Pull the package from our registry to out local cache
$ conda-oci pull-cache --channel conda-forge --subdir noarch --package zope.event --registry http://127.0.0.0:5000/dinosaur

You can also develop with a local registry (instead of ghcr.io):

$ docker run -it --rm -p 5000:5000 ghcr.io/oras-project/registry:latest

And then specify your local registry - oras will fall back to insecure mode given that you've provided http://.

$ conda-oci mirror --channel conda-forge --package testinfra --registry http://127.0.0.0:5000/dinosaur --subdir noarch

And run tests:

$ pixi run test

To use an already-running test registry instead of starting Docker containers, set CONDA_OCI_TEST_REGISTRY=http://127.0.0.1:5000. Tests use isolated namespaces in that registry.

See TODO.md for some questions and items to do.

Linting

We use ruff for linting and formatting, run through pixi:

$ pixi run lint  # check
$ pixi run fmt   # fix and format