cloudrepo-migrate

Moves a Nexus 2, Nexus 3 or Artifactory server's repositories into CloudRepo and verifies every file: cloudrepo-migrate nexus2, cloudrepo-migrate nexus3 or cloudrepo-migrate artifactory, with the same flags, report and exit codes.

• Maven: every file of each hosted (Nexus) or local (Artifactory) Maven repository, POMs, maven-metadata.xml and checksum files included.
• npm: every version of each package in each hosted or local npm repository, scoped packages included, oldest first, then each dist-tag the source has (latest, next and every other) that CloudRepo lacks. A dist-tag CloudRepo already holds is never moved or removed: each one that names another version than the source's, or that the source lacks, is listed as kept in the report, which is not a failure. A deprecated version moves with its deprecation message; a version the source has unpublished is not in its packument, so it does not move. This is the default from version 0.4.0 for Nexus, and from 0.5.0, the first version with artifactory, for Artifactory. Before 0.4.0, nexus2 and nexus3 moved only the version tagged latest, which --latest-only still does. For Artifactory, npm is tested against a stand-in for Artifactory's npm API, not yet against a real Artifactory server.
• PyPI (Nexus 3): every wheel and sdist of each hosted PyPI repository, uploaded through CloudRepo's PyPI upload form under the project name the file's own metadata gives (other files beside them, such as signatures, are counted in the report, not moved). --login does not cover PyPI yet: move it with CLOUDREPO_TOKEN.
• Docker (Nexus 3, from 0.6.0): every tag of each hosted Docker repository, with its manifest and every layer it names, a multi-platform tag with each platform's image (those first). Each layer must hash to the digest its manifest names and is uploaded by that digest; a layer CloudRepo already holds is not uploaded again, and one several images share is uploaded once. Each manifest is uploaded as Nexus serves it and read back by digest. CLOUDREPO_TOKEN must be a CloudRepo repository token with write access to the target Docker repository (no username to set). --login does not cover Docker yet.
• Skipped with a reason: proxy and remote repositories (create a CloudRepo proxy instead), group repositories (create a CloudRepo group of the moved ones), virtual and federated repositories, other formats, and files the server generates itself (Nexus's .index, .meta and root archetype-catalog.xml; Artifactory's .index and .npm). An Artifactory repository's root archetype-catalog.xml is not moved either (this version moves no archetype catalog), and the report names it on that repository's line.

Before upload, each file's bytes are compared with the checksum the source recorded for it (Nexus 2's sha1; every checksum a Nexus 3 asset lists, a wheel's or sdist's sha256 among them; the sha1 and sha256 Artifactory's AQL search lists, which Artifactory OSS answers as Pro does; an npm package's dist.shasum and dist.integrity), and a file that differs is not uploaded. After upload, every Maven, npm and PyPI file is read back from CloudRepo and its sha1 and sha256 compared with the source's bytes; a wheel or sdist must also be listed in its project's index, where pip looks for it. For Docker, every blob and manifest is checked by sha256 digest before upload and each manifest is read back by digest, but layers are not read back. A checksum file the source carries (X.sha1, X.sha256, X.sha512, X.md5) must name the digest of X. Artifactory serves X.sha1 and X.md5 without storing them, so for Artifactory the run writes both from X's bytes as downloaded and checked, never from Artifactory's numbers alone, and Maven finds them in CloudRepo as it did in Artifactory. Any mismatch fails the run and names the file. The source is only read, never written.

Install

pipx install cloudrepo-migrate

That installs it from pypi.org and puts cloudrepo-migrate on the PATH (pip install cloudrepo-migrate in a virtual environment works too).

It is one file, Python 3.8 or later, with no dependencies, so copying cloudrepo_migrate.py to a machine that can reach the Nexus or Artifactory server and running python3 cloudrepo_migrate.py --help works too.

Run

Create each target repository in CloudRepo first (same name as the source repository, or map it with --repo SOURCE=TARGET) at https://admin.cloudrepo.io/repositories/create, and a repository token that can write them.

export NEXUS_USERNAME=deployer NEXUS_PASSWORD=...   # leave both unset for anonymous read
export CLOUDREPO_TOKEN=...
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --report-json report.json
# Nexus 3: the same, with its base URL (no /nexus on a default install)
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --dry-run
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --report-json report.json

Artifactory (self-hosted or JFrog Cloud), with an access token, or a legacy API key in ARTIFACTORY_API_KEY:

export ARTIFACTORY_ACCESS_TOKEN=...   # read access to the local repositories you move
export CLOUDREPO_TOKEN=...
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --dry-run
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --report-json report.json

With the environment set, nothing prompts. On a terminal with a credential missing it asks for it.

Resume: run the same command again. A file CloudRepo already holds with the same bytes is counted as already present and not uploaded again; a 409 on a release CloudRepo already holds counts as moved once its bytes match.

Exit codes: 0 everything moved and verified; 1 at least one file failed (every failure is listed with its path; re-run to resume); 2 a usage or preflight problem, such as a missing target repository or a refused token (nothing moved).

Credentials

• Read only from the environment or a terminal prompt: never from the command line, never printed, never written to the report.
• Artifactory: ARTIFACTORY_ACCESS_TOKEN is sent as Authorization: Bearer; without it, ARTIFACTORY_API_KEY is sent as X-JFrog-Art-Api (JFrog has deprecated API keys, so prefer a token). Neither is ever prompted for.
• The source's login is sent only to --nexus-url or --artifactory-url, the CloudRepo token only to CloudRepo.
• Redirects are never followed, and no URL is taken from a server's answer (a Nexus 3 asset's downloadUrl is never requested; a packument's dist.tarball contributes only its file name), so neither credential can be steered to another host.
• CloudRepo must be https (plain http only to localhost). A plain-http Nexus or Artifactory gets a warning.

Browser login (not live yet)

CloudRepo's server side for --login is not live yet; until it is, use CLOUDREPO_TOKEN as above. With it, nexus2 --org ORG --login (or nexus3, or artifactory) needs no token: the run builds its plan, prints a code and the page where your organization's owner approves exactly those repositories, and waits. The token it receives lives in memory for that run only, never on disk. --open-browser also opens the page on this machine. A set CLOUDREPO_TOKEN still wins. Exit code 4: the approval was denied, expired or refused (nothing moved).