cloudrepo-migrate

Moves a Nexus 2, Nexus 3 or Artifactory server's repositories into CloudRepo and verifies every file: cloudrepo-migrate nexus2, cloudrepo-migrate nexus3 or cloudrepo-migrate artifactory, with the same flags, report and exit codes (--include is artifactory's alone).

From 0.9.0, artifactory moves Maven only, which a real Artifactory has run, unless --include npm, --include pypi or --include docker adds npm, PyPI or Docker, which are experimental: tested against a stand-in for Artifactory's API, not yet against a real Artifactory server. A run that finds a repository it is not moving says so, once at the start and again in its report, such as Skipped 1 Docker repository: add --include docker to move it (experimental). A --repo that names a repository of a format --include has not added is refused with exit code 2, nothing moved. nexus2 and nexus3 move what they always did, since a real server of each has run it.

• Maven: every file of each hosted (Nexus) or local (Artifactory) Maven repository, POMs, maven-metadata.xml and checksum files included.
• npm (Artifactory: only with --include npm, experimental): every version of each package in each hosted or local npm repository, scoped packages included, oldest first, then each dist-tag the source has (latest, next and every other) that CloudRepo lacks. A dist-tag CloudRepo already holds is never moved or removed: each one that names another version than the source's, or that the source lacks, is listed as kept in the report, which is not a failure. A deprecated version moves with its deprecation message; a version the source has unpublished is not in its packument, so it does not move. This is the default from version 0.4.0 for Nexus, and, with --include npm, from 0.5.0, the first version with artifactory, for Artifactory. Before 0.4.0, nexus2 and nexus3 moved only the version tagged latest, which --latest-only still does. For Artifactory, npm is tested against a stand-in for Artifactory's npm API, not yet against a real Artifactory server.
• PyPI (Nexus 3; Artifactory from 0.7.0 and only with --include pypi, experimental, tested against a stand-in for Artifactory's PyPI API, not yet against a real Artifactory server): every wheel and sdist of each hosted (Nexus 3) or local (Artifactory) PyPI repository, uploaded through CloudRepo's PyPI upload form under the project name the file's own metadata gives (other files beside them, such as signatures, are counted in the report, not moved). --login does not cover PyPI yet: move it with CLOUDREPO_TOKEN.
• Docker (Nexus 3 from 0.6.0; Artifactory from 0.7.0 and only with --include docker, experimental, tested against a stand-in for Artifactory's Docker API, not yet against a real Artifactory server): every tag of each hosted (Nexus 3) or local (Artifactory) Docker repository, with its manifest and every layer it names, a multi-platform tag with each platform's image (those first). Each layer must hash to the digest its manifest names and is uploaded by that digest; a layer CloudRepo already holds is not uploaded again, and one several images share is uploaded once. Each manifest is uploaded as the source serves it and read back by digest. CLOUDREPO_TOKEN must be a CloudRepo repository token with write access to the target Docker repository (no username to set). --login does not cover Docker yet.
• Raw and Generic (from 0.8.0; Nexus 3 raw is tested against a real Nexus 3, sonatype/nexus3:3.76.0, and Artifactory Generic against a real Artifactory OSS 7.146.39): a Nexus 3 raw or Artifactory Generic repository named with --repo SOURCE=TARGET moves into TARGET, a CloudRepo Maven repository, every file at the same path and served as it is, since a Maven repository serves any file at any path. Each file is checked against the sha1 or sha256 the source recorded for it before the upload, and read back after it; a file the source recorded neither for is not uploaded. No .sha1 or .md5 is written beside a raw file: those are a Maven client's, and only what the source stores moves (a checksum file the source stores moves as a file and must name its file's digest). A path CloudRepo cannot store, or a file over its 50 GB cap, is listed as a failure with its path and the run exits 1, in a --dry-run too. Unnamed, a raw or Generic repository is skipped with a reason that names the flag. --repo moves only the repositories it names, so name the others beside it, or move the raw one in a run of its own.
• Skipped with a reason: proxy and remote repositories (create a CloudRepo proxy instead), group repositories (a Maven or Docker group: create a CloudRepo group of the moved ones; an npm or PyPI group: point the client at a hosted repository for your own packages plus a proxy repository for the public ones, since CloudRepo groups are Maven and Docker only; a group of any other format: CloudRepo has no group for that format, so point the client at the moved hosted repositories directly), virtual and federated repositories, other formats, and files the server generates itself (Nexus's .index, .meta and root archetype-catalog.xml; Artifactory's .index and .npm). An Artifactory repository's root archetype-catalog.xml is not moved either (this version moves no archetype catalog), and the report names it on that repository's line.

Before upload, each file's bytes are compared with the checksum the source recorded for it (Nexus 2's sha1; every checksum a Nexus 3 asset lists, a wheel's or sdist's sha256 among them; the sha1 and sha256 Artifactory's AQL search lists, which Artifactory OSS answers as Pro does; an npm package's dist.shasum and dist.integrity), and a file that differs is not uploaded. After upload, every Maven, npm and PyPI file is read back from CloudRepo and its sha1 and sha256 compared with the source's bytes; a wheel or sdist must also be listed in its project's index, where pip looks for it. For Docker, every blob and manifest is checked by sha256 digest before upload and each manifest is read back by digest, but layers are not read back. A checksum file the source carries (X.sha1, X.sha256, X.sha512, X.md5) must name the digest of X. Artifactory serves X.sha1 and X.md5 without storing them, so for an Artifactory Maven repository the run writes both from X's bytes as downloaded and checked, never from Artifactory's numbers alone, and Maven finds them in CloudRepo as it did in Artifactory. Any mismatch fails the run and names the file. The source is only read, never written.

Install

pipx install cloudrepo-migrate

That installs it from pypi.org and puts cloudrepo-migrate on the PATH (pip install cloudrepo-migrate in a virtual environment works too).

It is one file, Python 3.8 or later, with no dependencies, so copying cloudrepo_migrate.py to a machine that can reach the Nexus or Artifactory server and running python3 cloudrepo_migrate.py --help works too.

Run

Create each target repository in CloudRepo first (same name as the source repository, or map it with --repo SOURCE=TARGET; a Maven repository for a raw or Generic one) at https://admin.cloudrepo.io/repositories/create, and a repository token that can write them.

Set each credential in your own shell (bash or zsh). Each IFS= read -rs waits for you to type or paste the value and press Enter: nothing is shown, the value never lands on a command line or in the shell's history, and IFS= keeps a space or tab at either end of it.

export NEXUS_USERNAME=deployer                    # leave NEXUS_USERNAME and NEXUS_PASSWORD unset for anonymous read
IFS= read -rs NEXUS_PASSWORD && export NEXUS_PASSWORD
IFS= read -rs CLOUDREPO_TOKEN && export CLOUDREPO_TOKEN
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --report-json report.json
# Nexus 3: the same, with its base URL (no /nexus on a default install)
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --dry-run
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --report-json report.json
# A raw repository moves only when named, into a CloudRepo Maven repository (here raw-installers into installers)
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --repo raw-installers=installers --dry-run

Artifactory (self-hosted or JFrog Cloud), with an access token, or a legacy API key in ARTIFACTORY_API_KEY:

IFS= read -rs ARTIFACTORY_ACCESS_TOKEN && export ARTIFACTORY_ACCESS_TOKEN   # read access to the local repositories you move
IFS= read -rs CLOUDREPO_TOKEN && export CLOUDREPO_TOKEN
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --dry-run
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --report-json report.json
# The experimental formats move only when asked, here Docker and PyPI beside Maven
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --include docker,pypi --dry-run

With the environment set, nothing prompts. On a terminal with a credential missing it asks for it.

Resume: run the same command again, --include and all. A file CloudRepo already holds with the same bytes is counted as already present and not uploaded again; a 409 on a release CloudRepo already holds counts as moved once its bytes match. The tool stores nothing between runs, so what a run moves is what its own command line asks: a resume that leaves out the --include of the first run skips those formats, and says so as any run does.

Exit codes: 0 everything moved and verified (with --dry-run: planned, and nothing written); 1 at least one file failed (every failure is listed with its path; re-run to resume); 2 a usage or preflight problem, such as a missing target repository or a refused token (nothing moved); 4 the --login approval was denied, expired or refused (nothing moved); 130 stopped with Ctrl-C (re-run to resume).

Quote

From 0.10.0, --quote asks for a quote and nothing else. It reads your Nexus or Artifactory and prints only totals, which names nothing about your servers, so you can send it to CloudRepo as it is:

cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --quote

For a small Nexus 3 it prints:

cloudrepo-migrate nexus3 quote: totals only, nothing here names your systems
  Maven  2 repositories, 13 files, 33754 bytes, 0 failed to list
  npm    1 repository, 6 versions, bytes not measured, 0 failed to list
  PyPI   1 repository, 2 distributions, 1062 bytes, 0 failed to list
  Docker 1 repository, 4 tags, 5688 bytes, 0 failed to list
Failed to list: 0 repositories.

• What it counts: for each format, the repositories, their items, and their bytes. An item is a Maven file, a PyPI distribution (a wheel or sdist), an npm version or a Docker tag. npm bytes are not measured. For an Artifactory Maven repository the count is the files Artifactory stores, not the .sha1 and .md5 it serves beside them.
• What it leaves out: proxy, group and virtual repositories, as a dry run does. On artifactory, npm, PyPI and Docker are counted only with --include, as a move does, and the quote says how many repositories it left out.
• Failed to list: the number of repositories the tool could not list in full. They are counted as repositories, and their items are not in the counts above them, so a repository that shows 0 files has either nothing in it or failed to list: this line tells you which. The exit code is 1 then, 0 otherwise.
• It names nothing: no host, port or path of your server, no repository, upstream, file, package or image name, and no error text, on stdout, on stderr and in the file. A quote that cannot run prints one fixed line and no reason; run the same command with --dry-run --org any-name in place of --quote to read the reason on your own screen (a quote takes no --org and a dry run needs one; with CLOUDREPO_TOKEN unset, as a quote needs, any name will do). A normal run and a normal --dry-run print and write what they always did.
• It reads only your server: it implies --dry-run, needs neither --org nor --cloudrepo-url, and refuses to run with CLOUDREPO_TOKEN set (exit code 2, before any call to your server): with a token set, a missing target repository stops a dry run at 0. --repo and --include narrow and widen it as they do a dry run; it cannot be used with --login or --latest-only.
• The files: --json prints the quote as one JSON document on stdout and the text on stderr, and --report-json PATH writes the same document. Its keys are exit_code, server (nexus2, nexus3 or artifactory), formats (for each format: repositories, items, unit, bytes, which is null for npm, and failed_to_list), failed_to_list (their total) and not_counted (the repositories of each format --include did not add).

JSON result

--json prints one JSON document on stdout, the same one --report-json writes, and the text report on stderr. A dry run of two Nexus 3 repositories prints:

{
  "exit_code": 0,
  "source": "https://nexus.example.com",
  "target": "https://acme.mycloudrepo.io",
  "dry_run": true,
  "repositories": [
    {
      "source": "central",
      "target": "",
      "format": "maven2",
      "action": "skipped",
      "reason": "proxy of https://repo1.maven.org: create a CloudRepo proxy repository instead",
      "files": 0,
      "bytes": 0,
      "verified": 0,
      "already_present": 0,
      "failed": 0
    },
    {
      "source": "releases",
      "target": "releases",
      "format": "maven",
      "action": "would move",
      "reason": "",
      "files": 9,
      "bytes": 21880,
      "verified": 0,
      "already_present": 0,
      "failed": 0
    }
  ],
  "failures": [],
  "kept": [],
  "minutes": 0.0,
  "totals": {
    "files": 9,
    "bytes": 21880,
    "verified": 0,
    "already_present": 0,
    "repositories": 1,
    "failed": 0
  }
}

• exit_code: the run's exit code, as above.
• dry_run: true with --dry-run, which writes nothing to CloudRepo.
• source and target: the source's base URL and CloudRepo's.
• repositories: one entry per repository. Its action is would move (a dry run), moved, skipped, or, when a target is missing or refused (exit code 2, nothing moved), blocked or ready. Its reason says why it was skipped or blocked, or notes what a moved one left behind. Its format is CloudRepo's name for a format the tool moves, else the source's. files, bytes, verified, already_present (verified files CloudRepo already held) and failed count its files.
• failures: each file that failed, with its repository, path and reason.
• kept: each npm dist-tag CloudRepo already held and kept as it was, with its repository, path (the package), tag, version (CloudRepo's) and source (the source's version, or null).
• totals: repositories, files, bytes, verified, already_present and failed, over the repositories that move.
• minutes: how long the run took.

A run that stops without a report (exit code 2 for a usage problem, 4 or 130) prints only exit_code and error, its message. A command line that cannot be parsed prints its usage on stderr, nothing on stdout, and exits 2.

Credentials

• Read only from the environment or a terminal prompt: never from the command line, never printed, never written to the report.
• Artifactory: ARTIFACTORY_ACCESS_TOKEN is sent as Authorization: Bearer; without it, ARTIFACTORY_API_KEY is sent as X-JFrog-Art-Api (JFrog has deprecated API keys, so prefer a token). Neither is ever prompted for.
• The source's login is sent only to --nexus-url or --artifactory-url, the CloudRepo token only to CloudRepo.
• Redirects are never followed, and no URL is taken from a server's answer (a Nexus 3 asset's downloadUrl is never requested; a packument's dist.tarball contributes only its file name), so neither credential can be steered to another host.
• CloudRepo must be https (plain http only to localhost). A plain-http Nexus or Artifactory gets a warning.

For agents

A coding agent can drive the whole move. The same steps ship as an Agent Skill, https://www.cloudrepo.io/docs/agents/cloudrepo-migrate/SKILL.md, and each guide has them for its server: Nexus 2, Nexus 3, Artifactory.

Tested against a real server of each kind:

• Nexus 2: hosted Maven and npm repositories.
• Nexus 3: hosted Maven, npm, PyPI and Docker repositories.
• Artifactory: local Maven repositories.

Ask the human to set up:

1. A CloudRepo organization (sign up if they have none).
2. A CloudRepo repository for each one that moves, of the same format and with the source repository's name (or the names to map with --repo SOURCE=TARGET), at https://admin.cloudrepo.io/repositories/create.
3. A CloudRepo repository token with Download and publish packages on those repositories.
4. Read access to the Nexus or Artifactory repositories: a Nexus login that can read them (none when Nexus allows anonymous reads), or an Artifactory access token (a legacy API key works too).

Ask them for the server's base URL and the CloudRepo organization.

Every credential is an environment variable the human sets in their own shell: never on a command line, never in a file, and never pasted into the chat. The tool reads only these:

• NEXUS_USERNAME and NEXUS_PASSWORD, for nexus2 and nexus3 (leave both unset to read anonymously).
• ARTIFACTORY_ACCESS_TOKEN, for artifactory, or ARTIFACTORY_API_KEY for a legacy API key.
• CLOUDREPO_TOKEN, the repository token.

Your commands see only the environment you were started with. Check each variable without printing it, such as test -n "$CLOUDREPO_TOKEN". If one is missing, ask the human to set it in their shell and start you again from that shell, or to run the commands themselves. If a credential is pasted into the chat anyway, do not use it: ask the human to replace it.

Steps:

1. Check that python3 --version is 3.8 or later. If cloudrepo-migrate --version does not answer, run pipx install cloudrepo-migrate; if it answers a version older than 0.6.0, run pipx upgrade cloudrepo-migrate.

2. Run the dry run, with the human's address and organization. Expect exit code 0 and a last line that starts Would move. On exit code 2, read the message: if a target repository is missing, ask the human to create it, then repeat the dry run. If the dry run lists a repository outside the tested set, ask the human before moving it, and name the ones to move with --repo. A dry run of artifactory that prints Skipped N ... repositories: add --include ... (experimental) has left those out on purpose: never add --include unless the human asks for that format, and then repeat the same --include in every run of that move, resumes included.

   cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org your-org --dry-run
   cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org your-org --dry-run
   cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org your-org --dry-run
   

3. Run the same command with --report-json report.json in place of --dry-run.

4. Verify all of these: the exit code is 0; the last line starts Moved and ends V of F verified, 0 failed. with V equal to F; in report.json, failures is empty, totals.failed is 0 and totals.verified equals totals.files; and no entry in repositories has the action blocked.

5. Tell the human the last line, every repository with the action skipped and its reason (the Skipped N lines of an artifactory run among them), and every entry in kept.

On exit code 1, read each failure, remove its cause, and run the same command again: it resumes. If the same file fails twice, stop and report its path and reason. Change nothing on the Nexus or Artifactory server, and do not repoint builds or CI to CloudRepo until the human says so.

Browser login (not live yet)

CloudRepo's server side for --login is not live yet; until it is, use CLOUDREPO_TOKEN as above. With it, nexus2 --org ORG --login (or nexus3, or artifactory) needs no token: the run builds its plan, prints a code and the page where your organization's owner approves exactly those repositories, and waits. The token it receives lives in memory for that run only, never on disk. --open-browser also opens the page on this machine. A set CLOUDREPO_TOKEN still wins. Exit code 4: the approval was denied, expired or refused (nothing moved).