cloudrepo-migrate

Moves a Nexus 2, Nexus 3 or Artifactory server's repositories into CloudRepo and verifies every file: cloudrepo-migrate nexus2, cloudrepo-migrate nexus3 or cloudrepo-migrate artifactory, with the same flags, report and exit codes.

• Maven: every file of each hosted (Nexus) or local (Artifactory) Maven repository, POMs, maven-metadata.xml and checksum files included.
• npm: every version of each package in each hosted or local npm repository, scoped packages included, oldest first, then each dist-tag the source has (latest, next and every other) that CloudRepo lacks. A dist-tag CloudRepo already holds is never moved or removed: each one that names another version than the source's, or that the source lacks, is listed as kept in the report, which is not a failure. A deprecated version moves with its deprecation message; a version the source has unpublished is not in its packument, so it does not move. This is the default from version 0.4.0 for Nexus, and from 0.5.0, the first version with artifactory, for Artifactory. Before 0.4.0, nexus2 and nexus3 moved only the version tagged latest, which --latest-only still does. For Artifactory, npm is tested against a stand-in for Artifactory's npm API, not yet against a real Artifactory server.
• PyPI (Nexus 3; Artifactory from 0.7.0, tested against a stand-in for Artifactory's PyPI API, not yet against a real Artifactory server): every wheel and sdist of each hosted (Nexus 3) or local (Artifactory) PyPI repository, uploaded through CloudRepo's PyPI upload form under the project name the file's own metadata gives (other files beside them, such as signatures, are counted in the report, not moved). --login does not cover PyPI yet: move it with CLOUDREPO_TOKEN.
• Docker (Nexus 3 from 0.6.0; Artifactory from 0.7.0, tested against a stand-in for Artifactory's Docker API, not yet against a real Artifactory server): every tag of each hosted (Nexus 3) or local (Artifactory) Docker repository, with its manifest and every layer it names, a multi-platform tag with each platform's image (those first). Each layer must hash to the digest its manifest names and is uploaded by that digest; a layer CloudRepo already holds is not uploaded again, and one several images share is uploaded once. Each manifest is uploaded as the source serves it and read back by digest. CLOUDREPO_TOKEN must be a CloudRepo repository token with write access to the target Docker repository (no username to set). --login does not cover Docker yet.
• Skipped with a reason: proxy and remote repositories (create a CloudRepo proxy instead), group repositories (a Maven or Docker group: create a CloudRepo group of the moved ones; an npm or PyPI group: point the client at a hosted repository for your own packages plus a proxy repository for the public ones, since CloudRepo groups are Maven and Docker only; a group of any other format: CloudRepo has no group for that format, so point the client at the moved hosted repositories directly), virtual and federated repositories, other formats, and files the server generates itself (Nexus's .index, .meta and root archetype-catalog.xml; Artifactory's .index and .npm). An Artifactory repository's root archetype-catalog.xml is not moved either (this version moves no archetype catalog), and the report names it on that repository's line.

Before upload, each file's bytes are compared with the checksum the source recorded for it (Nexus 2's sha1; every checksum a Nexus 3 asset lists, a wheel's or sdist's sha256 among them; the sha1 and sha256 Artifactory's AQL search lists, which Artifactory OSS answers as Pro does; an npm package's dist.shasum and dist.integrity), and a file that differs is not uploaded. After upload, every Maven, npm and PyPI file is read back from CloudRepo and its sha1 and sha256 compared with the source's bytes; a wheel or sdist must also be listed in its project's index, where pip looks for it. For Docker, every blob and manifest is checked by sha256 digest before upload and each manifest is read back by digest, but layers are not read back. A checksum file the source carries (X.sha1, X.sha256, X.sha512, X.md5) must name the digest of X. Artifactory serves X.sha1 and X.md5 without storing them, so for Artifactory the run writes both from X's bytes as downloaded and checked, never from Artifactory's numbers alone, and Maven finds them in CloudRepo as it did in Artifactory. Any mismatch fails the run and names the file. The source is only read, never written.

Install

pipx install cloudrepo-migrate

That installs it from pypi.org and puts cloudrepo-migrate on the PATH (pip install cloudrepo-migrate in a virtual environment works too).

It is one file, Python 3.8 or later, with no dependencies, so copying cloudrepo_migrate.py to a machine that can reach the Nexus or Artifactory server and running python3 cloudrepo_migrate.py --help works too.

Run

Create each target repository in CloudRepo first (same name as the source repository, or map it with --repo SOURCE=TARGET) at https://admin.cloudrepo.io/repositories/create, and a repository token that can write them.

Set each credential in your own shell (bash or zsh). Each IFS= read -rs waits for you to type or paste the value and press Enter: nothing is shown, the value never lands on a command line or in the shell's history, and IFS= keeps a space or tab at either end of it.

export NEXUS_USERNAME=deployer                    # leave NEXUS_USERNAME and NEXUS_PASSWORD unset for anonymous read
IFS= read -rs NEXUS_PASSWORD && export NEXUS_PASSWORD
IFS= read -rs CLOUDREPO_TOKEN && export CLOUDREPO_TOKEN
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --dry-run
cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org acme --report-json report.json
# Nexus 3: the same, with its base URL (no /nexus on a default install)
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --dry-run
cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org acme --report-json report.json

Artifactory (self-hosted or JFrog Cloud), with an access token, or a legacy API key in ARTIFACTORY_API_KEY:

IFS= read -rs ARTIFACTORY_ACCESS_TOKEN && export ARTIFACTORY_ACCESS_TOKEN   # read access to the local repositories you move
IFS= read -rs CLOUDREPO_TOKEN && export CLOUDREPO_TOKEN
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --dry-run
cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org acme --report-json report.json

With the environment set, nothing prompts. On a terminal with a credential missing it asks for it.

Resume: run the same command again. A file CloudRepo already holds with the same bytes is counted as already present and not uploaded again; a 409 on a release CloudRepo already holds counts as moved once its bytes match.

Exit codes: 0 everything moved and verified (with --dry-run: planned, and nothing written); 1 at least one file failed (every failure is listed with its path; re-run to resume); 2 a usage or preflight problem, such as a missing target repository or a refused token (nothing moved); 4 the --login approval was denied, expired or refused (nothing moved); 130 stopped with Ctrl-C (re-run to resume).

JSON result

--json prints one JSON document on stdout, the same one --report-json writes, and the text report on stderr. A dry run of two Nexus 3 repositories prints:

{
  "exit_code": 0,
  "source": "https://nexus.example.com",
  "target": "https://acme.mycloudrepo.io",
  "dry_run": true,
  "repositories": [
    {
      "source": "central",
      "target": "",
      "format": "maven2",
      "action": "skipped",
      "reason": "proxy of https://repo1.maven.org: create a CloudRepo proxy repository instead",
      "files": 0,
      "bytes": 0,
      "verified": 0,
      "already_present": 0,
      "failed": 0
    },
    {
      "source": "releases",
      "target": "releases",
      "format": "maven",
      "action": "would move",
      "reason": "",
      "files": 9,
      "bytes": 21880,
      "verified": 0,
      "already_present": 0,
      "failed": 0
    }
  ],
  "failures": [],
  "kept": [],
  "minutes": 0.0,
  "totals": {
    "files": 9,
    "bytes": 21880,
    "verified": 0,
    "already_present": 0,
    "repositories": 1,
    "failed": 0
  }
}

• exit_code: the run's exit code, as above.
• dry_run: true with --dry-run, which writes nothing to CloudRepo.
• source and target: the source's base URL and CloudRepo's.
• repositories: one entry per repository. Its action is would move (a dry run), moved, skipped, or, when a target is missing or refused (exit code 2, nothing moved), blocked or ready. Its reason says why it was skipped or blocked, or notes what a moved one left behind. Its format is CloudRepo's name for a format the tool moves, else the source's. files, bytes, verified, already_present (verified files CloudRepo already held) and failed count its files.
• failures: each file that failed, with its repository, path and reason.
• kept: each npm dist-tag CloudRepo already held and kept as it was, with its repository, path (the package), tag, version (CloudRepo's) and source (the source's version, or null).
• totals: repositories, files, bytes, verified, already_present and failed, over the repositories that move.
• minutes: how long the run took.

A run that stops without a report (exit code 2 for a usage problem, 4 or 130) prints only exit_code and error, its message. A command line that cannot be parsed prints its usage on stderr, nothing on stdout, and exits 2.

Credentials

• Read only from the environment or a terminal prompt: never from the command line, never printed, never written to the report.
• Artifactory: ARTIFACTORY_ACCESS_TOKEN is sent as Authorization: Bearer; without it, ARTIFACTORY_API_KEY is sent as X-JFrog-Art-Api (JFrog has deprecated API keys, so prefer a token). Neither is ever prompted for.
• The source's login is sent only to --nexus-url or --artifactory-url, the CloudRepo token only to CloudRepo.
• Redirects are never followed, and no URL is taken from a server's answer (a Nexus 3 asset's downloadUrl is never requested; a packument's dist.tarball contributes only its file name), so neither credential can be steered to another host.
• CloudRepo must be https (plain http only to localhost). A plain-http Nexus or Artifactory gets a warning.

For agents

A coding agent can drive the whole move. The same steps ship as an Agent Skill, https://www.cloudrepo.io/docs/agents/cloudrepo-migrate/SKILL.md, and each guide has them for its server: Nexus 2, Nexus 3, Artifactory.

Tested against a real server of each kind:

• Nexus 2: hosted Maven and npm repositories.
• Nexus 3: hosted Maven, npm, PyPI and Docker repositories.
• Artifactory: local Maven repositories.

Ask the human to set up:

1. A CloudRepo organization (sign up if they have none).
2. A CloudRepo repository for each one that moves, of the same format and with the source repository's name (or the names to map with --repo SOURCE=TARGET), at https://admin.cloudrepo.io/repositories/create.
3. A CloudRepo repository token with Read + write on those repositories.
4. Read access to the Nexus or Artifactory repositories: a Nexus login that can read them (none when Nexus allows anonymous reads), or an Artifactory access token (a legacy API key works too).

Ask them for the server's base URL and the CloudRepo organization.

Every credential is an environment variable the human sets in their own shell: never on a command line, never in a file, and never pasted into the chat. The tool reads only these:

• NEXUS_USERNAME and NEXUS_PASSWORD, for nexus2 and nexus3 (leave both unset to read anonymously).
• ARTIFACTORY_ACCESS_TOKEN, for artifactory, or ARTIFACTORY_API_KEY for a legacy API key.
• CLOUDREPO_TOKEN, the repository token.

Your commands see only the environment you were started with. Check each variable without printing it, such as test -n "$CLOUDREPO_TOKEN". If one is missing, ask the human to set it in their shell and start you again from that shell, or to run the commands themselves. If a credential is pasted into the chat anyway, do not use it: ask the human to replace it.

Steps:

1. Check that python3 --version is 3.8 or later. If cloudrepo-migrate --version does not answer, run pipx install cloudrepo-migrate; if it answers a version older than 0.6.0, run pipx upgrade cloudrepo-migrate.

2. Run the dry run, with the human's address and organization. Expect exit code 0 and a last line that starts Would move. On exit code 2, read the message: if a target repository is missing, ask the human to create it, then repeat the dry run. If the dry run lists a repository outside the tested set, ask the human before moving it, and name the ones to move with --repo.

   cloudrepo-migrate nexus2 --nexus-url https://nexus.example.com/nexus --org your-org --dry-run
   cloudrepo-migrate nexus3 --nexus-url https://nexus.example.com --org your-org --dry-run
   cloudrepo-migrate artifactory --artifactory-url https://artifactory.example.com/artifactory --org your-org --dry-run
   

3. Run the same command with --report-json report.json in place of --dry-run.

4. Verify all of these: the exit code is 0; the last line starts Moved and ends V of F verified, 0 failed. with V equal to F; in report.json, failures is empty, totals.failed is 0 and totals.verified equals totals.files; and no entry in repositories has the action blocked.

5. Tell the human the last line, every repository with the action skipped and its reason, and every entry in kept.

On exit code 1, read each failure, remove its cause, and run the same command again: it resumes. If the same file fails twice, stop and report its path and reason. Change nothing on the Nexus or Artifactory server, and do not repoint builds or CI to CloudRepo until the human says so.

Browser login (not live yet)

CloudRepo's server side for --login is not live yet; until it is, use CLOUDREPO_TOKEN as above. With it, nexus2 --org ORG --login (or nexus3, or artifactory) needs no token: the run builds its plan, prints a code and the page where your organization's owner approves exactly those repositories, and waits. The token it receives lives in memory for that run only, never on disk. --open-browser also opens the page on this machine. A set CLOUDREPO_TOKEN still wins. Exit code 4: the approval was denied, expired or refused (nothing moved).