ytclone

Copy static tables, files, YSON documents, and directories between YTsaurus and S3, preserving schemas and portable attributes.

Restoring from S3 requires a yt-s3-v2 archive created by ytclone. Earlier archive formats are rejected; re-export the source to a new S3 prefix.

Installation

pip install ytclone

Requires CPython 3.10+ on Linux x86_64 with glibc or macOS ARM64 (Apple Silicon).

Also available as standalone downloads and Docker images.

Quick start

Configure access to YTsaurus and S3:

export YT_PROXY="https://yt.example.com"
export YT_TOKEN="..."
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
export AWS_DEFAULT_REGION="us-east-1"

The S3 bucket must already exist. For an S3-compatible service, also set S3_ENDPOINT to its URL. Use AWS_SESSION_TOKEN for temporary credentials.

Copy to S3:

ytclone //home/analytics/events s3://my-bucket/export --log-file export.log

The full YT path is appended to the S3 prefix: this command stores events at s3://my-bucket/export/home/analytics/events/.

Restore to YTsaurus:

ytclone s3://my-bucket/export/home/analytics/events \
  //home/analytics/events_copy --log-file restore.log

The restore destination is used exactly as given: the children of events go directly under //home/analytics/events_copy. With --log-file, progress appears on the console and detailed logs are appended to the file.

Repeated runs

Rerun the command after an interruption or source update. Completed, unchanged nodes are skipped; other nodes are copied again. An unfinished table, file, or document restarts from the beginning.

Source data is never deleted. Destination nodes absent from the source are kept. Use a separate S3 prefix for each source cluster.

Parallel export

Use --max-streams to export large tables through several streams, or ytclone-parallel for multiple copy processes. Both apply to YT → S3; restore with ytclone.

Before you copy

• Dynamic tables are skipped and reported; links are unsupported.
• One current archive version. Failed uploads keep the previous copy available. Updates publish first, then try to delete the old copy. No history is kept.
• Permissions are not copied. ACLs, ownership, and account settings are excluded. Replacing an existing YT table, file, or document recreates the node, so its previous settings are not preserved either.
• Consistency is per node. A directory tree is not a single point-in-time snapshot.
• Concurrent exports can fail on conflicts. Updating an archive during restore can also fail the restore; rerun it. Concurrent restores to the same YT nodes are unsupported.
• S3 must support atomic conditional writes. See update and recovery details for rollback and cleanup behavior.

Documentation

Run ytclone --help or ytclone-parallel --help for all command-line options.

• User guide: paths, configuration, progress, retries, and parallel export.
• Architecture: archive format, change detection, and execution diagrams.
• Contributing: installation from source, development, and tests.

License: MIT.