Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

rucio-register

Command and API to add Butler specific information to Rucio metadata.

This is a guide to using the rucio-register command for registering Butler files with Rucio.

Butler files are expected to be located in a Rucio directory structure, below a directory named for a Rucio scope. For example, if the root of the Rucio directory is "/rucio/disks/xrd1/rucio" and the Rucio scope is "test", the files should be located below "/rucio/disks/xrd1/rucio/test".

Example

The command "rucio-register" registers files with Rucio. This command requires a YAML configuration file which specifies the Rucio rse and scope, as well as the root of the directory where files are deposited, and the external reference to the Rucio RSE. This configuration file can be specified on the command line, or in the environment variable RUCIO_REGISTER_CONFIG.

The command can register data-products or raws:

for data products:

rucio-register data-products --log-level INFO -r /rucio/disks/xrd1/rucio/test -c HSC/runs/RC2/w_2023_32/DM-40356/20230814T170253Z -t visitSummary -d rubin_dataset -C register_config.yaml

for raws:

rucio-register raws --log-level INFO -r /rucio/disks/xrd1/rucio/test -d rubin_dataset --collections LATISS/raw/all -C register_config.yaml \*

Note that for raws, this is similar to how one uses the butler command

This command looks for files registered in the butler repo "/repo/main" using the "dataset-type" and "collections" arguments to query the butler. Note that the repo name's suffix is the Rucio "scope". In this example, that scope is "main".

The resulting datasets' files are registered with Rucio, as specified in the "config.yaml" file. Additionally, those files are registered with the Rucio dataset specified by the "rucio-dataset" argument.

for zip files:

rucio-register zips -d rubin_dataset --log-level INFO -C /home/lsst/rucio_register/examples/register_config.yaml --zip-file file:///rucio/disks/xrd1/rucio/test/something/2c8f9e54-9757-54c0-9119-4c3ac812a2da.zip

Note for zip files, register a single zip file at a time.

for dimension record YAML files:

rucio-register dimensions -d rubin_dataset --log-level INFO -C /home/lsst/rucio_register/examples/register_config.yaml --dimension-file file:///rucio/disks/xrd1/rucio/test/something/dimensions.yaml

Note for zip files, register a single zip file at a time.

config.yaml

The config.yaml file includes information which specifies the Rucio RSE to use, the Rucio scope, the local root of the RSE, and the URL prefix of the location where Rucio stores the files.

rucio_rse: "XRD1"
scope: "main"
rse_root: "/rucio/disks/xrd1/rucio"
dtn_url: "root://xrd1:1094//rucio"

export-datasets

Command and to dump Butler dataset, dimension, and calibration validity range data to a YAML file.

This command works alongside "rucio-register". It can be used to record all the files registered into Rucio so that their transfer and ingestion at the destination can be confirmed. In addition, it preserves dimension data and calibration validity range data that is not otherwise transferred via Rucio. This additional data can be useful for repeated ingests of raw and calibration data into Butler repositories.

Examples

To record the dimension values (notably not including the visit dimension, which would have to be regenerated) for a set of raw images:

export-datasets \
    --root /sdf/group/rubin/lsstdata/offline/instrument/ \
    --filename Dataset-LSSTCam-NoTract-20250101-0000.yaml \
    --collections LSSTCam/raw/all \
    --where "instrument='LSSTCam' and day_obs=20250101 and exposure.seq_num IN (1..99)" \
    --limit 30000 \
    /repo/main raw

--root is needed here since the original files are ingested as full URLs with direct.

To record the datasets created by a multi-site processing workflow:

export-datasets \
    --filename Dataset-LSSTCam-Tract2024-Step3-Group5-metadata.yaml \
    --collections step3/group5 \
    --where "tract=2024" \
    $LOCAL_REPO '*_metadata'

Note the use of a glob pattern to select dataset types of interest.

Metadata

Release files for lsst-rucio-register 30.0.12rc1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for lsst-rucio-register 30.0.12rc1
File Size Uploaded Provenance Actions
lsst_rucio_register-30.0.12rc1.tar.gz 29.0 kB Without provenance Details for lsst_rucio_register-30.0.12rc1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for lsst-rucio-register 30.0.12rc1
File Interpreter ABI Platform Actions
lsst_rucio_register-30.0.12rc1-py3-none-any.whl Python 3 none any Details for lsst_rucio_register-30.0.12rc1-py3-none-any.whl

Total release size: 60.4 kB

Release files / lsst_rucio_register-30.0.12rc1.tar.gz

Download URL lsst_rucio_register-30.0.12rc1.tar.gz
Size 29.0 kB
Tags Source
SHA-256 checksum
How to use checksums
e037392c52a7a48641c46680c4dd9ece9faa5a6241bd5256addaf9065e93f913
BLAKE2b-256 checksum
How to use checksums
e9cf81a72ecdd1d949e4cfd6b988922c85e5150f236670728d6bd12e32a8689c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / lsst_rucio_register-30.0.12rc1-py3-none-any.whl

Download URL lsst_rucio_register-30.0.12rc1-py3-none-any.whl
Size 31.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1418b05ecc62bc5a846862edde15b5c59aaaa66b013872765e5d165a15e62140
BLAKE2b-256 checksum
How to use checksums
192ac00e8f4fbcdddbb2d37ff39f010d8f4565802eb7443876b06c4cb008e42c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for lsst-rucio-register 30.0.12rc1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page