Client for CTA Data dCache storage access
Disclaimer
This is a prototype development only for Swiss CTA DC. Please report bugs here, and for any details please inquire the CTAO CH DC management.
Purpose
CTA data are/will be stored at CTAO-CH data center at CSCS. These data should be accessible for selected external users. In addition, these files should be available within interactive analysis platform at CSCS. This client presents an API access to these services and data.
Accessing dCache with tokens
The latest version of ctadata implements direct download/upload mode without using downloadservice as proxy.
Installation
Currently the direct API uses oidc-agent tool for the token maintanance and davix tools for downloading and uploading the files. These tools can be either compiled locally and added to PATH environment variable or installed inside [micromamba] (https://mamba.readthedocs.io/en/latest/user_guide/micromamba.html) or [conda] (https://anaconda.org/anaconda/conda) environment. Below is the example of the installation using micromamba package manager.
$ micromamba create -n ctadata python oidc-agent davix # create micromamba environment with python and required binaries
$ micromamba activate ctadata # activate the environment
$ pip install ctadata # install ctadata library in the same environment
Token maintanance
To get access to CTA-CSCS storage you need to generate OpenID Connect token. The token is temporary and needs to be updated on regular bases. This process is implemented by the token agent service which can be started by the command
$ cta-data start-agent
The token is being stored in the user's home directory, so the above command needs to be started only on a single machine. During the agent initialization process an account is created for the token maintanance. The user is being asked to authenticate the account creation by visiting the authentication page in a browser on any device.
Basic usage
As soon as the account creation process completes the token is created and one can start using other direct API functions.
Listing directory contents
import ctadata
for path in ctadata.list_dir("cta"):
print(path)
Downloading files and directories
To download contents of some file or dir:
import ctadata
# downloading single file
ctadata.fetch_and_save_file_or_dir("lst/some-data-dir/some-data-file")
# recursively downloading a directory
ctadata.fetch_and_save_file_or_dir("lst/some-data-dir", recursive=True)
or, in bash:
cta-data get lst/some-data-dir/some-data-file
cta-data get --recursive lst/some-data-dir
Uploading files and directories
To upload a file:
import ctadata
ctadata.upload_file("latest.txt", "your-folder/new-file-name.md")
ctadata.upload_file("latest.txt", "your-folder/") # will autocomplete to `your-folder/latest.txt`
You can also use command line interface to do this:
$ cta-data put latest-file-list latest-file-list-bla-bla
Using token in external tools
You can print the token to use it with external tools with the following command
$ cta-data print-token
Using dev instance of dCache server
To access dev instance one may use -d or --dev option in command interface:
$ cta-data -d list /
In python API one should use dev_instance optional parameter:
import ctadata
for path in ctadata.list_dir("/", dev_instance=True):
print(path)
Note that in order to access dev instance server you will have to maintain separate token using agent service, which can be started with command:
$ cta-data -d start-agent
Usage from within CTA CSCS JupyterHub platform
The ctadata library can be used within the CTA CSCS JupyterHub platform, as described above, without any changes or limitations.
1 maintainer
Metadata
1 maintainer
Release files for ctadata 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ctadata-0.7.0.tar.gz | 6.9 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ctadata-0.7.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 15.1 kB
Release files / ctadata-0.7.0.tar.gz
| Download URL | ctadata-0.7.0.tar.gz |
|---|---|
| Size | 6.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1a6c815562755fb693361c1c0fe90dbfbe777ceba248202bd2ded68eff226ac5
|
|
BLAKE2b-256 checksum How to use checksums |
646ae72930f9463c317044be5428cfa16e8ddedcc4f1ef23d2fc3734619ce9d0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.1 CPython/3.11.10 Linux/6.8.0-52-generic
|
Release files / ctadata-0.7.0-py3-none-any.whl
| Download URL | ctadata-0.7.0-py3-none-any.whl |
|---|---|
| Size | 8.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d4b4cb22ec2f63665eba2d5d642f1280b98c012a4f03811350493f49f1752fa4
|
|
BLAKE2b-256 checksum How to use checksums |
7e2f2f00730e544682f72c8fc12c85fa0ab428d75ba1440d9216f105bedd703c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/2.1.1 CPython/3.11.10 Linux/6.8.0-52-generic
|
Security for ctadata 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
