Skip to main content

ckanext-dcor_depot

PyPI Version Build Status Coverage Status

This plugin manages how data are stored in DCOR. There are two types of files in DCOR:

  1. Resources uploaded by users, imported from figshare, or imported from a data archive

  2. Ancillary files that are generated upon resource creation, such as condensed DC data, preview images (see ckanext-dc_view).

This plugin implements:

  • Import datasets from figshare. Existing datasets on figshare are parsed, datasets are created and resources uploaded to S3 object storage. When running the following command, the “figshare-import” organization is created and the datasets listed in figshare_dois.txt are added to CKAN:

    ckan dcor-import-figshare --limit 2
  • CLI for appending a resource to a dataset

    ckan append-resource /path/to/file dataset_id --delete-source
  • CLI for listing all S3 objects for a dataset:

    ckan dcor-list-s3-objects-for-dataset c7a98a04-4e0a-98a7-fb0b-eca379d1f219
  • CLI for listing all resources:

    ckan list-all-resources
  • CLI for pruning stale multipart uploads:

    ckan dcor-prune-stale-multipart-uploads --initiated-before-days 5 --dry-run

The S3-compatible object storage is configured in the ckan.ini file:

dcor_object_store.access_key_id = ACCESS_KEY_ID
dcor_object_store.secret_access_key = SECRET_ACCESS_KEY
dcor_object_store.endpoint_url = S3_ENDPOINT_URL
dcor_object_store.ssl_verify = true
# The bucket name is by default defined by the circle ID. Resources
# are stored in the "RES/OUR/CEID-SCHEME" in that bucket.
dcor_object_store.bucket_name = circle-{organization_id}

Installation

pip install ckanext-dcor_depot

Add this extension to the plugins and default_views in ckan.ini:

ckan.plugins = [...] dcor_depot
ckan.storage_path=/data/ckan-HOSTNAME
ckanext.dcor_depot.users_depot_name=users-HOSTNAME

This plugin stores resources to /data:

mkdir -p /data/depots/users-$(hostname)
chown -R www-data /data/depots/users-$(hostname)

Testing

If CKAN/DCOR is installed and setup for testing, this extension can be tested with pytest:

pytest ckanext

Testing is implemented via GitHub Actions. You may also set up a local docker container with CKAN and MinIO. Take a look at the GitHub Actions workflow for more information.

Metadata

Release files for ckanext-dcor-depot 1.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ckanext-dcor-depot 1.1.1
File Size Uploaded Provenance Actions
ckanext_dcor_depot-1.1.1.tar.gz 659.9 kB Without provenance Details for ckanext_dcor_depot-1.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ckanext-dcor-depot 1.1.1
File Interpreter ABI Platform Actions
ckanext_dcor_depot-1.1.1-py3-none-any.whl Python 3 none any Details for ckanext_dcor_depot-1.1.1-py3-none-any.whl

Total release size: 1.3 MB

Release files / ckanext_dcor_depot-1.1.1.tar.gz

Download URL ckanext_dcor_depot-1.1.1.tar.gz
Size 659.9 kB
Tags Source
SHA-256 checksum
How to use checksums
74eeb21673823323dc514a4773f5347de81c6e95d4c362dd56680d8f31025307
BLAKE2b-256 checksum
How to use checksums
36ba59abde365e90fdfedd17219fe3282ba1205984dd0d926f635e90cb0a4eba
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Release files / ckanext_dcor_depot-1.1.1-py3-none-any.whl

Download URL ckanext_dcor_depot-1.1.1-py3-none-any.whl
Size 656.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ddffa4f1d1f8197e231aa4299c95a23ba820842ccdeefd8a57770fcd50fef095
BLAKE2b-256 checksum
How to use checksums
b0a66b7e88c4e53a0603fe0931aac52619fbebdce7816d41ed8edd78e419293c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Security for ckanext-dcor-depot 1.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page