Skip to main content

ckanext-dcor_depot

PyPI Version Build Status Coverage Status

This plugin manages how data are stored in DCOR. There are two types of files in DCOR:

  1. Resources uploaded by users, imported from figshare, or imported from a data archive

  2. Ancillary files that are generated upon resource creation, such as condensed DC data, preview images (see ckanext-dc_view).

This plugin implements:

  • A background job that uploads resources to S3 in after_resource_create if the resources were previously uploaded via the legacy upload route. This functionality is interesting for legacy setups that are not using S3 object storage by default.

  • Import datasets from figshare. Existing datasets on figshare are parsed, datasets are created and resources uploaded to S3 object storage. When running the following command, the “figshare-import” organization is created and the datasets listed in figshare_dois.txt are added to CKAN:

    ckan dcor-import-figshare --limit 2
  • CLI for running all background jobs (migration to S3):

    ckan run-jobs-dcor-depot
  • CLI for appending a resource to a dataset

    ckan append-resource /path/to/file dataset_id --delete-source
  • CLI for migrating data from block storage to an S3-compatible object storage service. For this, the following configuration keys must be specified in the ckan.ini file:

    dcor_object_store.access_key_id = ACCESS_KEY_ID
    dcor_object_store.secret_access_key = SECRET_ACCESS_KEY
    dcor_object_store.endpoint_url = S3_ENDPOINT_URL
    dcor_object_store.ssl_verify = true
    # The bucket name is by default defined by the circle ID. Resources
    # are stored in the "RES/OUR/CEID-SCHEME" in that bucket.
    dcor_object_store.bucket_name = circle-{organization_id}

    Usage:

    ckan dcor-migrate-resources-to-object-store --modified-days 2 --delete-after-migration --verify-checksum
  • CLI for listing all S3 objects for a dataset:

    ckan dcor-list-s3-objects-for-dataset c7a98a04-4e0a-98a7-fb0b-eca379d1f219
  • CLI for listing all resources:

    ckan list-all-resources
  • CLI for pruning stale multipart uploads:

    ckan dcor-prune-stale-multipart-uploads --initiated-before-days 5 --dry-run

Installation

pip install ckanext-dcor_depot

Add this extension to the plugins and default_views in ckan.ini:

ckan.plugins = [...] dcor_depot
ckan.storage_path=/data/ckan-HOSTNAME
ckanext.dcor_depot.users_depot_name=users-HOSTNAME

This plugin stores resources to /data:

mkdir -p /data/depots/users-$(hostname)
chown -R www-data /data/depots/users-$(hostname)

Testing

If CKAN/DCOR is installed and setup for testing, this extension can be tested with pytest:

pytest ckanext

Testing is implemented via GitHub Actions. You may also set up a local docker container with CKAN and MinIO. Take a look at the GitHub Actions workflow for more information.

Metadata

Release files for ckanext-dcor-depot 1.0.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ckanext-dcor-depot 1.0.6
File Size Uploaded Provenance Actions
ckanext_dcor_depot-1.0.6.tar.gz 4.3 MB Without provenance Details for ckanext_dcor_depot-1.0.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ckanext-dcor-depot 1.0.6
File Interpreter ABI Platform Actions
ckanext_dcor_depot-1.0.6-py3-none-any.whl Python 3 none any Details for ckanext_dcor_depot-1.0.6-py3-none-any.whl

Total release size: 8.6 MB

Release files / ckanext_dcor_depot-1.0.6.tar.gz

Download URL ckanext_dcor_depot-1.0.6.tar.gz
Size 4.3 MB
Tags Source
SHA-256 checksum
How to use checksums
09a0aad8d411034788f4360462f3e3b213671b0d26e1d50a27446594f2a83ad4
BLAKE2b-256 checksum
How to use checksums
d522c16cb52f53440a1997e9a84b20b64318589daa138d723813dca17fe21e6b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Release files / ckanext_dcor_depot-1.0.6-py3-none-any.whl

Download URL ckanext_dcor_depot-1.0.6-py3-none-any.whl
Size 4.3 MB
Tags Python 3
SHA-256 checksum
How to use checksums
7ecdb4f5308b47f0fb2de596282f6e573593ea8be933dc96f42e83745521148d
BLAKE2b-256 checksum
How to use checksums
9618ef3b37ccde215fc5c4fec1a5a66853d78591672cb10b2f969288890b797b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Security for ckanext-dcor-depot 1.0.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.6 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page