ckanext-file-keeper-cloud
This CKAN extension provides a way to store uploaded files on external cloud storage services. It extends management functionality introduced in CKAN v2.12 and provides a set of adapters for different cloud storage providers.
Requirements
Compatibility with core CKAN versions:
| CKAN version | Compatible? |
|---|---|
| 2.11 | no |
| 2.12 | yes |
Installation
To install ckanext-file-keeper-cloud:
-
Activate your CKAN virtual environment, for example:
. /usr/lib/ckan/default/bin/activate -
Install the package
# to install all adapters pip install 'ckanext-file-keeper-cloud[all]' # or, to install only specific adapter pip install 'ckanext-file-keeper-cloud[s3]' pip install 'ckanext-file-keeper-cloud[gcs]' pip install 'ckanext-file-keeper-cloud[azure]' pip install 'ckanext-file-keeper-cloud[libcloud]'
-
Add
file_keeper_cloudto theckan.pluginssetting in your CKAN config file (by default the config file is located at/etc/ckan/default/ckan.ini). -
Restart CKAN.
Available adapters
Note, cloud providers can be emulated using docker images. Even though these images do not replicate ideally corresponding cloud provider, generally they are close enough to be used for local development and testing.
Never use inline secrets in the config file. Use references to environment variables(like in examples below) or ckanext-envvars.
ckan:s3
AWS S3 adapter. Wraps file-keeper's implementation
Installation:
pip install 'ckanext-file-keeper-cloud[s3]'
Configuration example:
ckan.files.storage.my_cloud.type = ckan:s3
ckan.files.storage.my_cloud.bucket = my_bucket
ckan.files.storage.my_cloud.key = %(CKAN_S3_KEY)s
ckan.files.storage.my_cloud.secret = %(CKAN_S3_SECRET)s
Only bucket option is required as all other parameters can be read from
environment variables:
bucket: name of the storage bucketkey: the AWS Access Keysecret: the AWS Secret Keyregion: the AWS Region of the bucket
ckan:azure_blob
Microsoft Azure Blob Storage adapter. Wraps file-keeper's implementation
Installation:
pip install 'ckanext-file-keeper-cloud[azure]'
Configuration example:
ckan.files.storage.my_cloud.type = ckan:azure_blob
ckan.files.storage.my_cloud.container_name = my_container
ckan.files.storage.my_cloud.account_name = %(CKAN_AZURE_ACCOUNT)s
ckan.files.storage.my_cloud.account_key = %(CKAN_AZURE_KEY)s
Recommended options:
container_name: name of the storage containeraccount_name: name of the Azure accountaccount_key: key for the Azure account
ckan:gcs
Google Cloud Storage adapter. Wraps file-keeper's implementation
Installation:
pip install 'ckanext-file-keeper-cloud[gcs]'
Configuration example:
ckan.files.storage.my_cloud.type = ckan:gcs
ckan.files.storage.my_cloud.bucket_name = my_bucket
ckan.files.storage.my_cloud.project_id = my-project
ckan.files.storage.my_cloud.credentials_file = /etc/ckan/default/google-cloud-credentials.json
Recommended options:
bucket_name: name of the storage bucketcredentials_files: path to the JSON with cloud credentialsproject_id: the project which the client acts on behalf of
ckan:libcloud
Apache Libcloud adapter. Wraps file-keeper's implementation
Installation:
pip install 'ckanext-file-keeper-cloud[libcloud]'
Configuration example:
ckan.files.storage.my_cloud.type = ckan:libcloud
ckan.files.storage.my_cloud.provider = S3
ckan.files.storage.my_cloud.container_name = my_bucket
ckan.files.storage.my_cloud.key = %(CKAN_CLOUD_KEY)s
ckan.files.storage.my_cloud.secret = %(CKAN_CLOUD_SECRET)s
Requires following options:
provider: one of Apache Libcloud providerskey: API key or usernamesecret: Secret passwordcontainer_name: Name of the container/bucket
Majority of providers do not support permanent links out of the box. But if the
container supports public anonymous access and all files are available at URL
https://<PROVIDER>/<CONTAINER>/<FILENAME>, this shared
https://<PROVIDER>/<CONTAINER> part can be specified as public_prefix
of the storage. In this case, CKAN will append file's location to the
configured public_prefix whenever it needs a permanent public link for the
file.
Files are uploaded to the root of container. To specify nested location for all
uploads, use path option.
Any other provider specific option can be added inside params option which
expects a valid JSON object
Use cloud storage for resource uploads
When configuring storage for resource, group, user or admin uploads in CKAN, keep in mind that CKAN expects correct permissions set on the container. For example, this is the generic configuration of the AWS S3 bucket for resource uploads:
ckan.files.storage.resources.type = ckan:s3
ckan.files.storage.resources.bucket = ckan-resources
CKAN uploads files to bucket and resource's URL points to standard endpoint for
resource downloads:
<site_url>/dataset/<id>/resource/<resource_id>/download/<filename>. When user
follows the URL, CKAN emits a redirect to a signed S3 URL:
http://ckan-resources.s3.us-east.amazonaws.com/131/d3c/42-a2b3-4f03-b85d-73024402d219?AWSAccessKeyId=AWS_S3KEY&Signature=ABC123&Expires=123123123. This
URL remains valid for the limited amount of time and when user downloads
resource from CKAN's UI, new signed link is created. When link is expired(it
remains valid for 60 seconds), S3 bucket returns 403 response on attempt to use
the link.
But user can remove payload from the link(everything after ?) and try using
plain URL:
http://ckan-resources.s3.us-east.amazonaws.com/131/d3c/42-a2b3-4f03-b85d-73024402d219. Here
configuration of the bucket becomes important:
- if it's a private bucket, user sees 403 error: resource is protected
- if it's a public bucket, user successfully downloads the file: file is not protected
Always use private buckets for resource storage.
This redirect to signed URL differs from the behavior of the ckan:fs storage
adapter, which sends file immediately from the download URL. This happens
because resource's download view checks whether user has permission to download
the file and then calls as_response method of the storage adapter. This
method returns Flask response which leads to the file content. ckan:fs
adapter builds a response that sends static file via flask. ckan:s3 builds a
signed redirect response to the bucket, because it's much more efficient that
proxifying S3 content through Flask. It's possible to create a new storage
adapter that extends ckan:s3 and implements as_response in different way,
if redirect for some reasons is not acceptable.
Use cloud storage for user and group uploads
User, group and admin downloads are different from resources. Let's start from the similar configuration of the storage for the user uploads(avatars):
ckan.files.storage.users.type = ckan:s3
ckan.files.storage.users.bucket = ckan-users
Upload an image to user profile. If you check user_show output after this,
you'll see that image_display_url field points to the cloud bucket, but it's
a plain, not signed URL, with no query parameters:
http://ckan-users.s3.us-east.amazonaws.com/image.png. If you open this URL
the following can happen:
- if bucket is public, you'll see the image
- if bucket is private, you'll see 403 response
Ideally, bucket for public files, such as user or group images, or site logos, must be public. In this way all possible problems are solved and no further actions required. But why image URL is different from resource URL?
Resources historically have separate endpoint where permissions are checked. It's safe to serve the content of the file by the end of this endpoint, because user without required permissions is rejected with 403 response in the beginning of the endpoint.
Public images do not have dedicated endpoint where redirect response can be produced. Instead, CKAN must produce links that will be shown on the page or in API output and client can use these URLs when required. Because these links can be stored for a long time before they are accessed, signed URLs will not work here. The upper bound on lifetime of the signed URL is 7 days which may be not sufficient if content harvested and displayed on a different portal.
Because of this, CKAN uses permanent_link method of the storage to build an
image link. This method makes an assumption that bucket is public and builds
URL which will work in this case. But if bucket is misconfigured and has no
public access, URL will be rejected.
If you cannot change visibility of the bucket, images still can be accessed, but it requires additional steps.
First, you need to prevent creation of permanent_link. Storage relies on
"capabilities" when different actions are performed, and permanent_link is
bound to capability LINK_PERMANENT. If storage has this capability(ckan:s3
has it), permanent_link will be called and return the link. If storage does
not have this capability, link will not be used.
It's possible to disable any existing capability of the storage using
ckan.files.storage.users.disabled_capabilities config option. Add it to the
configuration of the storage and set LINK_PERMANENT as its values. Here the
updated configuration:
ckan.files.storage.users.type = ckan:s3
ckan.files.storage.users.bucket = ckan-users
ckan.files.storage.users.disabled_capabilities = LINK_PERMANENT
If you check user_show output after this change, you'll see that now
image_display_url has value null. Now invalid URL is not used anymore, but
image is still not accessible. To solve it, we can enable public flag of the
storage - for any storage that has public flag, CKAN has a fallback URL for
serving files. Add ckan.files.storage.users.public = true to the
configuration of the storage and check output of user_show once again.
The image_display_url looks like
<site_url>/file/public-download/users/image.png. If you visit this URL,
you'll be redirected to S3 bucket, which must be familiar after experiments
with the resource. The main difference here is that there is no permission
check here. Any user, even anonymous can access this file. Endpoint that
handles this last URL checks only two things:
- specified file exists in the specified storage
- specified storage has
publicflag enabled
If both conditions are met, anyone can access the file from the
storage. Enabling public on storage literally makes it public in terms of
read access, so make sure you mean it when you use it. It always better to use
the initial setup, without disabled_capabilities and public, and configure
bucket directly, instead of solving misconfiguration via storage settings on
CKAN side. But if bucket cannot be reconfigured, or you want to publicly serve
content from the single folder on otherwise private bucket, this solution can do the trick.
Developer installation
To install ckanext-file-keeper-cloud for development, activate your CKAN virtualenv and do:
git clone https://github.com/ckan/ckanext-file-keeper-cloud.git
cd ckanext-file-keeper-cloud
pip install -e '.[all]'
License
Metadata
Release files for ckanext-file-keeper-cloud 0.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ckanext_file_keeper_cloud-0.0.4.tar.gz | 28.5 kB | With provenance | Details for ckanext_file_keeper_cloud-0.0.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ckanext_file_keeper_cloud-0.0.4-py3-none-any.whl | Python 3 | none | any | Details for ckanext_file_keeper_cloud-0.0.4-py3-none-any.whl |
Total release size: 54.6 kB
Release files / ckanext_file_keeper_cloud-0.0.4.tar.gz
| Download URL | ckanext_file_keeper_cloud-0.0.4.tar.gz |
|---|---|
| Size | 28.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d499e45d4b0d6a7f8ffd58d4d2c401e87dcea0a02384e66bf6368a94eb04d9f3
|
|
BLAKE2b-256 checksum How to use checksums |
3a2503f8d0db9067955acefbaf9ed6176cfc28b5edbed6a3104427692186e557
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logRelease files / ckanext_file_keeper_cloud-0.0.4-py3-none-any.whl
| Download URL | ckanext_file_keeper_cloud-0.0.4-py3-none-any.whl |
|---|---|
| Size | 26.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7bbb138d06f38c1c4d965c5ae8ab912061d4f320643c3f30db5115b2312ba57b
|
|
BLAKE2b-256 checksum How to use checksums |
206bc67c035020cef78c4f58e4db0b723abf12c0fd1e5caf437cd16542347034
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logSecurity for ckanext-file-keeper-cloud 0.0.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for ckanext_file_keeper_cloud-0.0.4.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 5, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.