Skip to main content

dserver Notification Plugin

dtool pypi tag test zenodo

Features

Introduction

dtool is a command line tool for packaging data and metadata into a dataset. A dtool dataset manages data and metadata without the need for a central database.

However, if one has to manage more than a hundred datasets it can be helpful to have the datasets’ metadata stored in a central server to enable one to quickly find datasets of interest.

dservercore provides a web API for registering datasets’ metadata and provides functionality to lookup, list and search for datasets.

This plugin enables the dserver to listen to notifications for the registration and deregistration of datasets.

Installation

Install the dtool lookup server dependency graph plugin

$ pip install dserver-notification-plugin

Setup and configuration

Configure plugin behavior

The plugin needs to know how to convert a bucket name into a base URI. The environment variable DSERVER_NOTIFY_BUCKET_TO_BASE_URI is used to specify that conversion, e.g.:

DSERVER_NOTIFY_BUCKET_TO_BASE_URI={"bucket": "ecs://bucket"}

It is also advisable to limit access to the notification listener to a certain IP range. Use:

DSERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.0.0/16

to specify the allowed remote network. To specify a single IP, use:

DSERVER_NOTIFY_ALLOW_ACCESS_FROM=192.168.1.1/32

Configure webhook in minio

The Publish Events to Webhook minio docs walks through the configuration for sending S3 event notifications to a webhook. Assuming a dserver with this plugin activated running at http://dserver:5000, and your minio instance with a bucket test-bucket at https://s3server:9000, use

# mc: minio client
mc config host add s3server http://s3server:9000 {admin_user} {admin_password}

# Note that the endpoint must be reachable when configuring, otherwise minio will reject
mc admin config set s3server/ notify_webhook:dtool  endpoint="http://dserver:5000/webhook/notify"
mc admin service restart s3server  # restart is necessary

# Activate the actual notifications
mc event add s3server/test-bucket arn:minio:sqs::testbucket:dtool --event "put,delete"

to configure a webhook endpoint identified by dtool and activate put and delete event notification. Choose the parameters for --event "put,delete" from minio’s Supported Bucket Evenets.

Note that minio is very strict on whom they talk to. If your dserver communicates via https, make sure that the server certificate uses SANs and that the signing authority’s root certificate is available to minio. See Install Certificates from Third-party CAs in the minio docs. Also assure all services are reachable by valid hostnames. Within a containerized environment such as launched with docker-compose , host names containing underscores _ may occur, but minio refuses to speak with such.

Configure webhook in NetApp StorageGRID

NetAPP StorageGRID is able to submit S3 event notifications when configured to communicate with an SNS endpoint, refer to the according sections of the NetApp StorageGRID docs for creating service endpoints and configuring event notifications.

In short, create an endpoint http://dserver:5000/webhook/notify with a suitable URN, i.e. urn:dserver:sns:region:notify:all, where you may pick all fields freely except urn and sns.

Next, enable event notifications for the desired bucket, i.e. for object creation events with a policy snippet like this:

<NotificationConfiguration>
  <TopicConfiguration>
    <Id>Object created</Id>
    <Topic>urn:dserver:sns:region:notify:all</Topic>
    <Event>s3:ObjectCreated:*</Event>
  </TopicConfiguration>
</NotificationConfiguration>

Testing

Launch a minimal mongodb instance with

$ cd tests/container && docker-compose up -d

and run tests from within repository root using

pytest --log-cli-level=DEBUG

Refer to .github/workflows/test.yml for the recommended testing environment.

Metadata

Release files for dserver-notification-plugin 0.4.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dserver-notification-plugin 0.4.3
File Size Uploaded
dserver_notification_plugin-0.4.3.tar.gz 49.1 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for dserver-notification-plugin 0.4.3
File Interpreter ABI Platform
dserver_notification_plugin-0.4.3-py3-none-any.whl Python 3 none any Details

Total release size: 62.6 kB

Release files / dserver_notification_plugin-0.4.3.tar.gz

Download URL dserver_notification_plugin-0.4.3.tar.gz
Size 49.1 kB
Tags Source
SHA-256 checksum
How to use checksums
c4a7e728be8fe29243aa41fe03c3061b60974b5ad467fd682275d7b07f1d2ff7
BLAKE2b-256 checksum
How to use checksums
3800bd6cdfc99d8f8bd0d237fddb2e6f1de369d290523f53be2d17e860ff1fc0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/5.1.1 CPython/3.12.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2024.

Transparency log

Release files / dserver_notification_plugin-0.4.3-py3-none-any.whl

Download URL dserver_notification_plugin-0.4.3-py3-none-any.whl
Size 13.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
dbfa7347f45c82a9bee35fe02499c2bfba35da777198ef2765cf94f399908ec1
BLAKE2b-256 checksum
How to use checksums
c5f588943eae325ffb5060717cdaf93cca4cfe96443f99881be8bce1344f828b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/5.1.1 CPython/3.12.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 20, 2024.

Transparency log

Release history Release notifications | RSS feed

This release

0.4.3 This release

2 release files

0.4.2

2 release files

0.4.1

2 release files

0.4.0

2 release files

0.3.0

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page