dtool-lookup-server-direct-mongo-plugin 0.2.0

Features

• Query datasets via mongo language

• Funnel datasets through aggregation pipelines

Introduction

dtool is a command line tool for packaging data and metadata into a dataset. A dtool dataset manages data and metadata without the need for a central database.

However, if one has to manage more than a hundred datasets it can be helpful to have the datasets’ metadata stored in a central server to enable one to quickly find datasets of interest.

The dtool-lookup-server provides a web API for registering datasets’ metadata and provides functionality to lookup, list and search for datasets.

This plugin allows to submit plain mongo queries and aggregation pipelines directly to the lookup server.

Configuration

Inform this plugin about the Mongo database to use by setting the environment variables:

export MONGO_URI="mongodb://localhost:27017/"
export MONGO_DB="dtool_lookup_server"
export MONGO_COLLECTION="metadata"

If the Mongo search and retrieve plugins are used, then you may use the same database, but must use a different collection.

Use

export ALLOW_DIRECT_QUERY=true export ALLOW_DIRECT_AGGREGATION=false

to enable or disable direct mongo query and aggregation on this plugin.

ATTENTION: While direct queries respect user-wise access rights to database entries on the lookup server level, there is no guarantee for aggregation pipelines to do so per design. Don not enable direct aggregation in a production environment.

Authentication

The dtool lookup server makes use of the authorized header to pass through the JSON web token for authorization. Below we create environment variables for the token and the header used in the following curl command samples:

$ TOKEN=$(flask user token test-user)
$ HEADER="Authorization: Bearer $TOKEN"

Refer to the core dcumentation of dtool-lookup-server for more information.

Direct query

To look for a sepcific field key2: 42 in a dataset’s README.yml (provided the file is properly YAML-formatted), use

$ curl -H “$HEADER” -H “Content-Type: application/json” -X POST

-d ‘{“query”: {“readme.key2”: 42}}’ http://localhost:5000/mongo/query

Response content:

[
  {
    "base_uri": "s3://test-bucket",
    "created_at": 1683797360.056,
    "creator_username": "jotelha",
    "dtoolcore_version": "3.18.2",
    "frozen_at": 1683797362.855,
    "name": "test_dataset_2",
    "number_of_items": 1,
    "size_in_bytes": 19347,
    "tags": [],
    "type": "dataset",
    "uri": "s3://test-bucket/26785c2a-e8f8-46bf-82a1-cec92dbdf28f",
    "uuid": "26785c2a-e8f8-46bf-82a1-cec92dbdf28f"
  }
]

Direct aggregation

The following example of an aggregation pipeline identifies and counts instances of the same dataset at different base URIs:

$ curl -H "$HEADER" -H "Content-Type: application/json" -X POST \
    -d '{"aggregation": [
            {
                "$sort": {"base_uri": 1}
            }, {
                "$group":  {
                    "_id": "$name",
                    "count": {"$sum": 1},
                    "available_at": {"$push": "$base_uri"}
                }
            }, {
                "$project": {
                    "name": "$_id",
                    "count": true,
                    "available_at": true,
                    "_id": false
                }
            }, {
                "$sort": {"name": 1}
            }
        ]
    }' http://localhost:5000/mongo/aggregate

Response content:

[
  {
    "available_at": [
      "s3://test-bucket"
    ],
    "count": 1,
    "name": "test_dataset_1"
  },
  {
    "available_at": [
      "s3://test-bucket",
      "smb://test-share"
    ],
    "count": 2,
    "name": "test_dataset_2"
  }
]

Testing

Running unit tests with pytest requires a healthy lookup server installation and the availability of required services such as databases. Please refer to the core dtool-lookup-server for setup instructions.