ministryofjustice-data-platform-catalogue 0.2.1

Library to integrate the MoJ data platform with the catalogue component.

Data platform catalogue

This library is part of the Ministry of Justice data platform.

It provides functionality to publish object metadata to the OpenMetadata data catalogue so that data products are discoverable.

How to install

To install the package using pip, run:

pip install ministryofjustice-data-platform-catalogue

Topology

• Each moj data product is mapped to a database in the OpenMetadata catalogue
• We populate the schema level in openmetdata with a generic entry of Tables
• Each table is mapped to a table in openmetadata

Example usage

from data_platform_catalogue import (
  CatalogueClient, CatalogueMetadata,
  DataProductMetadata, TableMetadata,
  CatalogueError
)

client = CatalogueClient(
    jwt_token="***",
    api_uri="https://catalogue.apps-tools.development.data-platform.service.justice.gov.uk/api"
)

assert client.is_healthy()


data_product = DataProductMetadata(
    name = "my_data_product",
    description = "bla bla",
    version = "v1.0.0",
    owner = "7804c127-d677-4900-82f9-83517e51bb94",
    email = "[e-mail]",
    retention_period_in_days = 365,
    domain = "legal-aid",
    dpia_required = False
)

data_product_schema = DataProductMetadata(
    name = "Tables",
    description = "All the tables contained within my_data_product",
    version = "v1.0.0",
    owner = "7804c127-d677-4900-82f9-83517e51bb94",
    email = "[e-mail]",
    retention_period_in_days = 365,
    domain = "legal-aid",
    dpia_required = False
)

table = TableMetadata(
  name = "my_table",
  description = "bla bla",
  column_types = {"foo": "string", "bar": "int"},
  retention_period_in_days = 365
)

try:
  service_fqn = client.create_or_update_database_service(name="data_platform")
  database_fqn = client.create_or_update_database(metadata=data_product, service_fqn=service_fqn)
  schema_fqn = client.create_or_update_schema(metadata=data_product_schema, database_fqn=database_fqn)
  table_fqn = client.create_or_update_table(metadata=table, schema_fqn=schema_fqn)
except CatalogueError:
  print("oh no")