psqlgml 0.2.4

Summary

Sample data generation is a common step used for testing and verifying new and existing features that make use of the data commons dictionary. Without validation tools, this step can be super hard and prone to errors. This project aims to provide tooling that helps with generating and visualizing sample data. It is dictionary agnostic, so should work for any given gdc compatible dictionary.

Sample data graphs are represented using a customized GraphML format which can be represented in either json or yaml files. This projects provides tools for creating this schema based on selected dictionary and validating data that is targeting this schema.

Goals

psqlgml aims to provide the following for projects that makes use of psqlgraph:

1. test data validation and visualization

2. test data schema that can be integrated with IDE’s for easier test data generation

3. randomized test data generation based on user requirements

4. provide data structures and functions for use in external projects

5. provide alternate implementation for loading dictionary with better type checking

Requirements

• Python3.6+

• graphviz (used for visualization)

Installation

from pypi

$ pip install psqlgml

Quick Start

# install
$ pip install psqlgml

# validate install
$ psqlgml --help

# generate internal schema to aid validation
$ psqlgml generate -v 2.4.0 -n test_dictionary

# validation
$ psqlgml validate --help

# visualize
$ psqlgml visualize --help

import psqlgml

# load the default dictionary
dictionary: psqlgml.Dictionary = psqlgml.load(version="2.3.0")

GML Schema

This is a customized GraphML format based on JSON schema. It allows graphs to be represented as a set of nodes and edges. The schema makes it possible to validate a sample data.

unique_field: node_id
nodes:
  - label: program
    node_id: p_1
    name: SM-KD
  - label: project
    node_id: pr_1
edges:
  - src: p_1
    dst: pr_1
    label: programs

This example creats two nodes Program and Project that are linked together using the node_id property. The name of the edge connecting them is programs

Schema Generation

psqlgml can be used to generate dictionary specific schemas using exposed command line scripts. By default, gdcdictionary is assumed but parameters can be updated to work with a different project.

Generate schema using version 2.4.0 of the gdcdictionary

psqlgml generate -v 2.4.0 -n gdcdictionary

The generated schema can be used for validating sample data. It can also be added to IDEs like PyCharm for intellisense while creating sample data.

Sample Data Validation

$ psqlgml validate -f sample.yaml --data-dir <resource dir> -d <dictionary name> -v <dictionary version>

The following validations are currently supported:

• JSON Schema Validation

• Duplicate Definition Validation

• Undefined Link Validation

• Association Validation