Developing a Gen3 Data Dictionary

This repository aims to provide the documentation, learning materials, and software tools to facilitate the creation of a data model in Gen3.

Pre-Reading: Please become familiar with some core data modelling concepts before starting.

Using Gen3SchemaDev as a data modelling tool

• Quickstart
• Guide to creating your first dictionary
• Running a Gen3 dictionary repository
• Troubleshooting

generate does not overwrite your files

gen3schemadev generate never overwrites existing files by default. Generating into an empty folder works as it always has; generating into a folder that already contains files stops and lists exactly what it would have replaced, along with the ways forward.

This matters because a dictionary repository can be run in more than one way. Some treat the input_yaml as the source of truth and regenerate from it. Others generate once and then edit the Gen3 schemas directly, at which point the generated files are the dictionary. The tool cannot tell which you are doing, and guessing wrong destroys work.

• --input-driven — the input_yaml is the source of truth; regenerate everything, and fail if the folder holds a file the input cannot produce
• --only <node> — regenerate named nodes, leaving every other file untouched
• --check — report whether the folder still matches the input, write nothing, exit non-zero on drift. This is the CI gate
• --force — overwrite everything, discarding any hand edits

Nodes may also extend the packaged program, project and core_metadata_collection presets, adding properties while inheriting the node-level settings other Gen3 microservices depend on. See Running a Gen3 dictionary repository.

Null description placeholders

gen3schemadev validate warns about description: null placeholders anywhere in the dictionary (commonly in _definitions.yaml's enum definitions). The Gen3 metaschema requires descriptions to be strings; null placeholders cause "No Description" in the data-dictionary viewer and metaschema validation failures that surface on resolved node schemas, far from the offending definition. Remove the null description keys to resolve the warning.

Deep dive into Gen3 Data Modelling

Special Thanks to Marion Shadbolt for providing the source material

1. Gen3 Dictionary Overview
2. Gen3 Schema Explained
   1. Descriptors
   2. Links
   3. Properties
3. Example of a Gen3 Schema yaml
4. Handy tips
5. FAIR Concepts for Data Modelling

For Developers

Installation and testing

# To install
pip install poetry
poetry install
source $(poetry env info --path)/bin/activate
gen3schemadev --version

# To run tests
poetry run pytest

Contributing

Gen3SchemaDev is an open source project, and we highly encourage any contributions and PRs. Specifically we need the community to help with the following:

1. Keeping the gen3 metaschema up to date
2. Adding business rule logic to the rule validator module.
   1. For example, a node with the category data_file should have a collection of required data file properties such as md5sum, filesize, etc.

For PRs, please follow the contributing guidelines.

License

Apache 2.0