cidc-schemas
This repository contains formal definitions of the CIDC metadata model using json-schema syntax and vocabulary.
View documentation at https://nci-cidc.github.io/cidc-schemas/
Installation
To install the latest released version, run:
pip install nci-cidc-schemas
Development
Project Structure
cidc_schemas/- a python module for generating, validating, and reading manifest and assay templates.schemas/- json specifications defining the CIDC metadata model.templates/- schemas for generating and validating manifest and assay templates.assays/- schemas for defining assay data models.artifacts/- schemas for defining artifacts.
docs/- the most recent build of the data model documentation, along with templates and scripts for re-generating the documentation.template_examples/- example populated Excel files for template specifications inschemas/templates, and.csvs auto-generated from those.xlsxs that allow to transparently keep track of changes in them.tests/- tests for thecidc_schemasmodule.
Developer Setup
Install necessary dependencies.
pip install -r requirements.dev.txt
Install and configure pre-commit hooks.
pre-commit install
Updating dependencies
Use scripts/outdated_packages.py to see which packages have newer versions available and how old each new release is:
python scripts/outdated_packages.py
Only update packages whose latest version is at least 5 days old — this avoids pulling in releases that may be yanked or have undiscovered issues. The Age (days) column in the output indicates how long a release has been on PyPI.
Once you've identified packages to update, bump their versions in requirements.txt and requirements.dev.txt, then run the test suite to confirm nothing is broken.
Running tests
This repository has unit tests in the tests folder. After installing dependencies the tests can be run via the command
pytest tests
Template Versioning
When modifying an existing assay or analysis template (in cidc_schemas/schemas/templates/assays/ or cidc_schemas/schemas/templates/analyses/), you must increment the "version" field in its _template.json file using Semantic Versioning (e.g., "1.0.0" -> "1.1.0").
This ensures the template reflects that updates were made a pre-commit hook is configured to automatically verify this version bump before allowing your commit to pass.
Building documentation
To build documentation from schema updates, run the following commands:
python setup.py install # install helpers from the cidc_schemas library
python docs/generate_docs.py
This will output the generated html documents in docs/docs. If the updated docs are pushed up and merged into master, they will be viewable at https://nci-cidc.github.io/cidc-schemas/.
Using the Command-Line Interface
This project comes with a command-line interface for validating schemas and generating/validating assay and manifest templates.
Install the CLI
Clone the repository and cd into it
git clone git@github.com:NCI-CIDC/cidc-schemas.git
cd cidc-schemas
Install the cidc_schemas package (this adds the cidc_schemas CLI to your console)
python setup.py install
Run cidc_schemas --help to see available options.
If you're making changes to the module and want those changes to be reflected in the CLI without reinstalling the cidc_schemas module every time, run
python3 -m cidc_schemas.cli [args]
Creating a new assay or analysis type
In order to create a new assay type, your best bet is to just search for an existing assay and copy it.
Preferably, look at scrnaseq and copy exactly what it does. Make changes in the assay schema and template for your particular assay and/or analysis schema.
Once you update and update the version of this repo, update api-gae. You should only need to copy what scrnaseq did in api-gae in order for files to show up on the portal. Make sure to update the api-gae version. Update the api-gae version used in cloud-functions.
Finally, make sure to update the cli tool to include the new assay.
There are a lot of gotchas and hidden parsing going on behind the scenes. Listing them all would be hard, so the practical advice is to follow an existing working template.
Be sure to regenerate the docs after creating your schema, so the new schema is added to the reference docs.
Generate templates
Create a template for a given template configuration.
cidc_schemas generate_template -m templates/manifests/pbmc_template.json -o pbmc.xlsx
Validate filled-out templates
Check that a populated template file is valid with respect to a template specification.
cidc_schemas validate_template -m templates/manifests/pbmc_template.json -x template_examples/pbmc_template.xlsx
Validate JSON schemas
Check that a JSON schema conforms to the JSON Schema specifications.
cidc_schemas validate_schema -f shipping_core.json
1 maintainer
Metadata
1 maintainer
Release files for nci-cidc-schemas 0.30.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| nci_cidc_schemas-0.30.4.tar.gz | 2.7 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| nci_cidc_schemas-0.30.4-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 5.3 MB
Release files / nci_cidc_schemas-0.30.4.tar.gz
| Download URL | nci_cidc_schemas-0.30.4.tar.gz |
|---|---|
| Size | 2.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0de9b7604e4a7606dd85fb03f4fe327f5a51080421ded008f702f25a08c41536
|
|
BLAKE2b-256 checksum How to use checksums |
4cdfd16bcad68d4a79b4a4cbc443f079664546b42e2463c9758243f5a4ed4119
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / nci_cidc_schemas-0.30.4-py2.py3-none-any.whl
| Download URL | nci_cidc_schemas-0.30.4-py2.py3-none-any.whl |
|---|---|
| Size | 2.6 MB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
7ea65575bb61f2d7a0a17d62bf1bc37a2a9bc44f256e08bb480609e3b21dc499
|
|
BLAKE2b-256 checksum How to use checksums |
4ec4af14eb4432cba45b947f55764536324458529ce54aa1fdcd7b34b3c7944a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
