Skip to main content

Utils for Gen3 Commons management

Project description

gen3utils

Utils for Gen3 commons management

Install with pip

pip install gen3utils

manifest.json validation

Validate one or more manifest.json files:

gen3utils validate_manifest cdis-manifest/*/manifest.json

The validation settings can be updated by modifying this file.

etlMapping.yaml validation

Validate an etlMapping.yaml file against the dictionary URL specified in a manifest.json file:

gen3utils validate_etl_mapping etlMapping.yaml manifest.json

Portal Configuration (gitops.json) validation

Validate a gitops.json file against the dictionary URL specified in a manifest.json file and an etlMapping.yaml file. Adds a comment to a pull request listing all the errors encountered when validating against etlMapping.yaml

gen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json <username>/<repository> <pull request number>

To run without making a pull request comment

gen3utils validate_portal_config etlMapping.yaml manifest.json gitops.json

Comment on a PR with any deployment changes when updating manifest services

The command requires the name of the repository, the pull request number and a GITHUB_TOKEN environment variable containing a token with read and write access to the repository. It also comments a warning if a service is pinned on a branch.

pip install gen3utils
gen3utils post-deployment-changes <username>/<repository> <pull request number>

Log parser for CTDS log pipeline

pip install gen3utils
gen3utils s3log --help
gen3utils s3log [OPTIONS] BUCKET PREFIX SCRIPT

Run SCRIPT in Gen3 logs under S3 BUCKET:PREFIX. The SCRIPT should be importable defining a method like this:

def handle_row(obj, line):
    if 1 + 1 == 2:
        return line

For example, to process logs in bucket my-commons-logs at prefix my-logs with a gen3utils/script.py file:

pip install gen3utils
gen3utils s3log my-commons-logs my-logs gen3utils.script

Running tests locally

poetry install -vv
poetry run pytest -vv ./tests

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gen3utils-0.10.2.tar.gz (26.0 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page