Open Data Integration Nomenclature (ODIN) is Crux’s standard for declarative data delivery

These details have not been verified by PyPI

Project description

Odin logo

Crux-Odin Library

Open Data Integration Nomenclature (ODIN) is Crux’s standard for declarative data delivery. ODIN provides a nomenclature for delivery that incentivizes industry-standard GitOps practices. ODIN specs are inherently abstracted from their underlying control planes and workflow frameworks, but work with the Crux External Data Platform.

Development

This project depends on pyenv for python version management and Poetry for package management and configuration. For instruction on local machine setup please refer to the below resources:

Pyenv Install Poetry Install

Usage

Right now we store crux-odin in the Google Artifact Registry. This will be changed when we promote to Pypi. For now, make sure you have a file called ~/.pypirc in your home directory with the following content:

[distutils]
index-servers = crux-python

[crux-python]
repository: https://us-python.pkg.dev/crux-ci/crux-python/

To install with the various tools:

pipenv

Add the following section to your Pipfile:

[[source]]
name = "google"
url = "https://us-python.pkg.dev/crux-ci/crux-python/simple"
verify_ssl = true

[pipenv]
disable_pip_input = false

and add

keyring = "*"
"keyrings.google-artifactregistry-auth" = "*"
crux-odin = "*"

under the [packages] section. Then do a pipenv sync --dev.

pip

pip install crux-odin==1.5.7 --index-url https://us-python.pkg.dev/crux-ci/crux-python/simple

poetry

Set up the poetry authentication with

poetry config http-basic.google oauth2accesstoken $(gcloud auth print-access-token)

Add a gcp entry to your pyproject.toml:

poetry source add --priority=supplemental gcp https://us-python.pkg.dev/crux-ci/crux-python/simple/

and also add the crux-odin dependency with

poetry add crux-odin=1.5.7 --source=gcp

to edit the pyproject.toml. Then do a poetry install.

For all installation methods, you can always uninstall crux-odin with pip uninstall crux-odin.

Changing Your Code

from crux_odin.dict_utils import yaml_file_to_dict
from crux_odin.dataclass import create_workflow

workflow = create_workflow(yaml_file_to_dict("file.yaml"))  # Version of Workflow gotten from YAML file

Validating YAML

from crux_odin.validate_yaml import validate_yaml

validate_yaml('file.yaml')

Manage Versions

To update a version

create a branch

git checkout -b release/[VERSION]

bump version

poe version_check # shows output without bumping version
poe version_bump # to bump version default
poe version_bump_patch # to increment patch version
poe version_bump_minor # to increment minor version
poe version_bump_major # to increment major version

create PR and merge in main branch
push git tag

git push origin [VERSION]

Reference: https://commitizen-tools.github.io/commitizen/commands/bump/

Supported Versions

See this page for a description of the YAML fields.

V1.0.0 - Crux's proprietary PDK framework

Supported information:

ID (airflow specific)
Connection info + extraction info
Normalizer spec
Schema history + schema validations
Context / Environment Variables

id: sample_id
run_uber_step: true

global:
  global:
    encoding: ascii
    timedelta:
      days: -1
    schema_def:
      na_values: [ "", " " ]
    crux_api_conf: ${SAMPLE_ID_API}
    endpoint: ${API_HOST}
  extract:
    action_class: pipeline.crux_pdk.actions.extract.extractor.ShortCircuitExtractor
    connection_lib: pipeline.custom_libs.sample.connector
    fetch_method: fetch_directory
    remote_path: /pub/sparx/
    connection:
      type: SAMPLE_ID_CONNECTOR
      conf: ${CRUX_SPARTA_SFTP}
      zendesk_conf:
        wait_time: 60
        payload:
          organization_id: 123123123123
          role: end-user
          ticket_restriction: organization
          skip_verify_email: true

pipelines:
  - id: sample_id
    global:
      global:
        supplier_implied_date_regex: active_users_(?P<YYYY>\d{4})(?P<MM>\d{2})(?P<DD>\d{2})
        provenance_file_patterns:
          origin_patterns:
            - active_ts_users_(?P<YYYY>\d{4})(?P<MM>\d{2})(?P<DD>\d{2})
          return_patterns:
            - active_ts_users_(?P<YYYY>\d{4})(?P<MM>\d{2})(?P<DD>\d{2})
    steps:
      - id: extract
        category: short_circuit
        conf:
          file_patterns:
            - active_users_{FD_YYYY}{FD_MM}{FD_DD}\.csv

Note: the outside global is inherited by the pipelines, the 'inside' global is inherited by the steps. The IDs have to match, extract above matches - id: extract

V1.1.0 - True Declarative Dataset

This is the first version of the spec that replaces the .py DAG files with full declarative syntax in YAML.