Skip to main content

DataYoga Core

Introduction

datayoga-core is the transformation engine used in DataYoga, a framework for building and generating data pipelines.

Installation

datayoga-core requires Python 3.10 or newer.

pip install datayoga-core

Quick Start

This demonstrates how to transform data using a DataYoga job.

Create a Job

Use this example.dy.yaml:

steps:
  - uses: add_field
    with:
      fields:
        - field: full_name
          language: jmespath
          expression: concat([fname, ' ' , lname])
        - field: country
          language: sql
          expression: country_code || ' - ' || UPPER(country_name)
  - uses: rename_field
    with:
      fields:
        - from_field: fname
          to_field: first_name
        - from_field: lname
          to_field: last_name
  - uses: remove_field
    with:
      fields:
        - field: credit_card
        - field: country_name
        - field: country_code
  - uses: map
    with:
      expression:
        {
          first_name: first_name,
          last_name: last_name,
          greeting: "'Hello ' || CASE WHEN gender = 'F' THEN 'Ms.' WHEN gender = 'M' THEN 'Mr.' ELSE 'N/A' END || ' ' || full_name",
          country: country,
          full_name: full_name
        }
      language: sql

Transform Data Using datayoga-core

Use this code snippet to transform a data record using the job defined above. The transform method returns a tuple of processed, filtered, and rejected records:

import datayoga_core as dy
from datayoga_core.job import Job
from datayoga_core.result import Result, Status
from datayoga_core.utils import read_yaml

job_settings = read_yaml("example.dy.yaml")
job = dy.compile(job_settings)

assert job.transform([{"fname": "jane", "lname": "smith", "country_code": 1, "country_name": "usa", "credit_card": "1234-5678-0000-9999", "gender": "F"}]).processed == [
  Result(status=Status.SUCCESS, payload={"first_name": "jane", "last_name": "smith", "country": "1 - USA", "full_name": "jane smith", "greeting": "Hello Ms. jane smith"})]

The job can also be provided as a parsed json inline:

import datayoga_core as dy
from datayoga_core.job import Job
from datayoga_core.result import Result, Status
import yaml
import textwrap

job_settings = textwrap.dedent("""
  steps:
    - uses: add_field
      with:
        fields:
          - field: full_name
            language: jmespath
            expression: concat([fname, ' ' , lname])
          - field: country
            language: sql
            expression: country_code || ' - ' || UPPER(country_name)
    - uses: rename_field
      with:
        fields:
          - from_field: fname
            to_field: first_name
          - from_field: lname
            to_field: last_name
    - uses: remove_field
      with:
        fields:
          - field: credit_card
          - field: country_name
          - field: country_code
    - uses: map
      with:
        expression:
          {
            first_name: first_name,
            last_name: last_name,
            greeting: "'Hello ' || CASE WHEN gender = 'F' THEN 'Ms.' WHEN gender = 'M' THEN 'Mr.' ELSE 'N/A' END || ' ' || full_name",
            country: country,
            full_name: full_name
          }
        language: sql
""")
job = dy.compile(yaml.safe_load(job_settings))

assert job.transform([{"fname": "jane", "lname": "smith", "country_code": 1, "country_name": "usa", "credit_card": "1234-5678-0000-9999", "gender": "F"}]).processed == [
  Result(status=Status.SUCCESS, payload={"first_name": "jane", "last_name": "smith", "country": "1 - USA", "full_name": "jane smith", "greeting": "Hello Ms. jane smith"})]

As can be seen, the record has been transformed based on the job:

  • fname field renamed to first_name.
  • lname field renamed to last_name.
  • country field added based on an SQL expression.
  • full_name field added based on a JMESPath expression.
  • greeting field added based on an SQL expression.

Examples

  • Add a new field country out of an SQL expression that concatenates country_code and country_name fields after upper case the later:

    uses: add_field
    with:
      field: country
      language: sql
      expression: country_code || ' - ' || UPPER(country_name)
    
  • Rename fname field to first_name and lname field to last_name:

    uses: rename_field
    with:
      fields:
        - from_field: fname
          to_field: first_name
        - from_field: lname
          to_field: last_name
    
  • Remove credit_card field:

    uses: remove_field
    with:
      field: credit_card
    

For a full list of supported block types see reference.

Expression Language

DataYoga supports both SQL and JMESPath expressions. JMESPath are especially useful to handle nested JSON data, while SQL is more suited to flat row-like structures.

For more information about custom functions and supported expression language syntax see reference.

Metadata

Release files for datayoga-core 1.138.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datayoga-core 1.138.0
File Size Uploaded Provenance Actions
datayoga_core-1.138.0.tar.gz 62.0 kB Without provenance Details for datayoga_core-1.138.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datayoga-core 1.138.0
File Interpreter ABI Platform Actions
datayoga_core-1.138.0-py3-none-any.whl Python 3 none any Details for datayoga_core-1.138.0-py3-none-any.whl

Total release size: 168.7 kB

Release files / datayoga_core-1.138.0.tar.gz

Download URL datayoga_core-1.138.0.tar.gz
Size 62.0 kB
Tags Source
SHA-256 checksum
How to use checksums
f01d703002cf80a115fe8132cbcab89d5cec4e50e60b9f0ade62fc2914bb8a87
BLAKE2b-256 checksum
How to use checksums
a3bf5e65efceb6c331ce704c620ae82ca06f4c95c0c4b3299da99090633a38e7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Release files / datayoga_core-1.138.0-py3-none-any.whl

Download URL datayoga_core-1.138.0-py3-none-any.whl
Size 106.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0725e4a83135fd2f5959d3998ddf77c18b3876974bde3ee60c56caa636751f37
BLAKE2b-256 checksum
How to use checksums
b44001148f69a7f82a8fc8666b456a706f82676c50fd1ffec9839a0086d1b625
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.25

Security for datayoga-core 1.138.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.138.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page