Skip to main content

DX API Client Library

Welcome to the DX API Client Library! This library provides a convenient Python interface to interact with the DX API, allowing you to manage datasets, installations, and perform various operations with ease.

Table of Contents

Features

  • Authenticate with the DX API using JWT tokens.
  • Manage installations and datasets.
  • Upload and download data to and from datasets.
  • Synchronous and asynchronous support.
  • Context managers for handling authentication scopes.

Installation

You can install the library using pip:

pip install mig-dx-api

Prerequisites

  • Python 3.10 or higher.
  • An application ID (app_id),a workspace key (workspace_key), and a corresponding private key in PEM format from the Console.
  • DX API access credentials.

Getting Started

Authentication

The library uses JWT tokens for authentication. You need to provide your app_id, workspace_key, and the path to your private key file when initializing the client.

Initialization

from mig_dx_api import DX

# Initialize the client
dx = DX(app_id='your_app_id', private_key_path='path/to/private_key.pem')

# OR
dx = DX(app_id='your_app_id',private_key='your_private_key')

Alternatively, you can set the environment variables DX_CONFIG_APP_ID, DX_CONFIG_WORKSPACE_KEY, and DX_CONFIG_PRIVATE_KEY_PATH:

export DX_CONFIG_APP_ID='your_app_id'
export DX_CONFIG_WORKSPACE_KEY='your_workspace_key'

export DX_CONFIG_PRIVATE_KEY_PATH='path/to/private_key.pem'
# OR
export DX_CONFIG_PRIVATE_KEY='your_private_key'

And initialize the client without arguments:

dx = DX()

Usage

Who Am I

Retrieve information about the authenticated user:

user_info = dx.whoami()
print(user_info)

Managing Installations

Listing Installations

installations = dx.get_installations()
for installation in installations:
    print(installation.name)

Accessing an Installation Context

Use the installation context to perform operations related to a specific installation:

# Find an installation by name or ID
installation = dx.installations.find(install_id=1)
# Get an installation by workspace_id
installation = dx.get_installations(workspace_id=1)[0]

if DX_CONFIG_WORKSPACE_KEY is set you don't need to pass in the workspace_key as a param on dx.installation

Use the installation context

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    # Perform operations within the context
    datasets = list(ctx.datasets)
    for dataset in datasets:
        print(dataset.name)

Or enter context with a lookup by name:

with dx.installation(install_id=1) as ctx:
    # Perform operations within the context
    datasets = list(ctx.datasets)
    for dataset in datasets:
        print(dataset.name)

Or enter context with a lookup by workspace_id:

with dx.installation(workspace_id=1) as ctx:
    # Perform operations within the context
    datasets = list(ctx.datasets)
    for dataset in datasets:
        print(dataset.name)

Managing Datasets

Listing Datasets

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    for dataset in ctx.datasets:
        print(dataset.name, dataset.id)

Accessing Dataset Operations

Dataset operations may be accessed by name if the dataset name is unique within the installation, or by dataset_id.

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(name='My Dataset')
with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(dataset_id='123adatasetid')
with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(dataset_id=UUID('123adatasetid'))

Creating a Dataset

from mig_dx_api import DatasetSchema, SchemaProperty

# Define the schema
schema = DatasetSchema(
    properties=[
        SchemaProperty(name='my_string', type='string', required=True),
        SchemaProperty(name='my_integer', type='integer', required=True),
        SchemaProperty(name='my_boolean', type='boolean', required=False),
    ],
    primary_key=['my_string']
)

# Create the dataset
with dx.installation(installation, workspace_key=workspace_key) as ctx:
    new_dataset = ctx.datasets.create(
        name='My Dataset',
        description='A test dataset',
        schema=schema.model_dump()  # this can also be defined as a dictionary
    )

Creating a Dataset with specific tags

# Get tag options
with dx.installation(installation, workspace_key=workspace_key) as ctx:
    tag_options = ctx.datasets.get_tags()
    tags = [tags['data'][0]['movementAppId'], tags['data'][1]['movementAppId']]
    new_dataset = ctx.datasets.create(
        name='My Tagged Dataset',
        description='A test tagged dataset',
        schema={
            "primaryKey": ["my_van_id"],
            "properties": [
                {"name": "my_van_id", "type": "string"},
                {"name": "first_name", "type": "string"},
                {"name": "last_name", "type": "string"},
            ],
        },
        tag_ids=[tags]
    )

Uploading Data to a Dataset

data = [
    {'my_string': 'string1', 'my_integer': 1, 'my_boolean': True},
    {'my_string': 'string2', 'my_integer': 2, 'my_boolean': False},
    {'my_string': 'string3', 'my_integer': 3, 'my_boolean': True},
]

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(name='My Dataset')
    dataset_ops.load(data, validate_records=True)  # validate_records=True will validate the records against the schema using Pydantic
    

Load defaults to csv for newline-delimited json (NDJSON), but can support optional content types of tsv or other json types if the type is passed in, e.g. dataset_ops.load(data, validate_records=True, content_type='json')

While the load default upload_mode is create, append and replace are also available to add to a dataset or replace a dataset entirely, respectively.

Dataset Jobs

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(name='My Dataset')
    # Job Id comes from load, load_from_file or load_from_url
    job_id = dataset_ops.load(data, validate_records=True)["jobId"]
    # Get dataset job
    job = ctx.datasets.get_dataset_job(job_id)
    # Get logs for dataset job
    job_logs = ctx.datasets.get_dataset_job_logs(job_id)
    # Get latest status for dataset job
     job_logs_current = ctx.datasets.get_current_dataset_logs(job_id)

Retrieving Records from a Dataset

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(name='My Dataset')
    records = dataset_ops.records()
    for record in records:
        print(record)

Deleting A Dataset

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.delete(dataset_id=my_dataset_id)

Asynchronous Usage

The library supports asynchronous operations using async/await.

import asyncio

async def main():
    dx = DX()
    async with dx.installation(installation, workspace_key=workspace_key) as ctx:
        async for dataset in ctx.datasets:
            print(dataset.name)

        dataset = await ctx.datasets.find(name='My Dataset')

        data = [
            {'my_string': 'string1', 'my_integer': 1, 'my_boolean': True},
            {'my_string': 'string2', 'my_integer': 2, 'my_boolean': False},
            {'my_string': 'string3', 'my_integer': 3, 'my_boolean': True},
        ]

        await dataset.load(data)

        async for record in dataset.records():
            print(record)

asyncio.run(main())

Examples

Example: Loading Data from a File

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(name='My Dataset')
    dataset_ops.load_from_file('data.csv')

TSV and JSON files are also supported. Uses file extension to determine file type.

Example: Uploading Data from a URL

with dx.installation(installation, workspace_key=workspace_key) as ctx:
    dataset_ops = ctx.datasets.find(name='My Dataset')
    dataset_ops.load_from_url('https://example.com/data.csv', mode="create")

TSV and JSON files are also supported. Uses file extension to determine file type.

License

MIT License. See License


Note: This README assumes that the package name is mig-dx-api and that the code is properly packaged and available for installation via pip. Adjust the instructions accordingly based on the actual package name and installation method.

Metadata

Release files for mig-dx-api 0.2.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mig-dx-api 0.2.12
File Size Uploaded Provenance Actions
mig_dx_api-0.2.12.tar.gz 65.1 kB Without provenance Details for mig_dx_api-0.2.12.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mig-dx-api 0.2.12
File Interpreter ABI Platform Actions
mig_dx_api-0.2.12-py3-none-any.whl Python 3 none any Details for mig_dx_api-0.2.12-py3-none-any.whl

Total release size: 80.9 kB

Release files / mig_dx_api-0.2.12.tar.gz

Download URL mig_dx_api-0.2.12.tar.gz
Size 65.1 kB
Tags Source
SHA-256 checksum
How to use checksums
16613b0e127663727aa4f0ff56591e7d7352bf15559c3ccc54a1de20adb954af
BLAKE2b-256 checksum
How to use checksums
786f1c54c8e1b1580f6cf4d0f93da12b41c844c0b8d79c15e4a9c5d7550b3f39
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / mig_dx_api-0.2.12-py3-none-any.whl

Download URL mig_dx_api-0.2.12-py3-none-any.whl
Size 15.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
28ac7ab81d818a4b10c4d21c58131bb2abd35ea317a3c736708b4b2f69dfeafe
BLAKE2b-256 checksum
How to use checksums
eda904b94ec2d1d59ee4b720f6969d8b666b9ac9f55f39fcd67def901bd79e40
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for mig-dx-api 0.2.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page