Skip to main content

No project description provided

Project description

dagster-mssql-bcp

Unit tests

ODBC is slow 🐢 bcp is fast! 🐰

This is a custom dagster IO manager for loading data into SQL Server using the bcp utility.

What you need to run it

Pypi

PyPI

pip install dagster-mssql-bcp

BCP Utility

The bcp utility must be installed on the machine that is running the dagster pipeline.

See Microsoft's documentation for more information.

Ideally you should place this on your PATH, but you can specify in the IO configuration where it is located.

ODBC Drivers

You need the ODBC drivers installed on the machine that is running the dagster pipeline.

See Microsoft's documentation for more information.

Permissions

The user running the dagster pipeline must have the necessary permissions to load data into the SQL Server database.

  • CREATE SCHEMA
  • CREATE/ALTER TABLES

Basic Usage

Polars

Polars processes as a LazyFrame. Either a DataFrame or LazyFrame can be provided as an output of your asset before its cast automatically to lazy

from dagster import asset, Definitions
from dagster_mssql_bcp import PolarsBCPIOManager
import polars as pl

io_manager = PolarsBCPIOManager(
    host="my_mssql_server",
    database="my_database",
    user="username",
    password="password",
    query_props={
        "TrustServerCertificate": "yes",
    },
    bcp_arguments={"-u": ""},
    bcp_path="/opt/mssql-tools18/bin/bcp",
)

@asset(
    metadata={
        "asset_schema": [
            {"name": "id", "type": "INT"},
        ],
        "schema": "my_schema",
    }
)
def my_polars_asset(context):
    return pl.DataFrame({"id": [1, 2, 3]})


@asset(
    metadata={
        "asset_schema": [
            {"name": "id", "type": "INT"},
        ],
        "schema": "my_schema",
    }
)
def my_polars_asset_lazy(context):
    return pl.LazyFrame({"id": [1, 2, 3]})

defs = Definitions(
    assets=[my_polars_asset, my_polars_asset_lazy],
    io_managers={
        "io_manager": io_manager,
    },
)

Pandas

from dagster import asset, Definitions
from dagster_mssql_bcp import PandasBCPIOManager
import pandas as pd

io_manager = PandasBCPIOManager(
    host="my_mssql_server",
    database="my_database",
    user="username",
    password="password",
    query_props={
        "TrustServerCertificate": "yes",
    },
    bcp_arguments={"-u": ""},
    bcp_path="/opt/mssql-tools18/bin/bcp",
)

@asset(
    metadata={
        "asset_schema": [
            {"name": "id", "type": "INT"},
        ],
        "schema": "my_schema",
    }
)
def my_pandas_asset(context):
    return pd.DataFrame({"id": [1, 2, 3]})


defs = Definitions(
    assets=[my_pandas_asset],
    io_managers={
        "io_manager": io_manager,
    },
)

The asset schema defines your table structure and your asset returns your data to load.

Docs

For more details see assets doc, io manager doc, and for how its implemented, the dev doc.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_mssql_bcp-0.0.11.tar.gz (40.7 kB view details)

Uploaded Source

Built Distribution

dagster_mssql_bcp-0.0.11-py3-none-any.whl (28.1 kB view details)

Uploaded Python 3

File details

Details for the file dagster_mssql_bcp-0.0.11.tar.gz.

File metadata

  • Download URL: dagster_mssql_bcp-0.0.11.tar.gz
  • Upload date:
  • Size: 40.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.1.1 CPython/3.12.7

File hashes

Hashes for dagster_mssql_bcp-0.0.11.tar.gz
Algorithm Hash digest
SHA256 49785f72a756f7bd91c261664a0091bb12a5a9356d2c196da86506673017e768
MD5 1c6d74255d01582fd0b210f9707ddaef
BLAKE2b-256 4eac49b2d267a874cf3f83a56f5bcc6e7d914aaace13fa12a00f77234963af9d

See more details on using hashes here.

Provenance

The following attestation bundles were made for dagster_mssql_bcp-0.0.11.tar.gz:

Publisher: python-publish.yml on cody-scott/dagster-mssql-bcp

Attestations:

File details

Details for the file dagster_mssql_bcp-0.0.11-py3-none-any.whl.

File metadata

File hashes

Hashes for dagster_mssql_bcp-0.0.11-py3-none-any.whl
Algorithm Hash digest
SHA256 6ab187ddffef9363035dc387f2106ec8256c67e26fef849dd12c9d6a8b79bfcb
MD5 c52d817d664ea82b0080f9aee14b3181
BLAKE2b-256 5e78c87ab64f6e81bd001a4596b7f8017056601948568d538eba38a48583573c

See more details on using hashes here.

Provenance

The following attestation bundles were made for dagster_mssql_bcp-0.0.11-py3-none-any.whl:

Publisher: python-publish.yml on cody-scott/dagster-mssql-bcp

Attestations:

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page