dagster-weaviate

A dagster module that provides integration with Weaviate (both Cloud and Self-Hosted Weaviate instances).

Installation

The dagster_weaviate module is available as a PyPI package - install with your preferred python environment manager (We recommend uv).

source .venv/bin/activate
uv pip install dagster-weaviate

Example Usage (Local Weaviate Instance)

(Based on the Weaviate Quickstart Guide (Local))

from dagster import Definitions, asset
from dagster_weaviate import WeaviateResource, LocalConfig

@asset
def my_table(weaviate: WeaviateResource):
    with weaviate.get_client() as weaviate_client:
        questions = weaviate_client.collections.get("Question")
        questions.query.near_text(query="biology", limit=2)

defs = Definitions(
    assets=[my_table],
    resources={
        "weaviate": WeaviateResource(
            connection_config=LocalConfig(
                host="192.168.0.10",
                port=8080,
            )
        ),
    },
)

Example Usage (Weaviate Cloud Instance)

Based on the Weaviate Cloud Quickstart Guide

from dagster import Definitions, asset
from dagster_weaviate import WeaviateResource, CloudConfig

@asset
def my_table(weaviate: WeaviateResource):
    with weaviate.get_client() as weaviate_client:
        questions = weaviate_client.collections.get("Question")
        questions.query.near_text(query="biology", limit=2)

defs = Definitions(
    assets=[my_table],
    resources={
        "weaviate": WeaviateResource(
            connection_config=CloudConfig(
                cluster_url=wcd_url
            ),
            auth_credentials={
                "api_key": wcd_apikey
            },
            headers={
                "X-Cohere-Api-Key": cohere_apikey,
            }
        ),
    },
)

Development

The Makefile provides the tools required to test and lint your local installation

make test
make ruff
make check