Dagster Component Templates

A community library of 949 reusable Dagster component templates covering data ingestion, transformations, ML / analytics, AI / LLM enrichment, orchestration, infrastructure provisioning, reverse ETL, observability, sensors, asset checks, and enterprise tool integrations — all configurable via YAML with no Python required.

Field naming conventions across the registry are documented in FIELD_CONVENTIONS.md.

What are Dagster Components?

Components are reusable, YAML-configurable building blocks that package common data engineering patterns. Drop one into your project's defs/ folder and configure it in YAML — no boilerplate Python needed.

# defs/components/sqs_ingest.yaml
type: dagster_component_templates.SQSToDatabaseAssetComponent
attributes:
  asset_name: raw_events
  queue_url_env_var: SQS_QUEUE_URL
  database_url_env_var: DATABASE_URL
  table_name: raw_events
  partition_type: daily

Component Library

Category Count What's in it
transformation 125 Pandas-style transforms (filter, join, union, datetime, regex, etc.)
ai 115 LLMs, vision, embeddings, vector stores, document extraction
ingestion 107 Source-to-destination data movement
resource 95 Connection-handle resources for SaaS / data platforms
analytics 92 ML models, scoring, segmentation, forecasting, geo, A/B testing
sink 69 DataFrame writers (CSV, Parquet, warehouses, NoSQL)
sensor 66 Polling sensors that emit RunRequests
source 51 Read-only data sources
integration 47 Multi-asset wrappers for cloud platforms
infrastructure 38 IaC / provisioning + lineage anchors
io_manager 33 IO managers for the major warehouses + lakes
jobs 32 Op-job components (cleanup, triggers, exports, heartbeats)
external 32 Declare-only external assets (warehouse tables, S3 objects)
observation 21 Health-check sensors emitting AssetObservations
check 17 Asset-check components (Great Expectations, Soda, etc.)
job 5 Additional job-level wrappers
data_warehouse 3 Warehouse-native asset primitives
dbt 1 dbt-project wrapper
total 949

By Dagster primitive (produces field)

Every manifest entry declares which Dagster primitive(s) it emits when loaded — machine-readable metadata for tools that need to route users to the right authoring surface ("Add schedule" from the Schedules tab, "Add asset" from Graph view, etc.).

produces Count Meaning
asset 634 Single @asset (or one asset key per instance)
asset_check 218 @asset_check — many components ship free schema-drift checks via build_column_schema_change_checks
resource 138 Connection / auth handle passed to downstream assets
sensor 101 @sensor / polling RunRequest emitter
multi_asset 43 @multi_asset OR workspace-shape (many assets per config, e.g. snowflake_workspace, stripe_workspace)
job 41 @job / define_asset_job(...)
schedule 40 @schedule / ScheduleDefinition
io_manager — Component wraps an IOManagerDefinition (0 today — most io_managers live under resource or asset)

Filter with dagster-component search "" --produces schedule to list every scheduling component. See CONTRIBUTING.md#manifest-fields for the full enum and semantics.

Assets (90 analytics + 73 ai + 59 transformation + 49 ingestion + 23 sink + 21 external + 19 infrastructure + 18 source + 18 integration + 7 check = 377)

Ingestion — cloud storage s3_to_database_asset · gcs_to_database_asset · adls_to_database_asset

Ingestion — messaging & streaming kafka_to_database_asset · sqs_to_database_asset · kinesis_to_database_asset · eventhubs_to_database_asset · servicebus_to_database_asset · rabbitmq_to_database_asset · pubsub_to_database_asset · redis_streams_to_database_asset · nats_to_database_asset · pulsar_to_database_asset · mqtt_to_database_asset

Ingestion — files & databases sftp_to_database_asset · sql_to_database_asset · csv_file_ingestion · rest_api_fetcher · openapi_asset · graphql_asset

AI / LLM enrichment litellm_inference_asset · ollama_inference_asset · langchain_chain_asset · llm_prompt_executor · llm_chain_executor · document_summarizer · entity_extractor · embeddings_generator · moderation_scorer · anthropic_llm · conversation_memory · snowflake_cortex_asset

dbt dbt_docs_enriched_project — extends DbtProjectComponent with exposures, metrics, semantic models, contracts, source freshness, and clickable dbt docs links on every asset

Enterprise orchestration coalesce_run_asset · abinitio_run_asset · matillion_run_asset · rivery_run_asset · precisely_run_asset · step_functions_asset · dataiku_asset · autosys_asset

Infrastructure as Code (provision resources before pipeline runs) terraform_asset · terraform_cloud_asset · cloudformation_asset · ansible_asset · pulumi_asset · helm_deploy · aws_cdk_asset

Secrets & remote execution ssh_asset · hashicorp_vault

Notebooks & ML compute jupyter_notebook_asset · modal_asset

Vector stores pinecone_asset · pgvector_asset · chromadb_asset · elasticsearch_asset

Feature stores feast_asset · tecton_asset

ML experiment tracking wandb_asset

Data versioning lakefs_asset

Reverse ETL polytomic_asset

Schema discovery warehouse_schema_assets — introspects a warehouse at prepare time, creates one external AssetSpec per table with full column metadata

Analytics & ML — predictive decision_tree_model · random_forest_model · logistic_regression_model · linear_regression_model · naive_bayes_model · neural_network_model · gradient_boosting_model · svm · count_regression · gamma_regression · spline_model · survival_analysis · cross_validation · stepwise · model_coefficients · model_score · model_comparison · vif

Analytics & ML — clustering / dim. reduction k_means_clustering · spatial_cluster · pca · multidimensional_scaling · nearest_neighbors · append_cluster · k_centroids_diagnostics

Analytics — A/B testing ab_test_analysis · ab_treatments · ab_controls · ab_trend · test_of_means

Analytics — geospatial distance_calculator · bounding_box_filter · coordinate_transformer · point_in_polygon · spatial_join · create_points · buffer · make_grid · smooth

Analytics — time series time_series_generator · arima_forecast · ets_forecast · ts_filler · ts_compare · ts_forecast · ts_covariate_forecast · ts_model_factory

Analytics — customer / business anomaly_detection · customer_segmentation · customer_360 · customer_health_score · customer_journey_mapping · cohort_analysis · rfm_segmentation · funnel_analysis · lead_scoring · propensity_scoring · churn_prediction · ltv_prediction · subscription_metrics · revenue_attribution · multi_touch_attribution · campaign_performance · product_recommendations · product_usage_analytics · priority_scorer · market_basket_rules

Sensors (40)

Cloud storage s3_monitor · gcs_monitor · adls_monitor

Messaging & streaming kafka_monitor · sqs_monitor · kinesis_monitor · eventhubs_monitor · servicebus_monitor · rabbitmq_monitor · pubsub_monitor · redis_streams_monitor · nats_monitor · pulsar_monitor · mqtt_monitor

Files sftp_monitor · sql_monitor

Enterprise tools coalesce_job_sensor · abinitio_job_sensor · matillion_job_sensor · rivery_job_sensor · precisely_job_sensor

Notifications slack_notification · twilio_notification

SaaS event sensors github_event_sensor · gitlab_event_sensor · stripe_event_sensor · zendesk_ticket_sensor · jira_issue_sensor · pagerduty_incident_sensor · linear_issue_sensor · notion_database_sensor · servicenow_sensor

Generic / no-auth http_poll_sensor · rss_feed_sensor · filesystem_monitor

ML triggers mlflow_model_sensor

Asset Checks (7)

dq_check · great_expectations_check · soda_check · monte_carlo_check · sifflet_check · acceldata_check · freshness_check

Observations (20)

clickhouse_table_observation_sensor · snowflake_table_observation · bigquery_table_observation · postgres_table_observation · redshift_table_observation and more

External Assets (21)

external_clickhouse_table · external_snowflake_table · external_bigquery_table · external_postgres_table · external_s3_object · external_kafka_topic and more

Integrations (18)

aws_glue · aws_dms · aws_kinesis · aws_redshift · aws_sagemaker · azure_data_factory · azure_stream_analytics · azure_synapse · databricks_asset_bundle · databricks_workspace · google_bigquery · google_cloud_functions · google_cloud_run_jobs · google_dataflow · google_datastream · google_pubsub · google_vertex_ai · snowflake_workspace

Component Structure

Every component follows the same layout:

component_name/
├── component.py       # Dagster component class
├── example.yaml       # Working YAML configuration example
├── README.md          # Documentation and field reference
├── requirements.txt   # pip dependencies
└── schema.json        # Component registry metadata

StateBackedComponent

Several components that discover resources from external APIs use Dagster's StateBackedComponent pattern. The API call happens once at prepare time and is cached to disk — code-server reloads are instant with zero network calls.

Components using this pattern: coalesce_run_asset, azure_data_factory, aws_glue, databricks_workspace, openapi_asset, warehouse_schema_assets, step_functions_asset, dataiku_asset, polytomic_asset, autosys_asset, terraform_cloud_asset

To refresh the cached state after adding pipelines/jobs in the external system:

dg utils refresh-defs-state
# or simply restart: dagster dev

Asset Dependencies & Lineage

Every asset component supports a deps field for declaring upstream dependencies in the asset graph:

type: dagster_component_templates.LiteLLMInferenceAssetComponent
attributes:
  asset_name: enriched_tickets
  upstream_asset_key: raw_tickets   # loads data + draws lineage edge
  deps:                              # additional lineage-only edges
    - support_schema/tickets_raw
    - raw/other_table
  model: claude-3-5-sonnet-20241022
  prompt_template: "Classify: {body}"
  database_url_env_var: DATABASE_URL
  table_name: enriched_tickets

upstream_asset_key — for components that load upstream data (LiteLLM, Ollama, LangChain, SQL): draws a lineage edge and loads the asset value at runtime.

deps — for all components: draws additional lineage-only edges without loading data. Use this to express that an asset depends on another without consuming it directly.

Dependencies can also be wired externally via map_resolved_asset_specs() in definitions.py — the same approach used by Dagster Designer.

Partitioning

All ingestion assets support time-based partitioning out of the box:

attributes:
  partition_type: daily          # none | daily | weekly | monthly
  partition_start_date: "2024-01-01"
  table_name: events_{partition_key}   # {partition_key} is substituted at runtime

Per-asset config via post_processing:

Every asset-emitting component in this library supports post_processing: — a base-component YAML block that lets you attach any AssetSpec attribute (freshness_policy, automation_condition, tags, owners, kinds, group_name, description, code_version, partitions_def) to specific assets without editing the component's source or writing Python. The target: field accepts the full Dagster asset-selection language (hierarchical groups, is:external, kind:dbt, tag/kind selectors, boolean composition, and legacy fnmatch globs).

This is the recommended path for per-asset overrides. Reach for post_processing: before translation: templates (which only handle model-typed fields) or splitting components (which fights dagster-dbt's input tracking).

type: dagster_community_components.<AnyAssetEmittingComponent>
attributes: {...}

post_processing:
  assets:
    - target: 'group:"marts/*"'          # hierarchical group wildcard
      attributes:
        automation_condition: "{{ dg.AutomationCondition.eager() }}"

    - target: "kind:dbt and tag:tier=gold"
      attributes:
        freshness_policy:
          maximum_lag_minutes: 30

    - target: "*"
      attributes:
        owners: ["data-platform@company.com"]

Attributes always safe on every component: automation_condition, freshness_policy, owners, tags, kinds, group_name, description, code_version — pure metadata, no compute contract.

partitions_def requires a partition-aware compute path: 332 of 573 asset-emitting components read context.partition_key in their compute (safe). 241 do not — declaring partitions on those via post_processing will silently produce the same data every partition. See docs/partition_aware_components.md for the full list.

Canonical example: DbtDocsEnrichedProjectComponent — its class docstring shows the shape end-to-end, including reading meta.dagster.partitions_def directly from a dbt model's schema.yml (v0.10.49+).

Sensor → Asset Pairing

Most sensors are designed to trigger a companion ingestion asset. The sensor detects new data and fires a RunRequest with source info in run_config; the asset reads and ingests it.

Sensor Companion Asset
s3_monitor s3_to_database_asset
kafka_monitor kafka_to_database_asset
sqs_monitor sqs_to_database_asset
kinesis_monitor kinesis_to_database_asset
eventhubs_monitor eventhubs_to_database_asset
rabbitmq_monitor rabbitmq_to_database_asset
sftp_monitor sftp_to_database_asset
… … (all 15 pairs complete)

dbt Docs Enrichment

The dbt_docs_enriched_project component is a drop-in replacement for DbtProjectComponent that adds rich metadata from the dbt manifest to every asset in the Dagster UI:

type: dagster_component_templates.DbtDocsEnrichedProjectComponent
attributes:
  project: "{{ project_root }}/dbt_project"
  dbt_docs_url: "https://dbt-docs.internal.mycompany.com"
  include_exposures: true
  include_metrics: true
  include_semantic_models: true
  include_contracts: true
  include_source_freshness: true

Each dbt model asset then shows: a clickable link to the dbt docs page, downstream BI exposures, metrics, semantic models, contract status, and source freshness SLAs — all visible in the Dagster Asset Catalog. All include_* flags default to false — opt in to only what you need.

Quick Start

1. Install dagster

pip install dagster

2. Copy a component into your project

cp -r assets/s3_to_database_asset/ my_project/defs/components/

3. Configure in YAML

# defs/components/my_s3_ingest.yaml
type: dagster_component_templates.S3ToDatabaseAssetComponent
attributes:
  asset_name: raw_orders
  bucket_env_var: DATA_BUCKET
  database_url_env_var: DATABASE_URL
  table_name: raw_orders

4. Load in definitions.py

import dagster as dg
from pathlib import Path

defs = dg.load_from_defs_folder(project_root=Path(__file__).parent)

Contributing

See CONTRIBUTING.md for how to add new components.

License

MIT License

Project links

Data verified by PyPI on Aug 7, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.10.101)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ethomasii from gravatar.com ethomasii

Credits

Author: Eric Thomas

GitHub Statistics

Data verified by PyPI on Aug 7, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.10

Provides Extra

compute-log-managers otlp splunk

Tags

components dagster data-engineering etl

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dagster_community_components-0.10.64.tar.gz (4.9 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

dagster_community_components-0.10.64-py3-none-any.whl (8.8 MB view details)

Uploaded Python 3

File details

Details for the file dagster_community_components-0.10.64.tar.gz.

File metadata

File hashes

Hashes for dagster_community_components-0.10.64.tar.gz
Algorithm Hash digest
SHA256 ead2e5ba4bf70c5f301f2af73fb6133f8ef5571faf50cc18483357c2c37741d4
MD5 6c42866a1160938ebb9bed1f31c9eb65
BLAKE2b-256 97157a64b2e5de74bd49139f23ada35b96729d2b00e9ca5df47a86b383e22682

See more details on using hashes here.

Provenance

The following attestation bundles were made for dagster_community_components-0.10.64.tar.gz:

Publisher: publish.yml on eric-thomas-dagster/dagster-component-templates

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dagster_community_components-0.10.64-py3-none-any.whl.

File metadata

File hashes

Hashes for dagster_community_components-0.10.64-py3-none-any.whl
Algorithm Hash digest
SHA256 f357fce971430a23c152bb7e443880ad7b6eb7ab036bde2f401c3561de62b0ab
MD5 358410dfa34946bf69b0eef902c6007d
BLAKE2b-256 586587fb19051c9a4d780a91cfcf02e728c2ac8b454149061cf5bd1195590481

See more details on using hashes here.

Provenance

The following attestation bundles were made for dagster_community_components-0.10.64-py3-none-any.whl:

Publisher: publish.yml on eric-thomas-dagster/dagster-component-templates

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.10.101

1 file

0.10.100

1 file

0.10.99

1 file

0.10.98

1 file

0.10.97

2 files

0.10.96

2 files

0.10.95

2 files

0.10.94

2 files

0.10.93

2 files

0.10.92

2 files

0.10.91

2 files

0.10.90

2 files

0.10.89

2 files

0.10.88

2 files

0.10.87

2 files

0.10.86

2 files

0.10.85

2 files

0.10.84

2 files

0.10.83

2 files

0.10.82

2 files

0.10.81

2 files

0.10.80

2 files

0.10.79

2 files

0.10.78

2 files

0.10.77

2 files

0.10.76

2 files

0.10.75

2 files

0.10.74

2 files

0.10.73

2 files

0.10.72

2 files

0.10.71

2 files

0.10.70

2 files

0.10.69

2 files

0.10.68

2 files

0.10.67

2 files

0.10.65

2 files

This release

0.10.64 This release

2 files

0.10.63

2 files

0.10.62

2 files

0.10.60

2 files

0.10.59

2 files

0.10.58

2 files

0.10.57

2 files

0.10.56

2 files

0.10.55

2 files

0.10.53

2 files

0.10.52

2 files

0.10.51

2 files

0.10.50

2 files

0.10.49

2 files

0.10.48

2 files

0.10.47

2 files

0.10.46

2 files

0.10.45

2 files

0.10.44

2 files

0.10.43

2 files

0.10.42

2 files

0.10.41

2 files

0.10.40

2 files

0.10.39

2 files

0.10.38

2 files

0.10.37

2 files

0.10.36

2 files

0.10.35

2 files

0.10.34

2 files

0.10.33

2 files

0.10.32

2 files

0.10.31

2 files

0.10.30

2 files

0.10.29

2 files

0.10.28

2 files

0.10.27

2 files

0.10.26

2 files

0.10.20

2 files

0.10.19

2 files

0.10.18

2 files

0.10.17

2 files

0.10.16

2 files

0.10.15

2 files

0.10.14

2 files

0.10.13

2 files

0.10.12

2 files

0.10.11

2 files

0.10.10

2 files

0.10.9

2 files

0.10.8

2 files

0.10.7

2 files

0.10.6

2 files

0.10.5

2 files

0.10.4

2 files

0.10.3

2 files

0.10.2

2 files

0.10.1

2 files

0.10.0

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.10

2 files

0.8.9

2 files

0.8.8

2 files

0.8.7

2 files

0.8.6

2 files

0.8.5

2 files

0.8.4

2 files

0.8.3

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.0

2 files

0.6.6

2 files

0.6.5

2 files

0.6.4

2 files

0.6.3

2 files

0.6.2

2 files

0.6.1

2 files

0.6.0

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.0

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page