Skip to main content

Dagster Component Templates

A community library of 1,335 reusable Dagster component templates covering data ingestion, transformations, ML / analytics, AI / LLM enrichment, orchestration, infrastructure provisioning, reverse ETL, observability, sensors, asset checks, and enterprise tool integrations — all configurable via YAML with no Python required.

Field naming conventions across the registry are documented in FIELD_CONVENTIONS.md.

What are Dagster Components?

Components are reusable, YAML-configurable building blocks that package common data engineering patterns. Drop one into your project's defs/ folder and configure it in YAML — no boilerplate Python needed.

# defs/components/sqs_ingest.yaml
type: dagster_community_components.SQSToDatabaseAssetComponent
attributes:
  asset_name: raw_events
  queue_url_env_var: SQS_QUEUE_URL
  database_url_env_var: DATABASE_URL
  table_name: raw_events
  partition_type: daily
  partition_start: "2024-01-01"

Component Library

Category Count What's in it
ingestion 175 Source-to-destination data movement
resource 167 Connection-handle resources for SaaS / data platforms
ai 142 LLMs, vision, embeddings, vector stores, document extraction
transformation 128 Pandas-style transforms (filter, join, union, datetime, regex, etc.)
analytics 95 ML models, scoring, segmentation, forecasting, geo, A/B testing
source 95 Read-only data sources
reverse_etl 78 Push a DataFrame/record into a SaaS tool (upsert, custom-audience sync, workflow trigger)
sensor 71 Polling sensors that emit RunRequests
integration 69 Multi-asset wrappers for cloud platforms
sink 64 DataFrame writers (CSV, Parquet, warehouses, NoSQL)
jobs 63 Op-job components (cleanup, triggers, exports, heartbeats)
infrastructure 43 IaC / provisioning + lineage anchors
io_manager 33 IO managers for the major warehouses + lakes
external 32 Declare-only external assets (warehouse tables, S3 objects)
observation 21 Health-check sensors emitting AssetObservations
security 20 SIEM/audit-log export and normalization (Splunk, Sentinel, Chronicle, QRadar, etc.)
decorator 18 Asset-wrapping behaviors (caching, dry-run, lifecycle, hooks, data contracts)
check 17 Asset-check components (Great Expectations, Soda, etc.)
core 2 Cross-cutting primitives (dependency graph, asset references)
dbt 2 dbt-project wrappers
total 1,335

By Dagster primitive (produces field)

Every manifest entry declares which Dagster primitive(s) it emits when loaded — machine-readable metadata for tools that need to route users to the right authoring surface ("Add schedule" from the Schedules tab, "Add asset" from Graph view, etc.).

produces Count Meaning
asset 774 Single @asset (or one asset key per instance)
asset_check 235 @asset_check — many components ship free schema-drift checks via build_column_schema_change_checks
resource 206 Connection / auth handle passed to downstream assets
sensor 117 @sensor / polling RunRequest emitter
job 67 @job / define_asset_job(...)
schedule 66 @schedule / ScheduleDefinition
multi_asset 46 @multi_asset OR workspace-shape (many assets per config, e.g. snowflake_workspace, coalesce_workspace)
io_manager — Component wraps an IOManagerDefinition (0 today — most io_managers live under resource or asset)

Filter with dagster-component search "" --produces schedule to list every scheduling component. See CONTRIBUTING.md#manifest-fields for the full enum and semantics.

Assets (90 analytics + 73 ai + 59 transformation + 49 ingestion + 23 sink + 21 external + 19 infrastructure + 18 source + 18 integration + 7 check = 377)

Ingestion — cloud storage s3_to_database_asset · gcs_to_database_asset · adls_to_database_asset

Ingestion — messaging & streaming kafka_to_database_asset · sqs_to_database_asset · kinesis_to_database_asset · eventhubs_to_database_asset · servicebus_to_database_asset · rabbitmq_to_database_asset · pubsub_to_database_asset · redis_streams_to_database_asset · nats_to_database_asset · pulsar_to_database_asset · mqtt_to_database_asset

Ingestion — files & databases sftp_to_database_asset · sql_to_database_asset · csv_file_ingestion · rest_api_ingestion · openapi_ingestion · graphql_ingestion

AI / LLM enrichment litellm_inference_asset · ollama_inference_asset · langchain_chain_asset · llm_prompt_executor · llm_chain_executor · document_summarizer · entity_extractor · embeddings_generator · moderation_scorer · anthropic_llm · conversation_memory · snowflake_cortex_asset

dbt dbt_docs_enriched_project — extends DbtProjectComponent with exposures, metrics, semantic models, contracts, source freshness, and clickable dbt docs links on every asset

Enterprise orchestration abinitio_run_asset · matillion_run_asset · rivery_run_asset · precisely_run_asset · step_functions_asset · dataiku_asset · autosys_asset

Infrastructure as Code (provision resources before pipeline runs) terraform_asset · terraform_cloud_asset · cloudformation_asset · ansible_asset · pulumi_asset · helm_deploy · aws_cdk_asset

Secrets & remote execution ssh_asset · hashicorp_vault

Notebooks & ML compute jupyter_notebook_asset · modal_asset

Vector stores pinecone_asset · pgvector_asset · chromadb_asset · elasticsearch_asset

Feature stores feast_asset · tecton_asset

ML experiment tracking wandb_asset

Data versioning lakefs_asset

Reverse ETL polytomic_asset

Schema discovery warehouse_schema_assets — introspects a warehouse at prepare time, creates one external AssetSpec per table with full column metadata

Analytics & ML — predictive decision_tree_model · random_forest_model · logistic_regression_model · linear_regression_model · naive_bayes_model · neural_network_model · gradient_boosting_model · svm · count_regression · gamma_regression · spline_model · survival_analysis · cross_validation · stepwise · model_coefficients · model_score · model_comparison · vif

Analytics & ML — clustering / dim. reduction k_means_clustering · spatial_cluster · pca · multidimensional_scaling · nearest_neighbors · append_cluster · k_centroids_diagnostics

Analytics — A/B testing ab_test_analysis · ab_treatments · ab_controls · ab_trend · test_of_means

Analytics — geospatial distance_calculator · bounding_box_filter · coordinate_transformer · point_in_polygon · spatial_join · create_points · buffer · make_grid · smooth

Analytics — time series time_series_generator · arima_forecast · ets_forecast · ts_filler · ts_compare · ts_forecast · ts_covariate_forecast · ts_model_factory

Analytics — customer / business anomaly_detection · customer_segmentation · customer_360 · customer_health_score · customer_journey_mapping · cohort_analysis · rfm_segmentation · funnel_analysis · lead_scoring · propensity_scoring · churn_prediction · ltv_prediction · subscription_metrics · revenue_attribution · multi_touch_attribution · campaign_performance · product_recommendations · product_usage_analytics · priority_scorer · market_basket_rules


Sensors (40)

Cloud storage s3_monitor · gcs_monitor · adls_monitor

Messaging & streaming kafka_monitor · sqs_monitor · kinesis_monitor · eventhubs_monitor · servicebus_monitor · rabbitmq_monitor · pubsub_monitor · redis_streams_monitor · nats_monitor · pulsar_monitor · mqtt_monitor

Files sftp_monitor · sql_monitor

Enterprise tools coalesce_job_sensor · abinitio_job_sensor · matillion_job_sensor · rivery_job_sensor · precisely_job_sensor

Notifications slack_notification · twilio_notification

SaaS event sensors github_event_sensor · gitlab_event_sensor · stripe_event_sensor · zendesk_ticket_sensor · jira_issue_sensor · pagerduty_incident_sensor · linear_issue_sensor · notion_database_sensor · servicenow_sensor

Generic / no-auth http_poll_sensor · rss_feed_sensor · filesystem_monitor

ML triggers mlflow_model_sensor


Asset Checks (6)

dq_check · great_expectations_check · soda_check · monte_carlo_check · sifflet_check · acceldata_check


Observations (20)

clickhouse_table_observation_sensor · snowflake_table_observation · bigquery_table_observation · postgres_table_observation · redshift_table_observation and more


External Assets (21)

external_clickhouse_table · external_snowflake_table · external_bigquery_table · external_postgres_table · external_s3_object · external_kafka_topic and more


Integrations (18)

aws_glue · aws_dms · aws_kinesis · aws_redshift · aws_sagemaker · azure_data_factory · azure_stream_analytics · azure_synapse · databricks_asset_bundle · databricks_workspace · google_bigquery · google_cloud_functions · google_cloud_run_jobs · google_dataflow · google_datastream · google_pubsub · google_vertex_ai · snowflake_workspace


Component Structure

Every component follows the same layout:

component_name/
├── component.py       # Dagster component class
├── example.yaml       # Working YAML configuration example
├── README.md          # Documentation and field reference
├── requirements.txt   # pip dependencies
└── schema.json        # Component registry metadata

StateBackedComponent

Several components that discover resources from external APIs use Dagster's StateBackedComponent pattern. The API call happens once at prepare time and is cached to disk — code-server reloads are instant with zero network calls.

Components using this pattern: coalesce_workspace, azure_data_factory, aws_glue, databricks_workspace, openapi_ingestion, warehouse_schema_assets, step_functions_asset, dataiku_asset, polytomic_asset, autosys_asset, terraform_cloud_asset

To refresh the cached state after adding pipelines/jobs in the external system:

dg utils refresh-defs-state
# or simply restart: dagster dev

Asset Dependencies & Lineage

Every asset component supports a deps field for declaring upstream dependencies in the asset graph:

type: dagster_community_components.LiteLLMInferenceAssetComponent
attributes:
  asset_name: enriched_tickets
  upstream_asset_key: raw_tickets   # loads data + draws lineage edge
  deps:                              # additional lineage-only edges
    - support_schema/tickets_raw
    - raw/other_table
  model: claude-3-5-sonnet-20241022
  prompt_template: "Classify: {body}"

upstream_asset_key — for components that load upstream data (LiteLLM, Ollama, LangChain, SQL): draws a lineage edge and loads the asset value at runtime.

deps — for all components: draws additional lineage-only edges without loading data. Use this to express that an asset depends on another without consuming it directly.

Dependencies can also be wired externally via map_resolved_asset_specs() in definitions.py — the same approach used by Dagster Designer.


Partitioning

All ingestion assets support time-based partitioning out of the box:

attributes:
  partition_type: daily          # none | daily | weekly | monthly
  partition_start_date: "2024-01-01"
  table_name: events_{partition_key}   # {partition_key} is substituted at runtime

Per-asset config via post_processing:

Every asset-emitting component in this library supports post_processing: — a base-component YAML block that lets you attach any AssetSpec attribute (freshness_policy, automation_condition, tags, owners, kinds, group_name, description, code_version, partitions_def) to specific assets without editing the component's source or writing Python. The target: field accepts the full Dagster asset-selection language (hierarchical groups, is:external, kind:dbt, tag/kind selectors, boolean composition, and legacy fnmatch globs).

This is the recommended path for per-asset overrides. Reach for post_processing: before translation: templates (which only handle model-typed fields) or splitting components (which fights dagster-dbt's input tracking).

type: dagster_community_components.<AnyAssetEmittingComponent>
attributes: {...}

post_processing:
  assets:
    - target: 'group:"marts/*"'          # hierarchical group wildcard
      attributes:
        automation_condition: "{{ dg.AutomationCondition.eager() }}"

    - target: "kind:dbt and tag:tier=gold"
      attributes:
        freshness_policy:
          maximum_lag_minutes: 30

    - target: "*"
      attributes:
        owners: ["data-platform@company.com"]

Attributes always safe on every component: automation_condition, freshness_policy, owners, tags, kinds, group_name, description, code_version — pure metadata, no compute contract.

partitions_def requires a partition-aware compute path: 332 of 573 asset-emitting components read context.partition_key in their compute (safe). 241 do not — declaring partitions on those via post_processing will silently produce the same data every partition. See docs/partition_aware_components.md for the full list.

Canonical example: DbtDocsEnrichedProjectComponent — its class docstring shows the shape end-to-end, including reading meta.dagster.partitions_def directly from a dbt model's schema.yml (v0.10.49+).


Sensor → Asset Pairing

Most sensors are designed to trigger a companion ingestion asset. The sensor detects new data and fires a RunRequest with source info in run_config; the asset reads and ingests it.

Sensor Companion Asset
s3_monitor s3_to_database_asset
kafka_monitor kafka_to_database_asset
sqs_monitor sqs_to_database_asset
kinesis_monitor kinesis_to_database_asset
eventhubs_monitor eventhubs_to_database_asset
rabbitmq_monitor rabbitmq_to_database_asset
sftp_monitor sftp_to_database_asset
… … (all 15 pairs complete)

dbt Docs Enrichment

The dbt_docs_enriched_project component is a drop-in replacement for DbtProjectComponent that adds rich metadata from the dbt manifest to every asset in the Dagster UI:

type: dagster_community_components.DbtDocsEnrichedProjectComponent
attributes:
  project: "{{ project_root }}/dbt_project"
  dbt_docs_url: "https://dbt-docs.internal.mycompany.com"
  include_exposures: true
  include_metrics: true
  include_semantic_models: true
  include_contracts: true
  include_source_freshness: true

Each dbt model asset then shows: a clickable link to the dbt docs page, downstream BI exposures, metrics, semantic models, contract status, and source freshness SLAs — all visible in the Dagster Asset Catalog. All include_* flags default to false — opt in to only what you need.


Quick Start

1. Install dagster

pip install dagster

2. Copy a component into your project

cp -r assets/s3_to_database_asset/ my_project/defs/components/

3. Configure in YAML

# defs/components/my_s3_ingest.yaml
type: dagster_community_components.S3ToDatabaseAssetComponent
attributes:
  asset_name: raw_orders
  database_url: ${DATABASE_URL}
  table_name: raw_orders

4. Load in definitions.py

import dagster as dg
from pathlib import Path

defs = dg.load_from_defs_folder(project_root=Path(__file__).parent)

Contributing

See CONTRIBUTING.md for how to add new components.

License

MIT License

Metadata

Release files for dagster-community-components 0.10.132

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dagster-community-components 0.10.132
File Size Uploaded Provenance Actions
dagster_community_components-0.10.132.tar.gz 16.0 MB With provenance Details for dagster_community_components-0.10.132.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dagster-community-components 0.10.132
File Interpreter ABI Platform Actions
dagster_community_components-0.10.132-py3-none-any.whl Python 3 none any Details for dagster_community_components-0.10.132-py3-none-any.whl

Total release size: 39.3 MB

Release files / dagster_community_components-0.10.132.tar.gz

Download URL dagster_community_components-0.10.132.tar.gz
Size 16.0 MB
Tags Source
SHA-256 checksum
How to use checksums
50618ddb4f11edb10ed3b384109d538de48624696419adb105cf2338f74a9a13
BLAKE2b-256 checksum
How to use checksums
18db70797dfdee39b9ecd92f22f0098cd9c3fa339ac148645085f7cad084e7c0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / dagster_community_components-0.10.132-py3-none-any.whl

Download URL dagster_community_components-0.10.132-py3-none-any.whl
Size 23.4 MB
Tags Python 3
SHA-256 checksum
How to use checksums
ba505da30e8771c203581cbfce98e4fd6cb6e3d688d57b499b9cec2951c79b79
BLAKE2b-256 checksum
How to use checksums
b3c743b29bfd2828fc0b9c0107aac0a0918e148dca87f2752e7da4be05961425
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for dagster-community-components 0.10.132

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for dagster_community_components-0.10.132.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.10.132 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page