A great package.A Python library providing useful components for using [Dagster](https://dagster.io/) to create academic research cloud datalakes
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Project description
dagster-ncsa
A Python library providing useful components for using Dagster to create academic research cloud data lakes for the National Center for Supercomputing Applications (NCSA).
Overview
dagster-ncsa extends Dagster's capabilities with specialized tools designed
specifically for academic research workflows and data management at scale. It
provides abstractions and utilities to simplify building, managing, and
monitoring data pipelines in research-oriented cloud data lake environments.
Components
- S3ResourceNCSA: Extends the Dagster S3 resource to add some useful helper functions for working with S3 objects in a research data pipeline.
- AirTableCatalogResource: A resource for interacting with AirTable tables as a catalog for data assets in a research data pipeline.
Installation
Basic Installation
pip install dagster-ncsa
Development Installation
pip install -e ".[dev]"
Development
Setup Development Environment
# Clone the repository
git clone https://github.com/your-organization/dagster-ncsa.git
cd dagster-ncsa
# Install development dependencies
pip install -e ".[dev]
Project details
Verified details
These details have been verified by PyPIProject links
GitHub Statistics
Maintainers
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file dagster_ncsa-0.1rc1.tar.gz.
File metadata
- Download URL: dagster_ncsa-0.1rc1.tar.gz
- Upload date:
- Size: 11.3 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
43a45f33a2d4947906344c7df33461bf2d07b5d8d930d359a0477d21545093c8
|
|
| MD5 |
5e7c0ba9119bc25c36069ba6b056f816
|
|
| BLAKE2b-256 |
342aecd9fe850a5066afe9447a997611a06072594c2837c689011c6db53b3e4e
|
Provenance
The following attestation bundles were made for dagster_ncsa-0.1rc1.tar.gz:
Publisher:
cd.yml on ncsa/dagster-ncsa
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
dagster_ncsa-0.1rc1.tar.gz -
Subject digest:
43a45f33a2d4947906344c7df33461bf2d07b5d8d930d359a0477d21545093c8 - Sigstore transparency entry: 178992753
- Sigstore integration time:
-
Permalink:
ncsa/dagster-ncsa@73a18eb16ffccb2bb1a78fa4c80d48c0b4c2cd00 -
Branch / Tag:
refs/tags/0.1-rc.1 - Owner: https://github.com/ncsa
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
cd.yml@73a18eb16ffccb2bb1a78fa4c80d48c0b4c2cd00 -
Trigger Event:
release
-
Statement type:
File details
Details for the file dagster_ncsa-0.1rc1-py3-none-any.whl.
File metadata
- Download URL: dagster_ncsa-0.1rc1-py3-none-any.whl
- Upload date:
- Size: 5.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3f8361b6e24a759ce5c075c8458bb608548ac7407cee030b7a8a7990ac33b325
|
|
| MD5 |
cca66aa11e302b44408ca3f15ce40fc2
|
|
| BLAKE2b-256 |
f43f5ffb4d5197d438e538271911de5fe4ca0d46370577c48410df0dabf65006
|
Provenance
The following attestation bundles were made for dagster_ncsa-0.1rc1-py3-none-any.whl:
Publisher:
cd.yml on ncsa/dagster-ncsa
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
dagster_ncsa-0.1rc1-py3-none-any.whl -
Subject digest:
3f8361b6e24a759ce5c075c8458bb608548ac7407cee030b7a8a7990ac33b325 - Sigstore transparency entry: 178992757
- Sigstore integration time:
-
Permalink:
ncsa/dagster-ncsa@73a18eb16ffccb2bb1a78fa4c80d48c0b4c2cd00 -
Branch / Tag:
refs/tags/0.1-rc.1 - Owner: https://github.com/ncsa
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
cd.yml@73a18eb16ffccb2bb1a78fa4c80d48c0b4c2cd00 -
Trigger Event:
release
-
Statement type:
