A metaflow card that renders HTML inputs.
Project description
Installation
pip install metaflow-card-hf-dataset
Usage
After installing the module, you can add any HuggingFace dataset to your Metaflow tasks by using the @huggingface_dataset
decorator. There are two ways to use the decorator:
- Via the id
argument, which is the dataset ID from HuggingFace.
- Via the artifact_id
argument, which is the name of a FlowSpec artifact that contains the dataset ID.
Use the first if your workflow always reads from the same HuggingFace dataset ID. Use the second if your workflow pass in dataset IDs as parameters or changes them dynamically.
from metaflow import FlowSpec, step, huggingface_dataset, Parameter
class Flow(FlowSpec):
eval_ds = Parameter('eval_ds', default='argilla/databricks-dolly-15k-curated-en', help='HuggingFace dataset id.')
# Dynamically input: python flow.py run --eval_ds lighteval/mmlu
@huggingface_dataset(id="princeton-nlp/SWE-bench")
@step
def start(self):
self.another_one = 'wikimedia/wikipedia'
self.next(self.end)
@huggingface_dataset(artifact_id="another_one") # Use the dataset ID set to an artifact var.
@huggingface_dataset(artifact_id="eval_ds") # Use the dataset ID passed as a parameter.
@step
def end(self):
pass
if __name__ == '__main__':
Flow()
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for metaflow_card_hf_dataset-0.0.5.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3735a796e757dba59a64f769e51f48dcc82455b5dfdf280c099a94fc60329771 |
|
MD5 | 1ed3f7fc93bc5d203a942a69a0239498 |
|
BLAKE2b-256 | 137058a903a2197465e891755f59b6222fe609e0a5ff4e40dc8bbf10e1893e12 |
Close
Hashes for metaflow_card_hf_dataset-0.0.5-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01abc55f43378f87d38c7cdf344600093419437ef7f4b6cb9099d580d5685b68 |
|
MD5 | a6cdd4289e968217eb63231e77cb44ef |
|
BLAKE2b-256 | b8943f5ecdd389c2e9c28e53698f58be5edfa284e6244774330eafebd636bc90 |