Integrate VDK with Huggingface as both data source and target
Project description
Huggingface
Versatile Data Kit (VDK) plugin for integrating with Huggingface as both a data source and a target. This plugin allows you to ingest data payloads into a Huggingface repository and makes it easier to work with datasets stored in Huggingface.
Usage
pip install vdk-huggingface
The functionality adds new ingestion method "huggingface" which can be used like that:
job_input.send_object_for_ingestion(data, method="huggingface")
Configuration
(vdk config-help
is useful command to browse all config options of your installation of vdk)
Name | Description | (example) Value |
---|---|---|
HUGGINGFACE_TOKEN | HuggingFace API token for authentication. Get one from HuggingFace Settings | "" |
HUGGINGFACE_REPO_ID | HuggingFace Dataset repository ID | "username/test-dataset" |
Build and testing
pip install -r requirements.txt
pip install -e .
pytest
In VDK repo ../build-plugin.sh script can be used also.
Note about the CICD:
.plugin-ci.yaml is needed only for plugins part of Versatile Data Kit Plugin repo.
The CI/CD is separated in two stages, a build stage and a release stage. The build stage is made up of a few jobs, all which inherit from the same job configuration and only differ in the Python version they use (3.7, 3.8, 3.9 and 3.10). They run according to rules, which are ordered in a way such that changes to a plugin's directory trigger the plugin CI, but changes to a different plugin does not.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for vdk_huggingface-0.1.1431637373.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | bce75c93e8a39b2691d9dcda1977627d5e973300e284bd5d11f370cc0d592535 |
|
MD5 | acd3498c245cdafaa026fc199bd0c316 |
|
BLAKE2b-256 | 18192d4301fafa961458735cce412abbd3c474dd68a4c9ed168998b5a6903512 |