hyped 0.2.0a4

Easily streamline and customize data pipelines

:boom: Hyped

Hyped is a versatile framework built on top of HuggingFace Datasets, designed to simplify the management and execution of data pipelines. With Hyped, you can define data pipelines as Directed Acyclic Graphs (DAG) of data processors, leveraging the rich ecosystem of the datasets library while also providing the flexibility to implement custom processors when needed.

Hyped aims to offer an intuitive high-level interface for defining and executing data pipelines, all while being flexible and scalable.

Features

• Seamless Integration with Hugging Face Datasets: Utilize the extensive collection of datasets available through HuggingFace with ease. Hyped handles data loading and preprocessing using HuggingFace's powerful tools.
• Flexible Data Processing: Define complex data processing workflows by linking data processors in a DAG. Each data processor in Hyped is fully configurable, allowing users to fine-tune their behavior according to specific requirements. Hyped comes with a set of general-purpose processors out of the box, allowing for a wide range of transformations and manipulations on your data.
• Modular Design: Break down complex workflows into reusable components, improving code readability and maintainability. Hyped’s modular approach allows for the creation of clear and organized data processing pipelines. By designing data processors as reusable components, you can easily repurpose them across different pipelines, reducing redundancy and fostering efficient development practices.
• Custom Processor Support: Implement custom data processors tailored to your specific requirements. Whether you need to apply domain-specific transformations or integrate with external libraries, Hyped provides the flexibility to extend its functionality as needed.
• Efficient Execution: Execute your data pipelines efficiently, whether you’re working with small datasets or processing large volumes of data. Hyped supports multiprocessing and data streaming out of the box, enabling efficient utilization of computational resources and avoiding memory limitations when processing large datasets.
• Scalability: Hyped provides scalability to handle diverse workload demands, allowing you to seamlessly scale your data processing tasks as needed. Whether you’re processing small datasets on a single machine or dealing with large volumes of data across distributed computing environments, Hyped adapts to your workload requirements, ensuring efficient execution and resource utilization.

Getting Started

Get up and running with Hyped in no time! Follow these simple steps to install the framework and start defining and executing your data pipelines effortlessly.

For detailed documentation, please refer to the Hyped Documentation

Installation

Hyped is available on PyPI and can be installed using pip:

pip install hyped

Alternatively, you can install Hyped directly from the source code repository:

# Clone the Hyped repository from GitHub
git clone https://github.com/open-hyped/hyped.git

# Navigate to the cloned repository
cd hyped

# Install the package including optional developer dependencies
pip install -e .[linting, tests]

Now you're ready to start using Hyped for managing and executing your data pipelines!

Usage

Start by importing the necessary modules and classes. Note that the following example also requires the hyped-extensions-nlp extension:

import datasets
from hyped import DataFlow
from hyped.extensions.nlp import TransformersTokenizer

Next, load your dataset using the datasets library. In this example, we load the IMDb dataset:

ds = datasets.load_dataset("imdb", split="test")

With the dataset features available we can create a data flow instance:

flow = DataFlow(features=ds.features)

Now we can add processing steps by calling data processors on the features. In this example we add a tokenizer processor to tokenize the text input feature using a BERT tokenizer:

tokenizer = TransformersTokenizer(tokenizer="bert-base-uncased")
tokenized_features = tokenizer.call(text=flow.source["text"])

Finally, we can apply the data pipeline to your dataset using the apply method. Here we also need to specify the which features are to be collected into the output dataset:

ds = flow.apply(ds, collect=tokenized_features)

Now, your dataset has been processed according to the defined pipeline, and you can proceed with further analysis or downstream tasks in your application.

For more examples and advanced usage scenarios, check out the Tutorials.

Running Tests

Hyped includes a suite of tests to ensure its functionality. You can run these tests using pytest:

pytest tests

Ensure that you have pytest and the required testing packages installed in your environment. You can install them via pip:

pip install hyped[tests]

Running the tests will execute various test cases to validate the behavior of Hyped.

If you want to include integration tests during your test runs, you can use the following command:

pytest tests --with-integration

Contribution Guidelines

We welcome contributions from the community to help improve and expand Hyped. Before contributing, please review our Contribution Guidelines for instructions on reporting bugs, suggesting features, and submitting pull requests.

License

tbd