geniusrise-huggingface 0.4.2

Huggingface bolts for geniusrise

Huggingface Bolts

This is a collection of generic streaming and (micro) batch bolts interfacing with the huggingface ecosystem.

Table of Contents

• Huggingface Bolts
  • Usage
  • Usage
    • Text Classification

Includes:

No.	Name	Description	Input Type	Output Type
1	Text Classification	Fine-tuning for text classification tasks	Batch	Batch
2	Instruction Tuning	Fine-tuning for instruction tuning tasks	Batch	Batch
3	Commonsense Reasoning	Fine-tuning for commonsense reasoning tasks	Batch	Batch
4	Language Modeling	Fine-tuning for language modeling tasks	Batch	Batch
5	Named Entity Recognition	Fine-tuning for named entity recognition tasks	Batch	Batch
6	Question Answering	Fine-tuning for question answering tasks	Batch	Batch
7	Sentiment Analysis	Fine-tuning for sentiment analysis tasks	Batch	Batch
8	Summarization	Fine-tuning for summarization tasks	Batch	Batch
9	Translation	Fine-tuning for translation tasks	Batch	Batch

Usage

To test, first bring up all related services via the supplied docker-compose:

docker compose up -d
docker compose logs -f

These management consoles will be available:

Console	Link
Kafka UI	http://localhost:8088/

Postgres can be accessed with:

docker exec -it geniusrise-postgres-1 psql -U postgres

Usage

Text Classification

To fine-tune a model for text classification tasks, you can use the following command:

genius HuggingFaceClassificationFineTuner rise \
  batch \
      --input_folder my_dataset \
  streaming \
      --output_kafka_topic my_topic \
      --output_kafka_cluster_connection_string localhost:9094 \
  postgres \
      --postgres_host 127.0.0.1 \
      --postgres_port 5432 \
      --postgres_user postgres \
      --postgres_password postgres \
      --postgres_database geniusrise \
      --postgres_table state \
  load_dataset \
      --args