Skip to main content

A library of datasets, LLM tooling, and open source models that can be used for training and inference for prototyping purposes.

Reason this release was yanked:

intel's bigdl was migrated to ipex-llm which is not going to be supported long term

Project description

fun dialogues

A library of datasets, LLM tooling, and open source models that can be used for training and inference for prototyping purposes. The project began as a collection of fictitious dialogues that can be used to train language models or augment prompts for prototyping and educational purposes. It has now grown to include tooling for LLM application development purposes and open source models.

Key components of fun dialogues:

  • JudgyRAG: RAG eval tool
  • Benchmark Datasets: custom benchmark datasets
  • Dialogues: fictitious dialogue datasets for fun, experimentation, and testing.

You can install fun dialogues using pip: pip install fundialogues

Fun Dialogues Logo

JudgyRAG

JudgyRAG is a component of the FunDialogues Python library focused on evaluating the performance of Retrieval-Augmented Generation (RAG) systems. It facilitates this by creating synthetic datasets based on custom datasets, enabling a unique assessment of a RAG system's question-answering capabilities in a zero-shot Q&A context. Initially, JudgyRAG's primary functionality is the automatic generation of custom multiple-choice Q&A datasets. Future iterations will introduce further automation to seamlessly integrate with popular frameworks, enhancing testing and benchmarking processes.

Workflow

The workflow for JudgyRAG includes:

  1. Scraping PDFs: Information is extracted from PDFs into structured text formats.
  2. Chunking Data: Extracted data is chunked similarly to vector database embeddings for RAG, simulating data breakdown and storage.
  3. Question Generation: Each chunk acts as a knowledge base, with custom prompts instructing supported models (currently LLaMA 7B and 13B chat) to generate multiple-choice questions.
  4. Iterative Parsing: Chunks are processed iteratively, generating a multiple-choice question for each.
  5. Quality Checks: Poor-quality chunks leading to failed question generation are flagged for user review.
  6. Benchmark Compilation: The final document includes multiple-choice questions, correct answers, and source knowledge chunks.
  7. RAG System Evaluation: The synthetic benchmark dataset can be used to assess a RAG system, with automation for this process planned for future updates.

Environment Setup

Follow these steps to set up your environment for JudgyRAG:

Step 1

Install Visual Studio 2022 Community Edition with the “Desktop development with C++” workload.

Step 2

Update to the latest GPU driver.

Step 3

Install the Intel® oneAPI Base Toolkit 2024.0.

Step 4

Download the necessary wheels:

wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torch-2.1.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torchvision-0.16.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/intel_extension_for_pytorch-2.1.10%2Bxpu-cp39-cp39-win_amd64.whl

Step 5

Install the downloaded packages and BigDL LLM:

pip install torch-2.1.0a0+cxx11.abi-cp39-cp39-win_amd64.whl
pip install torchvision-0.16.0a0+cxx11.abi-cp39-cp39-win_amd64.whl
pip install intel_extension_for_pytorch-2.1.10+xpu-cp39-cp39-win_amd64.whl
pip install --pre --upgrade bigdl-llm[xpu]
conda install libuv

Step 6

Activate the Intel oneAPI environment:

call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"

For the latest setup instructions for BigDL LLM inference, visit BigDL Documentation

Example Usage of JudgyRAG

from fundialogues import benchgen, judgypdf

folder_path = ""
output_directory = ""
chunk_file = ""
benchmark_output_directory = ""

judgypdf(folder_path, output_directory)
benchgen(chunk_file, benchmark_output_directory)

Benchmark Datasets

opengeoquery-v1

OpenGeoQuery-v1 is the first edition of a benchmark dataset composed of statements associated with the geosciences. The content of the dataset touches on topics like geophysics, petrology, minerology, seismology, geomorphology, etc. The purpose of this dataset is to use as a benchmark and for fine-tuning small geoscience LLMs (coming soon).

Dialogues

  • Customer Service
    • Grocery Cashier: 100 fictitious examples of dialogues between a customer at a grocery store and the cashier.
    • Robot Maintenance: 100 fictitious examples of dialogues between a robot arm technician and a customer.
    • Apple Picker Maintenance: 100 fictitious examples of dialogues between a apple harvesting equipment technician and a customer.
  • Academia
    • Physics Office Hours: 100 fictitious examples of dialogues between a physics professor and a student during office hours.
  • Healthcare
    • Minor Consultation: 100 fictitious examples of dialogues between a doctor and a patient during a minor medical consultation.
  • Sports
    • Basketball Coach: 100 fictitious examples of dialogues between a basketball coach and the players on the court during a game.

How to Load Dialogues

Loading dialogues can be accomplished using the fun dialogues library or Hugging Face datasets library.

Load using fun dialogues

Assuming you've already installed fundialogues.

Use loader utility to load dataset as pandas dataframe. Further processing might be required for use.

from fundialogues import dialoader

# load as pandas dataframe
physics_office_hours = dialoader("FunDialogues/academia-physics-office-hours")

Loading using Hugging Face datasets

  1. Install datasets package pip install datasets

  2. Load using datasets

from datasets import load_dataset

physics_office_hours = load_dataset("FunDialogues/academia-physics-office-hours")

Disclaimer

The dialogues contained in this repository are provided for experimental purposes only. It is important to note that these dialogues are assumed to be original work by a human and are entirely fictitious, despite the possibility of some examples including factually correct information. The primary intention behind these dialogues is to serve as a tool for language modeling experimentation and should not be used for designing real-world products beyond non-production prototyping.

Please be aware that the utilization of fictitious data in these datasets may increase the likelihood of language model artifacts, such as hallucinations or unrealistic responses. Therefore, it is essential to exercise caution and discretion when employing these datasets for any purpose.

It is crucial to emphasize that none of the scenarios described in the fun dialogues dataset should be relied upon to provide advice or guidance to humans. These scenarios are purely fictitious and are intended solely for demonstration purposes. Any resemblance to real-world situations or individuals is entirely coincidental.

The responsibility for the usage and application of these datasets, tools, and codes rests solely with the individual or entity employing them. By accessing and utilizing these assets and all contents of the repository, you acknowledge that you have read and understood this disclaimer, and you agree to use them at your own discretion and risk.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fundialogues-0.1.0.tar.gz (22.2 kB view details)

Uploaded Source

Built Distribution

fundialogues-0.1.0-py3-none-any.whl (19.6 kB view details)

Uploaded Python 3

File details

Details for the file fundialogues-0.1.0.tar.gz.

File metadata

  • Download URL: fundialogues-0.1.0.tar.gz
  • Upload date:
  • Size: 22.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.9.18

File hashes

Hashes for fundialogues-0.1.0.tar.gz
Algorithm Hash digest
SHA256 5ee9b2d2c3d83db3533c142b3598fe4b3ba0deed94f74c46b10e0341672eabc7
MD5 329dad56ba2fc3fd11df1fbe09737d72
BLAKE2b-256 50f3c653a0821e7ee7a1a774cab8fb9c3895b42aac04379646fbec63fef94350

See more details on using hashes here.

File details

Details for the file fundialogues-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: fundialogues-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 19.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.0.0 CPython/3.9.18

File hashes

Hashes for fundialogues-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 929df7ca96a20bb1707555ea698df99c8f0a2d150d8ab392d858e7bf5e5199f4
MD5 6ac4f978c6ce4d662d850da82954f0ff
BLAKE2b-256 4ac61122c12973c92afc03d829e33e56fff7025c01e14251161195664646b625

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page