fiddlecube 0.1.2

Generate ideal question-answer dataset for testing your LLM.

FiddleCube - Generate ideal question-answers for testing RAG

FiddleCube generates an ideal question-answer dataset for testing your LLM. Run tests on this dataset before pushing any prompt or RAG upgrades.
Jump to: Ensuring diversity and correctness

Quickstart

Install FiddleCube

pip3 install fiddlecube

Register

Run

fiddlecube register

Follow the steps to get an API key.

Usage

from fiddlecube import FiddleCube

fc = FiddleCube(api_key="<api-key>")
dataset = fc.generate(
    [
        "The cat did not want to be petted.",
        "The cat was not happy with the owner's behavior.",
    ],
    10,
)
dataset

{
   "results":[
      {
         "query":"Question: Why did the cat not want to be petted?",
         "contexts":[
            "The cat did not want to be petted."
         ],
         "answer":"The cat did not want to be petted because it was not in the mood for physical affection at that moment.",
         "score":0.8,
         "question_type":"SIMPLE"
      },
      {
         "query":"Was the cat pleased with the owner's actions?",
         "contexts":[
            "The cat was not happy with the owner's behavior."
         ],
         "answer":"No, the cat was not pleased with the owner's actions.",
         "score":0.8,
         "question_type":"NEGATIVE"
      }
   ],
   "status":"COMPLETED",
   "num_tokens_generated":44,
   "rate_limited":false
}

Ensuring diversity and correctness

• The questions are spread across the vector embeddings to ensure completeness of testing.
• The queries and responses are evaluated for correctness and context relevancy.
• Citations to the database context are maintained for ease of testing and auditing.