aviary.litqa
LitQA2 environment implemented with aviary, allowing agents to perform question answering on the LitQA dataset.
LitQA (now legacy) is a dataset composed from 50 multiple-choice questions from recent literature. It is designed to test the LLM's the ability to retrieve information outside of the pre-training corpus. To ensure the questions are not in the pre-training corpus, the questions were collected from scientific papers published after September 2021 -- cut-off date of GPT-4's training data.
LitQA2 is part of the LAB-Bench dataset. LitQA2 contains 248 multiple-choice questions from the literature and was created ensuring that the questions cannot be answered by recalling from the pre-training corpus only. It considered scientific paper published within 36 months from the data of its publication. Therefore, LitQA2 is considered a scientific RAG dataset.
Installation
To install the LitQA environment, run:
pip install 'fhaviary[litqa]'
Usage
In litqa/env.py, you will find:
GradablePaperQAEnvironment: an environment that can grade answers given an evaluation function.
And in litqa/task.py, you will find:
LitQAv2TaskDataset: a task dataset designed to pull LitQA v2 from Hugging Face,
and create one GradablePaperQAEnvironment per question
Here is an example of how to use them:
import os
from ldp.agent import SimpleAgent
from ldp.alg import Evaluator, EvaluatorConfig, MeanMetricsCallback
from paperqa import Settings
from aviary.env import TaskDataset
from aviary.envs.litqa.task import TASK_DATASET_NAME
async def evaluate(folder_of_litqa_v2_papers: str | os.PathLike) -> None:
settings = Settings(paper_directory=folder_of_litqa_v2_papers)
dataset = TaskDataset.from_name(TASK_DATASET_NAME, settings=settings)
metrics_callback = MeanMetricsCallback(eval_dataset=dataset)
evaluator = Evaluator(
config=EvaluatorConfig(batch_size=3),
agent=SimpleAgent(),
dataset=dataset,
callbacks=[metrics_callback],
)
await evaluator.evaluate()
print(metrics_callback.eval_means)
References
[1] Lála et al. PaperQA: Retrieval-Augmented Generative Agent for Scientific Research. ArXiv:2312.07559, 2023.
[2] Skarlinski et al. Language agents achieve superhuman synthesis of scientific knowledge. ArXiv:2409.13740, 2024.
[3] Laurent et al. LAB-Bench: Measuring Capabilities of Language Models for Biology Research. ArXiv:2407.10362, 2024.
Metadata
Release files for aviary.litqa 0.29.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| aviary_litqa-0.29.0.tar.gz | 1.5 MB | Without provenance | Details for aviary_litqa-0.29.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| aviary_litqa-0.29.0-py3-none-any.whl | Python 3 | none | any | Details for aviary_litqa-0.29.0-py3-none-any.whl |
Total release size: 1.5 MB
Release files / aviary_litqa-0.29.0.tar.gz
| Download URL | aviary_litqa-0.29.0.tar.gz |
|---|---|
| Size | 1.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9a308fb2c421ba65d8030ee3c6c756ea142b1fe85bd5d27a5605951f1313f6ad
|
|
BLAKE2b-256 checksum How to use checksums |
42ca3addbd8803d15b52f7e905d0a205bc277ea18f9097f72a44e277b26dfa35
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / aviary_litqa-0.29.0-py3-none-any.whl
| Download URL | aviary_litqa-0.29.0-py3-none-any.whl |
|---|---|
| Size | 11.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
16f48fb22d757a6650d762ab75247f4e5eff558265522a9b085c75349ca8164b
|
|
BLAKE2b-256 checksum How to use checksums |
55671a94f18f16e95305650fe9f4a6560ab463479821681ce23b254e68d042a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for aviary.litqa 0.29.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.