A python wrapper over NESTFUL data
Project description
NESTFUL
This is the official repository for NESTFUL.
- Paper Title: NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
- Link: https://arxiv.org/abs/2409.03797v2
- HuggingFace Data Link: https://huggingface.co/datasets/ibm-research/nestful
Data
We have shared the latest NESTFUL evaluation set under data_v2 dir.
nestful_data.jsonl: It has 1861 evaluation data for nested sequencing.executable_functions: Contains the implementation of all the functions in the benchmark.
The data_v1 directory includes the data for the previous version of the paper - link.
executable: contains data and spec with necessary information to execute them through RapidAPI.non-executable: contains the nested sequencing data from SGD and GLAIVE that are hand-picked by human annotators from data synthetically generated using an LLM.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
nestful_wrapper-0.1.0.tar.gz
(12.9 MB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file nestful_wrapper-0.1.0.tar.gz.
File metadata
- Download URL: nestful_wrapper-0.1.0.tar.gz
- Upload date:
- Size: 12.9 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.9.23
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a81d40ddfee0c5b7ddb5f9c189a1ec1597a6343f812edf4773f4e33b1da14eee
|
|
| MD5 |
ac14767478341ba8463a9040395ea388
|
|
| BLAKE2b-256 |
835d4f2c77e000d66f38883e85441dfa1d58250e30a24350ea4109638579aa63
|
File details
Details for the file nestful_wrapper-0.1.0-py3-none-any.whl.
File metadata
- Download URL: nestful_wrapper-0.1.0-py3-none-any.whl
- Upload date:
- Size: 13.9 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.9.23
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e934b76b87f6ba4021d213e548e2db886d4d999a48156c693251276dd0e66b37
|
|
| MD5 |
3f9b659640bdd1029d56807d5f6ecc50
|
|
| BLAKE2b-256 |
7b17da3665da9dcfa03a649e5563172b6111660e7991448fc005ad8f8afefa76
|
