A python wrapper over NESTFUL data
Project description
NESTFUL
This is the official repository for NESTFUL.
- Paper Title: NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
- Link: https://arxiv.org/abs/2409.03797v2
- HuggingFace Data Link: https://huggingface.co/datasets/ibm-research/nestful
Data
We have shared the latest NESTFUL evaluation set under data_v2 dir.
nestful_data.jsonl: It has 1861 evaluation data for nested sequencing.executable_functions: Contains the implementation of all the functions in the benchmark.
The data_v1 directory includes the data for the previous version of the paper - link.
executable: contains data and spec with necessary information to execute them through RapidAPI.non-executable: contains the nested sequencing data from SGD and GLAIVE that are hand-picked by human annotators from data synthetically generated using an LLM.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
nestful_wrapper-0.0.0.tar.gz
(94.4 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file nestful_wrapper-0.0.0.tar.gz.
File metadata
- Download URL: nestful_wrapper-0.0.0.tar.gz
- Upload date:
- Size: 94.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.9.21
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1f84bb98f8342ff5dde8897506e4b751f31280458d45800da90a48e4f2469962
|
|
| MD5 |
813ca26246d30c0db403348c6c021f3b
|
|
| BLAKE2b-256 |
ffb5a96fe4fd4e0ba8f3de960289875bb42c1d2997269d565384734baa11aead
|
File details
Details for the file nestful_wrapper-0.0.0-py3-none-any.whl.
File metadata
- Download URL: nestful_wrapper-0.0.0-py3-none-any.whl
- Upload date:
- Size: 118.1 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.1.0 CPython/3.9.21
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f14c652047882349d8ce1e132f732563498074a079a4c6164c09bc98f8a8594e
|
|
| MD5 |
2980186bfa7dcdabc7f733101be37243
|
|
| BLAKE2b-256 |
795156b46cc5d021ba746594a11e916ca09eea82bc0136d2573d50b693140129
|
