Smart framework to interact with and fetch data from REST APIs.
Project description
AnyData
AnyData is a smart framework to interact with and fetch data from REST APIs. With AnyData you can define endpoint collections to reduce boilerplate code and improve code readability, while also accelerating development with OpenAPI (Swagger) parsers and Smart Functions.
Install
AnyData is available through PyPI. To use Smart Functions see the installation with optional dependencies at Smart Functions.
pip install anydata
DataAPI
DataAPI
is a collection of Endpoint
s that can be instantiated from a REST API base URL or directly from the API OpenAPI documentation.
→ From Base URL
from anydata import DataAPI
# Instantiate a DataAPI object
valet_api = DataAPI(base_url="https://www.bankofcanada.ca/valet")
# Set parameters to be shared across endpoints in this collection
valet_api.set_shared_params(params={"format": "json", "start_date": "2024-01-01"})
# Add endpoints
valet_api.add_endpoint("/observations/FXUSDCAD/{format}", method="GET", alias="USD_to_CAD")
valet_api.add_endpoint("/observations/FXCADUSD/{format}", method="GET", alias="CAD_to_USD")
At this point the data can be fetched directly to a DataFrame
object.
valet_api["USD_to_CAD"].to_pandas()
Or retrieved in its raw format.
valet_api["CAD_to_USD"].request().json()
{'terms': {'url': 'https://www.bankofcanada.ca/terms/'},
'seriesDetail': {'FXCADUSD': {'label': 'CAD/USD',
'description': 'Canadian dollar to US dollar daily exchange rate',
'dimension': {'key': 'd', 'name': 'Date'}}},
'observations': [{'d': '2024-01-02', 'FXCADUSD': {'v': '0.7510'}},
{'d': '2024-01-03', 'FXCADUSD': {'v': '0.7487'}},
{'d': '2024-01-04', 'FXCADUSD': {'v': '0.7488'}},
{'d': '2024-01-05', 'FXCADUSD': {'v': '0.7492'}},
...
{'d': '2024-05-28', 'FXCADUSD': {'v': '0.7332'}}]}
→ From OpenAPI Specification
A DataAPI
collection can also be instantiated from an OpenAPI specification or Swagger Documentation.
valet_api = DataAPI.from_openapi('https://www.bankofcanada.ca/valet/static/swagger/api-en.yml')
valet_api.endpoints()
All available endpoints will be added as Endpoint
objects with corresponding REST API methods and path parameters, if any.
[('/fx_rss', Endpoint(method="get", endpoint="/fx_rss")),
('/fx_rss/{seriesNames}',
Endpoint(method="get", endpoint="/fx_rss/{seriesNames}", path_params={'seriesNames': None})),
('/groups/{groupName}/{format}',
Endpoint(method="get", endpoint="/groups/{groupName}/{format}", path_params={'groupName': None, 'format': None})),
('/lists/{type}/{format}',
Endpoint(method="get", endpoint="/lists/{type}/{format}", path_params={'type': None, 'format': None})),
('/observations/group/{groupName}/{format}',
Endpoint(method="get", endpoint="/observations/group/{groupName}/{format}", path_params={'groupName': None, 'format': None})),
('/observations/{seriesNames}/{format}',
Endpoint(method="get", endpoint="/observations/{seriesNames}/{format}", path_params={'seriesNames': None, 'format': None})),
('/series/{seriesName}/{format}',
Endpoint(method="get", endpoint="/series/{seriesName}/{format}", path_params={'seriesName': None, 'format': None}))]
Which can then be called directly from the collection.
valet_api["/observations/{seriesNames}/{format}"].to_pandas(
params={"seriesNames": "FXUSDCAD", "format": "json"}
)
As a requests.Session
subclass, Endpoint
s can also be used as a context managers.
valet_api.set_shared_params(params={"format": "json"})
with valet_api["/observations/{seriesNames}/{format}"] as e:
usd_to_cad_df = e.to_pandas(params={"seriesNames": "FXUSDCAD"})
cad_to_usd_df = e.to_pandas(params={"seriesNames": "FXCADUSD"})
This will open and close a connection once to perform both API calls.
Endpoint
An Endpoint
can also be instantiated independently and still benefit from methods such as the to_pandas()
parser.
from anydata import Endpoint
usd_to_cad = Endpoint(
base_url="https://www.bankofcanada.ca/valet",
endpoint="/observations/FXUSDCAD/json",
method="GET"
)
usd_to_cad.to_pandas()
Smart Functions
Smart Functions leverage Large Language Models to interact with DataAPI
objects instantiated from an OpenAPI specification, so you can spend more time with the data, and less time going through the API documentation.
To enable Smart Functions, you need to perform an install that brings the necessary extra dependencies:
pip install 'anydata[guidance]'
You also need to set any API Key necessary to interact with remote LLMs. The following examples assume that an OpenAI API Key is set as an environment variable. You can easily do that with a .env file and Python dotenv
:
# .env file in the project directory
OPENAI_API_KEY=sk-***********************
from dotenv import load_dotenv
load_dotenv()
Smart Add Endpoint
The smart_add_endpoint()
method instantiates an Endpoint
that fulfill a prompt for data in natural language.
from anydata import DataAPI
from anydata.engine.models import OpenAI
# Instantiate from the Open API specification
valet_api = DataAPI.from_openapi('https://www.bankofcanada.ca/valet/static/swagger/api-en.yml')
# Load Language Model (OpenAI, Transformers, LamaCpp, Anthropic, or others...)
valet_api.set_lm(OpenAI('gpt-3.5-turbo'))
# Prompt for data from the API to instantiate an endpoint
valet_api.smart_add_endpoint(
'Data for Canadian Dollars to Japanese Yen conversion from January 2020 onwards',
alias='CAD_to_JPY'
)
# Fetch to pandas DataFrame
valet_api['CAD_to_JPY'].to_pandas()
The instantiated endpoint contains the parameter to fulfill the prompt.
valet_api['CAD_to_JPY']
Endpoint(method="get", endpoint="/observations/{seriesNames}/{format}", params={'start_date': '2020-01-01'}, path_params={'seriesNames': 'FXCADJPY', 'format': 'json'})
The performance of this feature is highly dependent on the quality of the API documentation provided, and it works with any Large Language Model supported by the guidance framework.
Contribution
AnyData is a project on its very early stages that is open to contributions. I will add a backlog of features that I plan to implement and the discussion sections is wide open to any suggestion - including reinterpretations of existing functionalities. A dedicated contributions documentation should also be coming up soon.
If you want to be an early contributor, feel free to fork the repository and open a PR.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.