Export Yahoo Fantasy Football data from historical records and live data.
Project description
TODO
- Data Loader:
- Get Data from the yahoo api
- Split accross seperate "Loaders" within Mage
- Export Raw Json to Supabase
- Transform Raw Json:
- Transform data from raw json to parsed json
- Export parsed json
- Transform Parsed Json:
- Transform parsed json to tabular form
- Export to public schema
Process
- Query yahoo api
- DONE
- TODO:
- Split across multiple "blocks" for each "live" api connection needed
- Could use multiple api keys to help with rate limiting and speed
- Once queried, data should load to database immediately
- Jsonb format in postgresql
- yahoo_data.raw_json
- Transformation 1
- Convert data to "parsed_json"
- Upload to yahoo_data.parsed_json
- Transformation 2
- Convert to tabular format
- Upload to yahoo_data.public
- Split across multiple "blocks" for each "live" api connection needed
Need to knows
- How to use the yahoo_api package created within Mage?
- Should I be using SQLAlchemy or could is suffice with psycopg (perferably v3, but can use v2 if needed)
- Should this be Async?
- Would multiple api calls/inserts affect ACID/locks?
- If so, probably should use ORM?
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
yahoo_export-0.1.8.tar.gz
(9.9 kB
view hashes)
Built Distribution
Close
Hashes for yahoo_export-0.1.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5ee4ff6be4d6dee7318a159239f5de29c209bddba4e481ce8e4551d7715d032 |
|
MD5 | 9fa804dd5ae6824e612fd9106e85e854 |
|
BLAKE2b-256 | fae7f4f4c3bfab011445fa20144750fdaaafd4b38f0c99389b6eef98c069c654 |