Export Yahoo Fantasy Football data from historical records and live data.
Project description
TODO
- Data Loader:
- Get Data from the yahoo api
- Split accross seperate "Loaders" within Mage
- Export Raw Json to Supabase
- Transform Raw Json:
- Transform data from raw json to parsed json
- Export parsed json
- Transform Parsed Json:
- Transform parsed json to tabular form
- Export to public schema
Process
- Query yahoo api
- DONE
- TODO:
- Split across multiple "blocks" for each "live" api connection needed
- Could use multiple api keys to help with rate limiting and speed
- Once queried, data should load to database immediately
- Jsonb format in postgresql
- yahoo_data.raw_json
- Transformation 1
- Convert data to "parsed_json"
- Upload to yahoo_data.parsed_json
- Transformation 2
- Convert to tabular format
- Upload to yahoo_data.public
- Split across multiple "blocks" for each "live" api connection needed
Need to knows
- How to use the yahoo_api package created within Mage?
- Should I be using SQLAlchemy or could is suffice with psycopg (perferably v3, but can use v2 if needed)
- Should this be Async?
- Would multiple api calls/inserts affect ACID/locks?
- If so, probably should use ORM?
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
yahoo_export-3.0.1.tar.gz
(9.7 kB
view hashes)
Built Distribution
Close
Hashes for yahoo_export-3.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ccc883f3848b3a4237551d5144cf50e902e781483be68c6012d0fbd6ccbca30 |
|
MD5 | 67152262cba0e1dd5c7aed18ffe1b6f1 |
|
BLAKE2b-256 | 966fa4ab990d6e021ed040ef12c5df24adde46c1a318fa8a07a7632797769e03 |