Export Yahoo Fantasy Football data from historical records and live data.
Project description
TODO
- Data Loader:
- Get Data from the yahoo api
- Split accross seperate "Loaders" within Mage
- Export Raw Json to Supabase
- Transform Raw Json:
- Transform data from raw json to parsed json
- Export parsed json
- Transform Parsed Json:
- Transform parsed json to tabular form
- Export to public schema
Process
- Query yahoo api
- DONE
- TODO:
- Split across multiple "blocks" for each "live" api connection needed
- Could use multiple api keys to help with rate limiting and speed
- Once queried, data should load to database immediately
- Jsonb format in postgresql
- yahoo_data.raw_json
- Transformation 1
- Convert data to "parsed_json"
- Upload to yahoo_data.parsed_json
- Transformation 2
- Convert to tabular format
- Upload to yahoo_data.public
- Split across multiple "blocks" for each "live" api connection needed
Need to knows
- How to use the yahoo_api package created within Mage?
- Should I be using SQLAlchemy or could is suffice with psycopg (perferably v3, but can use v2 if needed)
- Should this be Async?
- Would multiple api calls/inserts affect ACID/locks?
- If so, probably should use ORM?
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
yahoo_export-2.0.0.tar.gz
(10.3 kB
view hashes)
Built Distribution
Close
Hashes for yahoo_export-2.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33baace3340c747bf38b09b613021aaa8c46adc17a47a55a0779399d5e4e4491 |
|
MD5 | ab4234f5295b3e996c68422ae59735b5 |
|
BLAKE2b-256 | 2eed32a3eda7d2a4870081e3c7c7c5af8f12c17b6717bccbea6a4efcf43232f5 |