Export Yahoo Fantasy Football data from historical records and live data.
Project description
TODO
- Data Loader:
- Get Data from the yahoo api
- Split accross seperate "Loaders" within Mage
- Export Raw Json to Supabase
- Transform Raw Json:
- Transform data from raw json to parsed json
- Export parsed json
- Transform Parsed Json:
- Transform parsed json to tabular form
- Export to public schema
Process
- Query yahoo api
- DONE
- TODO:
- Split across multiple "blocks" for each "live" api connection needed
- Could use multiple api keys to help with rate limiting and speed
- Once queried, data should load to database immediately
- Jsonb format in postgresql
- yahoo_data.raw_json
- Transformation 1
- Convert data to "parsed_json"
- Upload to yahoo_data.parsed_json
- Transformation 2
- Convert to tabular format
- Upload to yahoo_data.public
- Split across multiple "blocks" for each "live" api connection needed
Need to knows
- How to use the yahoo_api package created within Mage?
- Should I be using SQLAlchemy or could is suffice with psycopg (perferably v3, but can use v2 if needed)
- Should this be Async?
- Would multiple api calls/inserts affect ACID/locks?
- If so, probably should use ORM?
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
yahoo_export-3.0.0.tar.gz
(9.6 kB
view hashes)
Built Distribution
Close
Hashes for yahoo_export-3.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d984033b19fc4bbb46904cd8ca19f833c46ef8ece5e0bfcfb4a0f5627f103df |
|
MD5 | 4dca526b5cd5a74e3a88605e9fa7bdea |
|
BLAKE2b-256 | 99175a923283b0f0e43c41d76d12a8f6b85aaeb8853f2f6da565a6ec0acad93d |