Tools for exporting Celo blockchain data to CSV or JSON
Project description
Celo ETL
Celo ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.
Quickstart
Install Celo ETL:
pip3 install celo-etl
Export blocks and transactions (Schema, Reference):
> celoetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--provider-uri https://alfajores-forno.celo-testnet.org --blocks-output blocks.csv --transactions-output transactions.csv
Export ERC20 and ERC721 transfers (Schema, Reference):
> celoetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv
Export traces (Schema, Reference):
> celoetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv
Stream blocks, transactions, logs, token_transfers continually to console (Reference):
> pip3 install celo-etl[streaming]
> celoetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt
Find other commands here.
For the latest version, check out the repo and call
> pip3 install -e .
> python3 celoetl.py
Useful Links
- Schema
- Command Reference
- Documentation
- Exporting the Blockchain
- Querying in Amazon Athena
- Querying in Google BigQuery
- Querying in Kaggle
- Airflow DAGs
- Postgres ETL
Running Tests
> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> pytest -vv
Running Tox Tests
> pip3 install tox
> tox
Running in Docker
-
Install Docker https://docs.docker.com/install/
-
Build a docker image
> docker build -t celo-etl:latest . > docker image ls
-
Run a container out of the image
> docker run -v $HOME/output:/celo-etl/output celo-etl:latest export_all -s 0 -e 200000 -b 50000 -p https://alfajores-forno.celo-testnet.org > docker run -v $HOME/output:/celo-etl/output celo-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://alfajores-forno.celo-testnet.org
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
celo-etl-0.0.1.tar.gz
(323.0 kB
view hashes)
Built Distribution
celo_etl-0.0.1-py3.7.egg
(907.0 kB
view hashes)