celo-etl 0.0.1

Tools for exporting Celo blockchain data to CSV or JSON

Celo ETL

Celo ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Quickstart

Install Celo ETL:

pip3 install celo-etl

Export blocks and transactions (Schema, Reference):

> celoetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--provider-uri https://alfajores-forno.celo-testnet.org --blocks-output blocks.csv --transactions-output transactions.csv

Export ERC20 and ERC721 transfers (Schema, Reference):

> celoetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> celoetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

---

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install celo-etl[streaming]
> celoetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 celoetl.py

Useful Links

• Schema
• Command Reference
• Documentation
• Exporting the Blockchain
• Querying in Amazon Athena
• Querying in Google BigQuery
• Querying in Kaggle
• Airflow DAGs
• Postgres ETL

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

1. Install Docker https://docs.docker.com/install/

2. Build a docker image

    > docker build -t celo-etl:latest .
    > docker image ls
   

3. Run a container out of the image

    > docker run -v $HOME/output:/celo-etl/output celo-etl:latest export_all -s 0 -e 200000 -b 50000 -p https://alfajores-forno.celo-testnet.org
    > docker run -v $HOME/output:/celo-etl/output celo-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://alfajores-forno.celo-testnet.org