Skip to main content

Tools for exporting Theta blockchain data to CSV or JSON

Project description

Ethereum ETL

Build Status Join the chat at Telegram Discord

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.


Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3

Useful Links

Running Tests

> pip3 install -e .[dev,streaming]
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

  1. Install Docker

  2. Build a docker image

     > docker build -t ethereum-etl:latest .
     > docker image ls
  3. Run a container out of the image

     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p
     > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p
  4. Run streaming to console or Pub/Sub

     > docker build -t ethereum-etl:latest -f Dockerfile .
     > echo "Stream to console"
     > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
     > echo "Stream to Pub/Sub"
     > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum

Projects using Ethereum ETL

  • Google - Public BigQuery Ethereum datasets
  • Nansen by D5 - Analytics platform for Ethereum

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

theta-etl-0.1.26.tar.gz (301.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page