Ethereum ETL

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Quickstart

Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/239a1d18eba14f0f9dc1c882de0dc872

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/239a1d18eba14f0f9dc1c882de0dc872

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 ethereumetl.py

Useful Links

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

Install Docker https://docs.docker.com/install/

Build a docker image

 > docker build -t ethereum-etl:latest .
 > docker image ls

Run a container out of the image

 > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
 > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io

Run streaming to console or Pub/Sub

 > docker build -t ethereum-etl:latest -f Dockerfile .
 > echo "Stream to console"
 > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
 > echo "Stream to Pub/Sub"
 > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum

Projects using Ethereum ETL

Google - Public BigQuery Ethereum datasets
Nansen by D5 - Analytics platform for Ethereum

Metadata

Release files for theta-etl 0.1.41

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for theta-etl 0.1.41
File	Size	Uploaded
theta-etl-0.1.41.tar.gz	301.3 kB	Oct 22, 2020	Details

Release files / theta-etl-0.1.41.tar.gz

Download URL	theta-etl-0.1.41.tar.gz
Size	301.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`c7db38c9e03221981c4ce3b9d646a320c80ff8e51f6c10d3ab8d762c7f2a2f92`
BLAKE2b-256 checksum How to use checksums	`852d3659ade2698e1dc80a4b149c02aea9e2d689a6976b4aef49584a6f4ebaa5`
Upload date	Oct 22, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.48.0 CPython/3.7.3`