Tools for exporting XDC blockchain data to CSV or JSON
Project description
XDC ETL
XDC ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.
Do you just want to query XDC data right away? Use the public dataset in BigQuery.
Full documentation available here.
Quickstart
Install XDC ETL:
pip3 install xdc-etl
Export blocks and transactions (Schema, Reference):
> xdcetl export_blocks_and_transactions --start-block 0 --end-block 100000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri Https://rpc.XDC.org
Export ERC20 and ERC721 transfers (Schema, Reference):
> xdcetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv
Export traces (Schema, Reference):
> xdcetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv
Stream blocks, transactions, logs, token_transfers continually to console (Reference):
> pip3 install xdc-etl[streaming]
> xdcetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri Https://rpc.XDC.org
Find other commands here.
For the latest version, check out the repo and call
> pip3 install -e .
> python3 xdcetl.py
Running Tests
> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> export PROVIDER_URL=<your_porvider_uri>
> pytest -vv
Running Tox Tests
> pip3 install tox
> tox
Running in Docker
-
Install Docker: https://docs.docker.com/get-docker/
-
Build a docker image
> docker build -t xdc-etl:latest . > docker image ls
-
Run a container out of the image
> docker run -v $HOME/output:/xdc-etl/output xdc-etl:latest export_all -s 0 -e 5499999 -b 100000 -p Https://rpc.XDC.org > docker run -v $HOME/output:/xdc-etl/output xdc-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p Https://rpc.XDC.org
-
Run streaming to console or Pub/Sub
> docker build -t xdc-etl:latest . > echo "Stream to console" > docker run xdc-etl:latest stream --start-block 500000 --log-file log.txt > echo "Stream to Pub/Sub" > docker run -v /path_to_credentials_file/:/xdc-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/xdc-etl/credentials_file.json xdc-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum
If running on Apple M1 chip add the --platform linux/x86_64
option to the build
and run
commands e.g.:
docker build --platform linux/x86_64 -t xdc-etl:latest .
docker run --platform linux/x86_64 xdc-etl:latest stream --start-block 500000
Projects using Ethereum ETL
Useful Links on Orginal ETH ETL
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.