Skip to main content

Tools for exporting Polygon blockchain data to CSV or JSON

Project description

Polygon ETL CLI

Polygon ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Full documentation available here.

Quickstart

Install Polygon ETL CLI:

pip3 install polygon-etl

Export blocks, actions and logs (Schema, Reference):

> polygonetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--provider-uri grpcs://api.mainnet.polygon.one:443 \
--blocks-output blocks.csv --transactions-output transactions.csv

Stream blocks, actions and logs to console (Reference):

> pip3 install polygon-etl[streaming]
> polygonetl stream --start-block 500000 -e block,action,log --log-file log.txt \
--provider-uri grpcs://api.mainnet.polygon.one:443

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 polygonetl.py

Useful Links

Running Tests

> pip3 install -r ../requirements_test.txt -e .[streaming]
> export POLYGONETL_RUN_SLOW_TESTS=true
> export POLYGONETL_PROVIDER_URI=grpcs://api.mainnet.polygon.one:443
> pytest -vv

Running Tox Tests

> pip3 install -r ../requirements_test.txt
> export POLYGONETL_RUN_SLOW_TESTS=true
> export POLYGONETL_PROVIDER_URI=grpcs://api.mainnet.polygon.one:443
> tox

Running in Docker

  1. Install Docker https://docs.docker.com/install/

  2. Build a docker image

     > docker build -t polygon-etl:latest .
     > docker image ls
    
  3. Run a container out of the image

     > docker run -v $HOME/output:/polygon-etl/output polygon-etl:latest export_blocks -s 1 -e 5499999 -b 1000 -o out
    
  4. Run streaming to console or Pub/Sub

     > docker build -t polygon-etl:latest -f Dockerfile .
     > echo "Stream to console"
     > docker run polygon-etl:latest stream --start-block 500000 --log-file log.txt
     > echo "Stream to Pub/Sub"
     > docker run -v /path_to_credentials_file/:/polygon-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/polygon-etl/credentials_file.json polygon-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/mainnet
    

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

polygon-etl-0.5.0.tar.gz (55.1 kB view details)

Uploaded Source

File details

Details for the file polygon-etl-0.5.0.tar.gz.

File metadata

  • Download URL: polygon-etl-0.5.0.tar.gz
  • Upload date:
  • Size: 55.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.6

File hashes

Hashes for polygon-etl-0.5.0.tar.gz
Algorithm Hash digest
SHA256 8ad5227a3ac25cb6a908bc9f63db4c68cb4d9f312c59532de2393381ffb9dd43
MD5 1641558c57c4e9338e13f0e31b1b3a7f
BLAKE2b-256 4fca1f6c8d69e1927d298c124f3ac249438985637e24ede4e6d75aeb78a20bd7

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page