Python Library to connect to the NEAR Lake S3 and stream the data
Project description
NEAR Lake Framework for Python
Available in programming languages: Rust | Javascript | Python3
NEAR Lake Framework is a small library companion to NEAR Lake. It allows you to build your own indexer that subscribes to the stream of blocks from the NEAR Lake data source and create your own logic to process the NEAR Protocol data.
Official NEAR Lake Framework launch announcement has been published on the NEAR Gov Forum Greetings from the Data Platform Team! We are happy and proud to announce an MVP release of a brand new word in indexer building approach - NEAR Lake Framework.
Example
import asyncio
import os
from near_lake_framework import LakeConfig, streamer
async def main():
config = LakeConfig.mainnet()
config.start_block_height = 69130938
config.aws_access_key_id = os.getenv("AWS_ACCESS_KEY_ID")
config.aws_secret_key = os.getenv("AWS_SECRET_ACCESS_KEY")
stream_handle, streamer_messages_queue = streamer(config)
while True:
streamer_message = await streamer_messages_queue.get()
print(
f"Received Block #{streamer_message.block.header.height} from Lake Framework"
)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
How to use
Dependencies
Install near-lake-framework
$ pip3 install near-lake-framework
Credentials
To be able to access the data from NEAR Lake you need to provide credentials. Please, see the Credentials article
Configuration
Everything should be configured before the start of your indexer application via LakeConfig
struct.
Available parameters:
s3_bucket_name: str
- provide the AWS S3 bucket name (near-lake-testnet
,near-lake-mainnet
or yours if you run your own NEAR Lake)s3_region_name: str
- provide the region for AWS S3 bucketstart_block_height: BlockHeight
- block height to start the stream fromblocks_preload_pool_size: int
- provide the number of blocks to preload (default: 200)
Cost estimates
TL;DR approximately $18.15 per month (for AWS S3 access, paid directly to AWS) for the reading of fresh blocks
Explanation:
Assuming NEAR Protocol produces accurately 1 block per second (which is really not, the average block production time is 1.3s). A full day consists of 86400 seconds, that's the max number of blocks that can be produced.
According the Amazon S3 prices list
requests are charged for $0.005 per 1000 requests and get
is charged for $0.0004 per 1000 requests.
Calculations (assuming we are following the tip of the network all the time):
86400 blocks per day * 5 requests for each block / 1000 requests * $0.0004 per 1k requests = $0.173 * 30 days = $5.19
Note: 5 requests for each block means we have 4 shards (1 file for common block data and 4 separate files for each shard)
And a number of list
requests we need to perform for 30 days:
86400 blocks per day / 1000 requests * $0.005 per 1k list requests = $0.432 * 30 days = $12.96
$5.19 + $12.96 = $18.15
The price depends on the number of shards
Publishing to PyPi
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for near-lake-framework-0.0.6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1e3fd5a256dc8ceb8a85a0b2b16f59b227dad22c0033aae8157c0cce5e6717f |
|
MD5 | f8afac383c07a8048665ecbccd14fb9b |
|
BLAKE2b-256 | 62474e159c9dbf3b3525025f1e5a13888cd05dd95aff1de4fa865a8b3a689801 |
Hashes for near_lake_framework-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b46d496296e52bee2e78ea8bb5a7cc64353b413213a78b862f8c394bcdfa401d |
|
MD5 | e76bd5073ceac2cd48fb6477c49de1e2 |
|
BLAKE2b-256 | 7e5a704b42bc5ab07dc0fda776304885f2625770c467738928c450eac10db1dd |