pullframe 0.1.0

pull based pandas dataframe syncing

pullframe

pull based pandas dataframe syncing

---

To reduce network consumption, it syncs dataframe from the other nodes only on demand. When your task is divide and conquer style, you should consider dask instead.

Features

• Once the cache has been synced, it will not call remotes. So cache's locality is 1.
• Ideal situations is that you need to read some dataframe multiple times on serveral nodes and the data frame should be updated frequently.
• Only unique str name is required configuration when you add a new dataframe on the system.
• No configuration, no operation is needed when a new node is added and a node is crashed and restored.
• No configuration, no operation makes it be easy to scale up in the cloud.

Communications

• Coordination via zookeeper
• Synchronize files via http POST

Start Service

$ uvicorn pullframe.sender:app

Example

Load / Save

from pullframe import pullframe

with pullframe(hosts, directory, sync_timeo 60.0) as pf:
    # set start as None if you want to load from the very beginning
    # set end as None if you want to load from the very ending
    df = pf.load(name, start: Optional[datetime], end: Optional[datetime])

    pf.save(name, df)

TODO

• Check cache discrepency/corruption between nodes.
• Stable backup using Amazon S3 / Google cloud storage.
• Replace zookeeper client to zake (fake kazoo client) during tests.

Requirements

• zookeeper
• the dataframe's index should be datetime
• linux
• python>=3.7
• python = "^3.7"
• pandas = "^1.0.0"
• tables = "^3.6.1"
• fastapi = "^0.58.0"
• aiofiles = "^0.5.0"
• kazoo = "^2.7.0"

Free software: MIT License

Credits

• This package was created with Cookiecutter
• Also was copied and modified from the audreyr/cookiecutter-pypackage project template.