DZDBuffy 0.1.1.dev0

Versioning Caching kinda-Proxy for decoupling external responses

Buffy

Versioning Caching kinda-Proxy for decoupling external HTTP responses

Maintainer: Tim Bleimehl

status: alpha (WIP - do not use productive yet)

What is this?

Buffy is a server/client framework to buffer/cache your http requests. It decouples your dependency on external webservers that are not under your control. It manages downloads from slow and unreliable webservers in the background:

• Resume broken downloads
• Retry corrupted downloads
• Versioning of changing downloads

You can ignore any issues with external webservers and just focus on your application.

At the moment there is only a python client library. But the server has a REST API that can be consumed from any coding language. You are welcome to write a client in your language.

Example use

from buffy.buffypyclient import BuffyPyClient

# connect to buffy server
c = BuffyPyClient(ssl=False)

# create a request
req = c.create_request(
    url="https://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n0003.xml.gz"
)

# save requested file
req.download_response_content_to("/tmp/pubmed22n0003.xml.gz")

# or alternatively stream requested file
with open("/tmp/pubmed22n0003_2.xml.gz", "wb") as f:
    for chunk in req.download_response_content():
        f.write(chunk)

This is all it takes to request a file. Next time the webserver at ftp.ncbi.nlm.nih.go should be down the buffy client will just serve you the cached answer. Should your Buffy server be down, the client will fall back to direct downloading the request from the source.

Setup Server

docker run -v ${PWD}/buffy-server:/data -p 8008:8008 registry-gl.connect.dzd-ev.de:443/dzdpythonmodules/buffy:prod

Python Client API

todo

Server REST API

todo