tidehunter 0.1.6a

HTTP streaming toolbox with flow control.

TideHunter
==========

HTTP streaming toolbox with flow control, written in Python.

.. image:: https://travis-ci.org/amoa/tidehunter.png?branch=master
:target: https://travis-ci.org/amoa/tidehunter

.. image:: https://coveralls.io/repos/amoa/tidehunter/badge.png?branch=master
:target: https://coveralls.io/r/amoa/tidehunter?branch=master

.. image:: https://pypip.in/d/tidehunter/badge.png?
:target: https://pypi.python.org/pypi/tidehunter

Background
----------

This project was born for a need in my day time job at Addictive Mobility (Addictive Tech Corp.). Before this project was open sourced, it went through two major overhauls. I briefly mentioned the techniques I used in previous versions in a `blog post <http://runzhou.li/blog/2013/07/02/tame-py-curl/>`_.

Highlights
----------

- Accurate quota limit - total control over your stream quota.
- An instant off switch - when sht hits the fan and you don't want to crash your process.
- Redis backed control tools - semi-persisted, fast, and scalable.
- Core mechanisms based on the solid cURL and PycURL - inherits the built-in goodness (gzip support and more).
- OAuth support based on python-oauth2 - see `this demo <https://github.com/amoa/tidehunter/blob/master/demo/five_tweets.py>`_ in action.

Installation
------------

.. code-block:: bash

$ pip install tidehunter

Or to update:

.. code-block:: bash

$ pip install tidehunter --upgrade

Note: the package will install all Python dependencies for you. However you need to have both cURL (the headers from dev package are also required for PycURL) and Redis installed.

Usage
-----

Example 1 (with limit):
~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: python

from tidehunter.stream import StateCounter, Queue, Hunter

# The state machine and record counter (state counter)
sc = StateCounter(key='demo_sc', host='localhost', port=6379, db=0)

# The data queue
q = Queue(key='demo_q', host='localhost', port=6379, db=0)

# The Hunter!
conf = {
'url': 'https://httpbin.org/stream/100',
'limit': 5, # desired limit
'delimiter': '\n'
}
h = Hunter(conf=conf, sc=sc, q=q)

# Start streaming
h.tide_on()

# Consume the data which should be in the data queue now
while len(q):
print q.get() # profit x 5

# You can re-use the same Hunter object, and add a one-time limit
h.tide_on(limit=1) # this time we only want one record

assert len(q) == 1 # or else there's a bug, create an issue!

print q.get() # more profit


Example 2 (without limit):
~~~~~~~~~~~~~~~~~~~~~~~~~~

Assume you have a process running the following code:

.. code-block:: python

from tidehunter.stream import StateCounter, Queue, Hunter

# The state machine and record counter (state counter)
sc = StateCounter(key='demo_sc', host='localhost', port=6379, db=0)

# The data queue
q = Queue(key='demo_q', host='localhost', port=6379, db=0)

# The Hunter!
conf = {'url': 'https://some.forever.streaming.api.endpoint'}
h = Hunter(conf=conf, sc=sc, q=q)

# Start streaming, FOREVA
h.tide_on()


You can delegate the responsibility of data consumption and stream control to another process:

.. code-block:: python

from tidehunter.stream import StateCounter, Queue

# The SAME state machine and record counter (state counter)
sc = StateCounter(key='demo_sc', host='localhost', port=6379, db=0)

# The SAME data queue
q = Queue(key='demo_q', host='localhost', port=6379, db=0)

while sc.started():
data = q.get() # dequeue and do something with data
# ...

if SHT_HITS_THE_FAN:
sc.stop() # instant off switch, end of while loop, as well as the process above



See demo_ for more examples.

.. _demo: https://github.com/amoa/tidehunter/tree/master/demo

Test (Unit Tests)
-----------------

The tests are done through Travis-CI already.

However, running the full test within your local environment is just three lines, provided that you have Redis installed and running:

.. code-block:: bash

$ pip install -r requirements.txt
$ pip install -r test_requirements.txt
$ nosetests --with-coverage --cover-package=tidehunter

Documentation
-------------

Coming up very soon!

License
-------

Copyright (c) 2013 Addictive Tech Corp., under The MIT License (MIT). See the full LICENSE_.

.. _LICENSE: https://github.com/amoa/tidehunter/blob/master/LICENSE

.. :changelog:

Changelog
---------

0.1.6a (2013-07-22)
++++++++++++++++++

- Minor: Massive update to README.rst
- Minor: Fixed PyPi rendering of long description.


0.1.5 (2013-07-22)
++++++++++++++++++

- NEW: Hunter.tide_on() now accepts an optional limit parameter for on the fly limit adjustment. The adjustment is not permanent, meaning if you want to reuse the same Hunter object, the old limit (or default None) is in effect.
- Major: Fixed a potential issue of Hunter puts in more records than desired limit.
- Minor: Added temp Basic Auth test case (no stream, need to find a better source).


0.1.3 (2013-07-13)
++++++++++++++++++

- Major: use the great httpbin.org (by Kenneth Reitz) for unit test now.
- Major: auth (oauth or basic) is no longer required, as long as the target stream server supports access without auth.


0.1.2 (2013-07-12)
++++++++++++++++++

- Minor: include CHANGES (changelog) to be shown on PyPi.
- Minor: use with statement to open files for setup.py.
- Minor: added the first `demo <https://github.com/amoa/tidehunter/tree/master/demo>`_.


0.1.1 (2013-07-12)
++++++++++++++++++

- Minor: clean up setup.py to ensure requirements are installed/updated.


0.1.0 (2013-07-12)
++++++++++++++++++

- Initial release