AwesomeStream

AwesomeStream is a set of tools for creating a “stream server”. That is, a server which can store information about events that happen, and can query back those events in reverse-chronological order, sliced in interesting ways.

Example and Use Case

Say that you run a website like GitHub, where people interact in various different ways. People can create repositories, fork them, watch or unwatch repositories, add friends, etc. There are all kinds of things that a user can do on the site. Let’s look at how AwesomeStream can help.

First, we’ll set up a simple redis-based server:

>>> from awesomestream.backends import RedisBackend
>>> from awesomestream.jsonrpc import create_app, run_server
>>> backend = RedisBackend(
...     keys=['user', 'kind', 'repo'],
...     host='127.0.0.1',
...     port=6379
... )
>>>
>>> app = create_app(backend)
>>> run_server(app, 8080)

This simple script sets up a Redis-based AwesomeStream server–one that pays special attention to the ‘user’, ‘kind’, and ‘repo’ keys. This will make a bit more sense in a bit.

In another console, we’re going to instantiate a client.

>>> from awesomestream.jsonrpc import Client
>>> c = Client('http://127.0.0.1:8080/')

OK, now that we’ve set up our client, lets start logging user actions. Look, a user has just created a new repo!

>>> c.insert({
...     'kind': 'create-repo',
...     'repo': 17,
...     'user': 291,
...     'name': 'frist',
...     'description': 'This is my first repo ever!',
... })
>>>

But the user made a mistake, and named it ‘frist’ instead of ‘first’. So they go ahead and delete it:

>>> c.insert({
...     'kind': 'delete-repo',
...     'repo': 17,
...     'user': 291,
...     'reason': 'Made a typo :(',
... })
>>>

Then they give up and decide to watch another user’s repo instead:

>>> c.insert({'kind': 'watch', 'repo': 2842, 'user': 291, 'owner': 23})

And finally they add that user as a friend:

>>> c.insert({'kind': 'friend', 'user': 291, 'friend': 23})

That second user notices that someone is following them, and follows back:

>>> c.insert({'kind': 'friend', 'user': 23, 'friend': 291})

Now that we have data inserted into the stream server, we can query it to get back the full stream. Here’s how something like that might look:

>>> c.items()
[{'kind': 'friend', 'user': 23, 'friend': 291},
 {'kind': 'friend', 'user': 291, 'friend': 23},
 {'repo': 2842, 'owner': 23, 'kind': 'watch', 'user': 291},
 {'repo': 17, 'kind': 'delete-repo', 'reason': 'Made a typo :(', 'user': 291},
 {'repo': 17, 'kind': 'create-repo', 'user': 291, 'name': 'frist', 'description': 'This is my first repo ever!'}
]

As you can see, we got the entire stream back, in reverse chronological order. But let’s say we want to filter this out, to only see ‘friend’ requests. We can do that easily:

>>> c.items(kind='friend')
[{'kind': 'friend', 'user': 23, 'friend': 291},
 {'kind': 'friend', 'user': 291, 'friend': 23}
]

Notice that they are still in reverse chronological order. We can also combine our predicates, to get only friend requests made by a specific user.

>>> c.items(kind='friend', user=23)
[{'kind': 'friend', 'user': 23, 'friend': 291}]

But an extremely common case is that you want to see only your activity that is generated by your friends. With AwesomeStream, that’s simple:

>>> c.items(user=[23, 291])
[{'kind': 'friend', 'user': 23, 'friend': 291},
 {'kind': 'friend', 'user': 291, 'friend': 23},
 {'repo': 2842, 'owner': 23, 'kind': 'watch', 'user': 291},
 {'repo': 17, 'kind': 'delete-repo', 'reason': 'Made a typo :(', 'user': 291},
 {'repo': 17, 'kind': 'create-repo', 'user': 291, 'name': 'frist', 'description': 'This is my first repo ever!'}
]

As you can see, every user ID passed into that list is retrieved. By default, the items() function retrieves 20 items, but often times we’ll need to customize that. Here’s how that would look:

>>> c.items(user=[23, 291], start=1, end=3)
[{'kind': 'friend', 'user': 291, 'friend': 23},
 {'repo': 2842, 'owner': 23, 'kind': 'watch', 'user': 291}
]

Supported Backends

• In-Memory (mostly for testing)

• SQL

• Redis

Planned Support

• CouchDB

• Cassandra

Maturity

I’m writing this for eventual deployment on http://radiosox.com/, but have not yet deployed it in production. Do so at your own risk.

Requirements

Short Summary:

Use pip, and do pip install -U -r requirements.txt

Longer Summary:

Strictly speaking, the only requirement is simplejson. That being said, if you want redis support, you need redis installed. If you want SQL support, you need SQLAlchemy installed. If you want support for creating a WSGI app to expose this over HTTP, you’ll need werkzeug installed. Finally, if you want a simple, pure-python way of running that WSGI app, you’ll want to install cherrypy.