Hyperstream is a large-scale, flexible and robust software package for processing streaming data
Project description
HyperStream
Hyperstream is a large-scale, flexible and robust software package for processing streaming data.
HyperStream homepage
Tutorial notebooks
Gitter chat room
Developer documentation
Hyperstream overcomes the limitations of other computational engines and provides high-level interfaces to execute complex nesting, fusion, and prediction both in online and offline forms in streaming environments. Although developed specifically for SPHERE, Hyperstream is a general purpose tool that is well-suited for the design, development, and deployment of algorithms and predictive models in a wide space of sequential predictive problems.
This software has been designed from the outset to be domain-independent, in order to provide maximum value to the wider community. Key aspects of the software include the capability to create complex interlinked workflows, and a computational engine that is designed to be “compute-on-request”, meaning that no unnecessary resources are used.
Installation
Docker images
If you do not want to install all the packages separately you can use our Docker bundle available here.
Local machine
Install via pip
pip install hyperstream python -c 'from hyperstream import HyperStream'
To get the latest version
pip install -U git+git://github.com/IRC-SPHERE/HyperStream.git#egg=hyperstream pip install -r requirements.txt
Or clone the repository
git clone git@github.com:IRC-SPHERE/HyperStream.git
cd HyperStream
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
python -c 'from hyperstream import HyperStream'
Additionally, one of the requirements to run Hyperstream is a MongoDB server. By default, Hyperstream tries to connect to the port 27017 on the localhost.
To see the installation steps of MongoDB go to the official documentation. E.g. in a Debian OS it is possible to install with the following command
sudo apt-get install mongodb
Once the MongoDB server is installed, it can be started with the following command
service mongod start
Running tests
Run the following command
nosetests
Note that for the MQTT logging test to succeed, you will need to have an MQTT broker running (e.g. Mosquitto). For example:
docker run -ti -p 1883:1883 -p 9001:9001 toke/mosquitto
or on OSX you will need pidof and mosquitto:
brew install pidof brew install mosquitto brew services start mosquitto
Tutorials
The following tutorials show how to use HyperStream in a step-by-step guide.
from hyperstream import HyperStream, StreamId, TimeInterval
from hyperstream.utils import utcnow, UTC
from datetime import timedelta
hs = HyperStream(loglevel=20)
M = hs.channel_manager.memory
T = hs.channel_manager.tools
clock = StreamId(name="clock")
clock_tool = T[clock].window().last().value()
ticker = M.get_or_create_stream(stream_id=StreamId(name="ticker"))
now = utcnow()
before = (now - timedelta(seconds=30)).replace(tzinfo=UTC)
ti = TimeInterval(before, now)
clock_tool.execute(sources=[], sink=ticker, interval=ti, alignment_stream=None)
print(list(ticker.window().tail(5)))
[StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>))]
HyperStream Viewer
The HyperStream Viewer is a python/Flask web-app for interacting with HyperStream. In order to keep HyperStream to a minimum, this web-app is released as a separate repository that takes the core as a dependency.
License
This code is released under the MIT license.
Acknowledgements
This work has been funded by the UK Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/K031910/1 - “SPHERE Interdisciplinary Research Collaboration”.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.