Parallel computing framework
Project description
link.parallel is a database agnostic query system.
See documentation for more informations.
Installation
pip install link.parallel
Features
parallel loops interface with IPython and multiprocessing drivers
Map/Reduce middleware
Examples
Create your map/reduce functions:
from b3j0f.task import register_task
@register_task('mymap')
def mymap(mapper, item):
if item['i'] < 5:
mapper.emit('i_lt_5', item)
elif item['i'] > 5:
mapper.emit('i_gt_5', item)
else:
mapper.emit('i_eq_5', item)
@register_task('myreduce')
def myreduce(key, values):
return (key, len(values))
Get input data and pass it to the middleware:
from link.middleware.core import Middleware
mapreduce = Middleware.get_middleware_by_uri(
'mapreduce+ipython:///test/classify?mapcb=mymap&reducecb=myreduce'
)
items = # load items
result = dict(mapreduce(items))
print(result)
Donating
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
link.parallel-1.0.tar.gz
(6.4 kB
view hashes)
Built Distribution
Close
Hashes for link.parallel-1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fba90ab281531dbc8bccc4c9745dfc36557e25ac7c938e73ec3cb7c9249b7485 |
|
MD5 | 0fd427ba53e122f536d3a5476c589f3e |
|
BLAKE2b-256 | 1715fb0b70ac8f46777569ef6d5e8762d469acaf90bf0d36e88c2e9dd16e7360 |