Parallel computing framework
Project description
link.parallel is a database agnostic query system.
See documentation for more informations.
Installation
pip install link.parallel
Features
parallel loops interface with IPython and multiprocessing drivers
Map/Reduce middleware
Examples
Create your map/reduce functions:
from b3j0f.task import register_task
@register_task('mymap')
def mymap(mapper, item):
if item['i'] < 5:
mapper.emit('i_lt_5', item)
elif item['i'] > 5:
mapper.emit('i_gt_5', item)
else:
mapper.emit('i_eq_5', item)
@register_task('myreduce')
def myreduce(key, values):
return (key, len(values))
Get input data and pass it to the middleware:
from link.middleware.core import Middleware
mapreduce = Middleware.get_middleware_by_uri(
'mapreduce+ipython:///test/classify?mapcb=mymap&reducecb=myreduce'
)
items = # load items
result = dict(mapreduce(items))
print(result)
Donating
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
link.parallel-0.4.tar.gz
(6.1 kB
view hashes)
Built Distribution
Close
Hashes for link.parallel-0.4-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8d9f3770ba3d18ca41899977ff66730b94abaf83e9a0609fe211c911e2f0289 |
|
MD5 | a6f0674250f8ffa6276d422689069cc9 |
|
BLAKE2b-256 | bd64d301584f0f517a111d0a6b4cc9803c3c071c61cb1cc102a3c4e8b5fc37be |