cocotask 0.2.1.1

An unified framework to build task queue on Rabbitmq | Kafka | Redis

# cocotask
### Build task queue on either Rabbitmq, Kafka or Redis! Simple! Easy! and FAST!!!!

**Why creating this framework?**
* Most task queues using Rabbitmq/Kafka/Redis are doing the same thing, but there is no unified wrapper. It's a waste of time to write code for different MQs if all you need is a task queue to distribute jobs
* No need to worry about using Rabbitmq, Kafka or Redis. Just pick one and go! It's simply a few lines of config changes in future if you want to switch to a different underlying system
* Hide all details for connecting/subscribing/publishing/etc. You can just use the same API for any of Kafka/Rabbitmq/Redis!
* Minimize the efforts for team members to handcraft the code for exchange/queue handling
* Team members should focus on how to handle messages
* There lacks good mananger tool/lib to handle creating multiple consumers (although it's simple)
* Celery is the original idea, but it doesn't support Kafka and Windows platform. It's also tightly coupled with Python alone.
<hr>

## Test

1. install rabbitmq or kafka on local machine (either docker or pure rabbitmq).
- Rabbitmq: https://www.rabbitmq.com/
- Kafka: https://kafka.apache.org/quickstart (for kafka, you have to manually create a topic named `test_topic_1` in order to run the test. In order to try multiple consumers, you need to set partitions to 2 or above, not 1)
- Redis: https://redis.io/download

2. make sure it's running Python 3.5 above

3. pip install pika kafka-python redis jsmin

4. pip install cocotask

5. now you have cocotask setup on your machine. To test, you can go to the ./test folder
- Run: `python producer_test.py` (this will post a string. Code is very simple)
- In another window, under ./test, run: **`cocotask ./config.json userworkers TestWorker 4`**
you'll see the worker starts and process 1 message we just posted.

6. So the key part is how we use cocotask command tool. the parameters are:
- config_path: path to config file
- module_name: module of your customized worker class
- class_name: the class name of your own worker class (in the example above, it's userworkers.TestWorker, so the module name is userworkers, the class name is TestWorker)
- number of workers: the total number of customer workers
- logginglevel(optional): python logging level INFO/DEBUG/etc.
- modulepath(optional): the relative path to find the module you defined. Default is '.'

7. Develop your own worker class and try

## Development

**Build your own worker**

```
from cocotask import CocoBaseWorker

class TestWorker(CocoBaseWorker):

def process(self, body):
print(body)
```
Check userworkers/test_worker.py for reference.

**Post a message**
```
from cocotask import CocoProducerManager as pm
import json

with open('config.json', 'r') as f:
config = json.load(f)

producer = pm.create_instance(config)

producer.connect()
producer.send('12345678')
producer.close()
```
**Switch from RabbitMQ to Kafka or reverse**
In test/config.json
```
{
"MQ_TYPE": "RMQ", # change this to KAFKA if your underlying MQ is KAFKA

"RMQ": {
...
},

"KAFKA": {
...
},

"REDIS": {
...
}
}

```
**We do support SASL_PLAINTEXT for kafka and simple auth in Redis as in the comments of the config file. Check their website to see how to setup the authentication**

You can build your own dictionary object as configuration for sure, as long as it contains the required fields.


That's it. Simple and straightforward.