atk-training-rin-q-basic 0.1.0

No project description provided

Assignment

Design a system with producer and a consumer.
Producer produces some entities independent of consumer
Consumer consumes the entities, producing new entities.
Click here for usage/commands.

---

Solution

Here, let the job be a string of length 10.

Using async and await, functions can be turned into coroutines to be passed to a execute_function command.

graph TD;
    A["PersistentQInterface"]-- child class ---B["Producer(s)"];
    A-- child class ---C["Consumer(s)"]
    B-- submit jobs -->D["PersistentQSQlite"]
    D-- process jobs -->C

System

Consists of abstract class (ABC) PersistentQInterface, children Producer, Consumer and Cleaner.
Also consists of class PersistentQSQLite that performs SQL queries on a database from Python.

On being called, Producer generates a file name (and file), and submits this file name as a job. PersistentQSQLite adds this name to a queue/SQL database. Occurs every 5 seconds
Consumer reads this queue and processes them, generating a new file with the processed content. Occurs every 7-15 seconds.
If after n tries a file is not able to be processed by Consumer, it gets renamed with a .failed extension. Cleaner checks for .failed files every 30 seconds and deletes them.
Wait times for Producers, Consumers and Cleaners are independent amongst themselves and each other.

---

Usage

atk-training-rin-q-basic --help

To run the web app, try

atk-training-rin-q-basic web-app

The program will open in localhost:8000. producer and consumer will be running in the background -- any logs/output produced (primarily by consumer) will be visible in the terminal.
NOTE: All files produced will be in the CURRENT DIRECTORY BY DEFAULT. To set your custom path, modify the path in config.yaml.

Setting up your own config

Parameters can be customised by giving the command path to a config.yaml file.
The parameters along with their default values are as follows. If any parameter is missing from the file, the default value will be used.

consumer:
    n_time: 3600
    n_tries: 3
general:
    primary_path: current/directory
    n_prod: 1
    n_cons: 1

n_time : Seconds a consumer is allowed to spend on one job before skipping n_tries : Number of time a consumer is allowed to try a job (in case of errors) before skipping primary_path: Path where files are created n_prod : Number of producers n_cons : Number of consumers