db-tqdm 1.1.3

A database tqdm process monitor with web user interface.

db-tqdm

A web based tqdm process bars.

This TQDM based module is very useful for web, airflow, servers or similar environments which do not have console to show the typical tqdm process bars. In these environments are not useful to use the standard TQDM module because the access to the console or log files are not easy, and if it writes into logs, this one are fulfilled with the progress bar information instead of the real information log data.

This module, by default, works the same way that the standard TQDM module. Only if the mode parameter is changed by 'mongo', or the environment variable 'TQDM_MODE' is defined with 'mongo', then the progress bar will store into a database,instead of printing it in the standard error output. This module also include the Uvicorn server in order to show this information with a browser like this:

Content

• db-tqdm module
  • Install db-tqdm
  • Execute the server
  • Use db-tqdm module
• db-tqdm environment variables
• db-tqdm docker
• To do

db-tqdm module

You can use db-tqdm as a usual Python module and integrate it into your Python projects. This section explains how to install, execute the server and test the db-tqdm module.

Install db-tqdm

To install only need to do the following:

pip install db-tqdm

If you are using the pymongo database instead the standard TQDM module, you need to install:

pip install monutils~=0.1.3

In the case, this module will be not installed, then, db-tqdm are going to work as a normal tqdm progress bar.

If you are using also the web user interface, you need to install the following modules:

pip install fastapi~=0.70.0 Jinja2~=3.0.2 uvicorn~=0.15.0 importlib-resources~=5.1.3

Execute the server

You can execute the db-tqdm server in both, docker image or natively. If you choose the second option, you can use the following command:

usage: dbtqdm [-h] [-H HOST] [-P PORT] [-t TYPE] [--db_host HOST] [--db_port PORT] [--root_path PATH] 
              [-r NAME] [-d NAME] [-u USER] [-p PASS] [--cert_key_file FILE] 
              [--ca_file FILE] [--session_token SESSION] [-i SECONDS] [TITLE]

Start the server to serve the bar progress data.

positional arguments:
  TITLE                 The web page title. By default, "Process monitors".

optional arguments:
  -h, --help            show this help message and exit
  -H HOST, --host HOST  The server host. By default, localhost.
  -P PORT, --port PORT  The server port. By default, 5000.
  --root_path PATH      The server root path. By default, "/".
  -t TYPE, --db_type TYPE
                        The database host. By default, mongo. Available databases: ['mongo'].
  --db_host HOST        The database host. By default, localhost.
  --db_port PORT        The database port. By default, 27017.
  -r NAME, --replicaset NAME
                        The replicaset. By default, none.
  -d NAME, --database NAME
                        The database name. By default, tqdm.
  -u USER, --user USER  The database user. By default, none.
  -p PASS, --password PASS
                        The user password. By default, none.
  --cert_key_file FILE  The cert key fle to connect to the database. By default, none.
  --ca_file FILE        The CA file to connect to the database. By default, none.
  --session_token SESSION
                        The session token to connect to the database. By default, none.
  -i SECONDS, --interval SECONDS
                        The default web refresh interval. By default, 5.

For example:

# Execute the server to connect with mongodb in localhost and port 27017 
# without authentication method and the database name "tqdm"
dbtqdm

# The same as above but defining database connection data
dbtqdm --db_host mymongoserver.com --db_port 18 -u mymongouser -p mymongopass

# Also, you can use environment variables instead command arguments
export TQDM_DB_HOST=mymongoserver.com
export TQDM_DB_PORT=18
export TQDM_DB_USER=mymongouser
export TQDM_DB_PASSWORD=mymongopass

dbtqdm

Use db-tqdm module

The use of db-tqdm is very similar to normal tqdm. For example, if you are using the progress bar into normal console application or jupyter notebook, you only need to execute:

from dbtqdm.mongo import tqdm
from time import sleep

for i in tqdm(range(0, 10), desc='Normal bar'):
    sleep(1)

This code will show in the suitable output the following progress bar as usual:

Normal bar:  60%|██████    | 6/10 [00:06<00:04,  1.00s/it]

You can change te normal tqdm progrsss bar by the database based one, using the own db-tqdm parameters, specially mode and name, but not uniquely:

from dbtqdm.mongo import tqdm
from time import sleep

# Create a tqdm based on MongoDB in localhost:27017 with the name "test1" and red colour
for _ in tqdm(range(0, 5000), desc=f'Description of the progress bar 1',
              mode='mongo', name='test1', colour='red'):
    sleep(1)

# The same as above but with the database connection
for _ in tqdm(range(0, 5000), desc=f'Description of the progress bar 1',
              mode='mongo', name='test1', colour='red',
              host='mymongoserver.com', port=18, user='mymongouser', password='mymongopassword'):
    sleep(1)

This code will generate the following web-based bar progress:

However, we strongly recommend to use environment variables instead of the function parameters because, this way, your code will be the same if you use the tqdm locally or web-based. For example. this code will generate a normal tqdm bar progress:

from dbtqdm.mongo import tqdm
from time import sleep

for _ in tqdm(range(0, 5000), desc=f'Description of the progress bar 1', colour='red'):
    sleep(1)

Nevertheless, if you define previously the follwoing environment variables, the same code will generate a web-based one:

export TQDM_MODE=mongo
export TQDM_NAME=test1

But this approach has a problem in the case you want to use several progress bars at the same time, because each progress bar is identified by its name. The solution is to change the previous code adding the suffix parameter:

from dbtqdm.mongo import tqdm
from time import sleep

for _ in tqdm(range(0, 10), desc=f'Description of the progress bar 1', colour='red', suffix='_main'):
    for _ in tqdm(range(0, 20), desc=f'Description of the progress bar 1', colour='yellow', suffix='_secondary'):
        sleep(1)

If you do not define the DB-TQDM environment variables, both progress bars will be normal ones, ignoring the suffix parameter. However, if you define those variables, you will have two progress bars with the name "<bar_name>_main" and "<bar_name>_secondary", respectively.

Note: At the moment, do not put spaces neither, the bar name nor bar suffix.

As we can see above, using environment variables instead the parameters allow you to use exactly the same code in different platforms (for example, local workstation or in remote airflow server), and depending on the defined environment variables can switch between different user interfaces.

Note: At the moment, the argument 'db_type' is not supported, and it will be ignored.

db-tqdm environment variables

You can define environment variables to change the way tqdm progress bar appears and the MondogDB connection data. All the environment variables you can define are:

Variable	Description
TQDM_MODE	If you want a normal or mongo bar progress. By default, normal.
TQDM_NAME	The bar progress name.
TQDM_SUFFIX	The suffix of the bar progress name. This value is better to set by function parameter.
TQDM_HOST	The Web server host address. By default, localhost.
TQDM_PORT	The Web server port. By default, 5000.
TQDM_TYPE	The database type. In the current implementation only "mongo" is available. By default, mongo.
TQDM_DB_HOST	The database host address. By default, localhost.
TQDM_DB_PORT	The database port. By default, 27017.
TQDM_DB_REPLICASET	The mongo replicaset. By default, none.
TQDM_DB_NAME	The database name. By default, tqdm.
TQDM_DB_USER	The database username. By default, none.
TQDM_DB_PASSWORD	The user password. By default, none.
TQDM_DB_CERT_KEY_FILE	The certificate key file. By default, none.
TQDM_DB_CA_FILE	The CA file. By default, none.
TQDM_INTERVAL	The web refresh interval in seconds. By default, 5s.
TQDM_TITLE	The web title. By default, 'Process monitors'.

The TQDM_HOST and TQDM_PORT if only for the Web server.

db-tqdm docker

You can install the db-tqdm server by means a docker image.

docker pull ialife/db-tqdm:1.1

Remember you can define the [environment variables] to set the database connection. For example, if you have the mongodb in your localhost, you can execute the following command to connect the service in the db-tqdm docker with your mongodb:

docker run -e TQDM_DB_USER=myusename -e TQDM_DB_PASSWORD=mypassword -e TQDM_DB_HOST=localhost \
           -e ROOT_URL=http://localhost -e MONGO_URL=mongodb://localhost:27017 --network="host" ialife/db-tqdm

To do

Message when there are any active process bar

Show a message when there are no active process bars.

User and password for MongoDB connection

Add the possibility to set the user and password for the MongoDB connection.

Create a historical view of finished processes

In the collection _stats_ are the historical information about the finished processes. It could be interesting to use them in a view with this information. The idea is to use the home page, below the progress bars in the main page, to add a section with a paged table with the finished processes ordered descending by start time.

Generalize to be able to use other database managers

The core of the module is ready to use another database managers creating classes inherited from DatabaseTqdm. However, the server process is not prepared to use other database managers, only MongoDB. It could be interesting to refactor the code to add this functionality.

Use Redis or Kafka

This can be adapted (but without historical information) to be used with Redis or Kafka platforms.

Pass collections to registers

I made the design decision to create each process bar in a MongoDB collection instead of create a register for each process. I do not sure because I made that decision, and I think it would be better to replace the different collections by registers of a unique MongoDB collection.