lsbatch 1.1.3

lsbatch LeadSquared provided Batch Jobs software development kit. This allows developers to code and test Batch Jobs offline.

About lsbatch

lsbatch is a LeadSquared provided Batch Jobs software development kit. This allows developers to code and test Batch Jobs offline. It can also be used to prepare final deliverable that can uploaded to configure Batch Jobs in user's LeadSquared account.

Getting Started

Assuming that you have Python 3.7 and virtualenv installed, install lsbatch

pip install lsbatch

Usage

Initiate a new Batch Job project

To create a new Batch Job in your local machine, execute the following command in cmd or shell:

lsbatch init

You will be asked to provide name of this Batch Job

>Your Batch Job Name:

A new folder with the provided name will be created in the current directory. This folder will be having all the libraries required for Batch Jobs local development. You will find custom virtualenv in folder name batch-virtualenv with following default packages already installed

• boto3==1.14.33
• mysql-connector-python==8.0.21
• pysftp==0.2.9
• sendgrid==6.4.4
• pandas==1.1.0
• pyyaml==5.3.1
• openpyxl==3.0.4
• requests==2.24.0
• tabulate==0.8.7
• numpy==1.19.1

After initiating batch with lsbatch init , make sure all the following commands are executed at the project folder level

cd {batch_job_folder_name}

General rules for writing a Batch Job

1. Your Batch Job folder should have a folder by name src which will be used to write your actual code. This folder is automatically created during the init process
2. User should keep their Batch Job files in 'src' folder only
3. src folder should have main.py file that is basically the starting point of your application. By default this file is created with sample code during the init process
4. Your src folder may contain a requirements.txt file which can be used to provide any custom packages which you may require

Using inbuilt Batch Jobs function

Batch Job provides various functions for comman tasks like logging, sending notification emails or making DB queries. lsbatch packs and mocks the same functions and are listed below:

Logging

LS.log.info('Hello')
LS.log.info('Hello', {"Support Email": "[e-mail]"})
LS.log.info(LS.settings.mykey)
LS.log.debug('Debug Log Example')
LS.log.error(type(err).__name__, traceback.format_exc())
LS.log.fatal(type(err).__name__, traceback.format_exc())

Using Variable

Used to store key/value pair with sensitive/private data. For example API key for some third party service or your SFTP password. It can be retrieved using following key

LS.settings.<key_name>

Send Batch Job status notification email

LS.email.send_email_notification(<EmailType>, <Subject>, <EmailBody>, isHTML =<True/False>)

Where

• EmailType: one of "Success" or "Failure" which corresponds to success or failure of execution status
• Subject: can be maximum 255 character, above which it will be trimmed
• EmailBody: plain text or HTML content
• isHTML: is an optional parameter with default value as True

Making DB Calls

You can connect to your LeadSquared account database and ETL database as well. Use the following function to query the same

LS.db.execute_DB_query(<query>, multi=False)
LS.db.execute_ETL_query(<query>, multi=False)

by default the function considers the query are single statement. For multiline query, pass multi parameter as True

Mocking DB query result To mock the DB result use the file query.json which is provided in the root project folder. It contains a json where

• key is the DB query(exact - case sensitive) and
• value is the csv file name that contains the result.

You will also find a sample_query_result.csv in project's root folder which is mapped as sample response in query.json.

You can create csv file by any name. Please note that this is purely for mocking purpose and does not evaluate the query, syntax or result in any way. Use the admin panel in your LeadSquared account to validate the same.

Executing a Batch Job

To run Batch Job, run command lsbatch run

lsbatch run

This will execute the code in context of virtual environment that has already been setup

Install LeadSquared Batch Job dependencies in current folder

The root folder already contains a .gitignore file that will commit only necessary files to your source code repository. Feel free to update the file as per your need. If you clone the repository, run the lsbatch install command inside the project directory to setup the Batch Job environment again.

lsbatch install

This command can also be used in existing projects to reset the Batch Jobs environment at any time.

To run this command succesfully, current directory should have src that contains your code

Install your custom packages

lsbatch install {package_name}

This installs your custom package to batch-virtualenv folder. Also it creates/updates a requirements.txt file inside src folder. requirements.txt file will be used to identify dependencies at the time of actual execution of the Batch Job

Packaging your project for deployment

Batch Job accepts the deliverables as a zip file. To zip the project in a correct way, execute the following command

lsbatch pack

This will create a code.zip file in the directory named deliverables. Remember to add this directory to the .gitignore file in order to reduce the size of repository

Uninstall package

To uninstall a custom package, execute the following command

lsbatch uninstall {package_name}

This will also remove the package name from requirements.txt file

Configuring Tenant Profile

You can create multiple tenant profiles with various access levels using the configure command

$ lsbatch configure

You will be prompted to enter

• Profile Name : Define the profile name. We suggest you to enter a profile name in the following convention, so that it is easy to remember and re-use - + (e.g., 1234Admin)
• Orgcode : LeadSquared organization code (e.g., 1234). To find your orgcode, see How to Find My LeadSquared Orgcode.
• Region : You may pass 1, 11, 31 or 21. Alternatively, you can pass SGP, US, IR or MUM. 1 denotes Singapore | 11 denotes U.S.A | 21 denotes India | 31 denotes Ireland.
• Token : Pass the token generated from the Batch Jobs UI.

You can also simply run lsbatch, it will prompt all available commands

$ lsbatch

Create and Upload a Batch Job Code for Deployment

To learn how to upload the batch job code directly from the LeadSquared UI, see Create a Batch Job

Pass the command line parameters --orgcode --region --token. You may also pass these parameters as -o, -r and -t.

E.g.

$ lsbatch save --orgcode 1234 --region 1 --token asdfghijkl1234

Alternatively, if you've configured a profile use --profile or -p.

E.g.

$ lsbatch save --profile mytenant1234

Note:

• If the Batch Job Id doesn't exist in the batch_details.json file, the command creates a Batch Job and uploads the code.
• If the Batch Job Id exists in batch_detals.json, it packs and uploads the code to the respective Batch Job.

To save the settings, pass command line parameter --settings, or -s.

E.g.

$ lsbatch save --profile mytenant1234 --settings

Note: Only Admins and Developer-Configurator can save the settings.

Deploy or Publish a Batch Job

The following command publishes a Batch Job.

Pass the command line parameters --orgcode --region --token. You may also pass these parameters as -o, -r and -t.

E.g.

$ lsbatch publish --orgcode 1234 --region 1 --token asdfghijkl1234

Alternatively, if you've configured a profile use --profile or -p.

E.g.

$ lsbatch publish --profile mytenant1234

Note: Only Admins can publish a Batch Job.