Skip to main content

This package contains a bunch of Python tools developed for Support, Test and Automation IT Engineers

Project description

ParTools

Introduction

This package contains a bunch of Python tools I have developed and used as an IT Test / Support / Automation Engineer.
You can mainly use them for:

  • Performing multi threaded SQL queries on an Oracle DB (sql)
  • Performing multi threaded SQL queries on a given perimeter on an Oracle DB (rl)
  • Comparing and sorting potentially big files (dq)
  • Reading and searching potentially big files (tools.bf)
  • Parsing potentially big XML files (tools.xml)
  • Searching and removing duplicates (tools.dup)
  • Filtering potentially big file with flexible conditions (tools.filter)
  • Splitting potentially big files (tools.split)
  • Extracting doc string from Python code (tools.extract_doc)
  • Sending mails with gmail, outlook or without authentication (mail)
  • Simple logging or logging in loops (log and step_log), manipulating files and strings, running shell commands, generating threaded message boxes and many others (utils)

Quickstart

pip install partools

Then just run import partools (or run any partools function) with python to initialise the package.
You should then see something like this in the console:

...
Checkout the README.md on GitHub: https://github.com/paularnaud2/ParTools
Get started here c:\Git\Test\venv\lib\site-packages\partools\quickstart
Set up your conf here: c:\Git\Test\venv\lib\site-packages\partools\conf.py
Happy scripting!

This gives you the links to the quickstart folder where you'll find examples of use and descriptions of the different available packages and functions. You'll also find here the link to the conf.py file (see below).

If you want to use the cx_Oracle dependant packages (sql and rl), you'll need an Oracle instant client whose directory you can set in the conf.py file.

The conf.py file

The partools/conf.py file contains the main user settings for the partools package such as the path to the Oracle instant client (ORACLE_CLIENT). If needed, you can create a PTconf.py file at the root that will be used instead of the native partools/conf.py file. Similarly, you can also create a PTconf_perso.py file at the root that will take over the two previous files described. This can be useful if you work on a shared repository but still want/need to have your own local configurations.

The global variable files gl.py and main functions inputs

Each package has a gl.py file which sets its global variables and constants. Each of these variables can be passed to the main package function (e.g. rl.reqlist for the rl package) and if so, overwrites the value defined in the gl.py file. In that respect, constants defined in the gl.py file can be seen as default input values.

The utils package

The partools package includes a utils package which provides generic functions used by the other packages. As you may want to use some of them for your own code, I recommend you to check out the list of those functions in partools/utils/__init__.py. Here are a few examples:

  • save_list: saves a list into a text file
  • load_txt: loads a text file into a string or a list
  • list_files: returns the list of the files in a folder
  • like: behaves as the LIKE of Oracle SQL (you can match strings with wildcard character '*'). Example: like('Hello World', 'He*o w*d') returns True
  • big_number: converts a potentially big number into a lisible string. For example big_number(10000000) returns '10 000 000'.
  • get_duration_string: outputs a string representing the time elapsed since the input start_time. For example '3 minutes and 20 seconds'.
  • run_cmd: runs a Windows shell command (sTools)
  • run_sqlplus: connects to sqlplus as sysdba and runs a sql script (sTools)
  • msg_box: opens a threaded message box containing the 'msg' input string (sTools)

Logging with the utils package

If you want the log function to actually fill a log file, you have to run init_log() before using it, otherwise it will just print out the log info in the console.
You can specify a log_format for the log timestamp which by default is '%H:%M:%S -' (conf.LOG_FORMAT setting). Here is what a default log line looks like:

19:45:04 - This line has been generated by the partools.utils.log function

The step_log function allows you to log some information only when the input counter is a multiple of the input step (thus, it is used in loops). step_log can be useful to track the progress of long processes such as reading or writing millions of lines in a file. The what input expects a description of what is being counted. It's default value is 'lines written'.
In order to correctly measure the elapsed time for the first log line, the step_log function has to be initialised by running init_sl_time().
So for example, if you input step=500 and don't input any what value, you should get something like this:

19:45:04 - 500 lines written in 3 ms. 500 lines written in total.
19:45:04 - 500 lines written in 2 ms. 1 000 lines written in total.
19:45:04 - 500 lines written in 2 ms. 1 500 lines written in total.

Checkout the utils_log.py file in partools/quickstart for a simple example of use.

The PT folder

When first used, the utils package gets initialised by creating a PT directory (which you can set in conf.py). It is intended to contain the log files and the temporary files generated by the different ParTools's scripts. It also has in and out directories used by the test package (and of course that you can also use for your own scripts using ParTools).

Recover functionalities

sql.download, rl.reqlist and sql.execute have a recovery functionality. This means that if the process is unexpectedly stopped (e.g. because of network issues), then when relaunched, the script automatically restarts from where it stopped. When you run long processes (e.g. extracting millions of lines from a database), this can save you a significant amount of time if something goes wrong (especially when close to the end!).
The reliability of these recovery mechanisms is ensured by automated tests using the multiprocessing library to simulate the unexpected process interruption.

Happy scripting!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

ParTools-1.0.3-py3-none-any.whl (122.8 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page