pyreports 1.5.0

pyreports is a python library that allows you to create complex report from various sources

pyreports

pyreports is a python library that allows you to create complex reports from various sources such as databases, text files, ldap, etc. and perform processing, filters, counters, etc. and then export or write them in various formats or in databases.

Test package

To test the package, follow these instructions:

$ git clone https://github.com/MatteoGuadrini/pyreports.git
$ cd pyreports
$ python -m unittest discover tests

Install package

To install package, follow these instructions:

$ pip install pyreports #from pypi

$ git clone https://github.com/MatteoGuadrini/pyreports.git #from official repo
$ cd pyreports
$ python setup.py install

Why choose this library?

pyreports wants to be a library that simplifies the collection of data from multiple sources such as databases, files and directory servers (through LDAP), the processing of them through built-in and customized functions, and the saving in various formats (or, by inserting the data in a database).

How does it work

pyreports uses the tablib library to organize the data into Dataset object.

Simple report

I take the data from a database table, filter the data I need and save it in a csv file

import pyreports

# Select source: this is a DatabaseManager object
mydb = pyreports.manager('mysql', host='mysql1.local', database='login_users', user='dba', password='dba0000')

# Get data
mydb.execute('SELECT * FROM site_login')
site_login = mydb.fetchall()

# Filter data
error_login = pyreports.Executor(site_login)
error_login.filter([400, 401, 403, 404, 500])

# Save report: this is a FileManager object
output = pyreports.manager('csv', '/home/report/error_login.csv')
output.write(error_login.get_data())

Combine source

I take the data from a database table, and a log file, and save the report in json format

import pyreports

# Select source: this is a DatabaseManager object
mydb = pyreports.manager('mysql', host='mysql1.local', database='login_users', user='dba', password='dba0000')
# Select another source: this is a FileManager object
mylog = pyreports.manager('file', '/var/log/httpd/error.log')

# Get data
mydb.execute('SELECT * FROM site_login')
site_login = mydb.fetchall()
error_log = mylog.read()

# Filter database
error_login = pyreports.Executor(site_login)
error_login.filter([400, 401, 403, 404, 500])
users_in_error = set(error_login.select_column('users'))

# Prepare log
myreport = dict()
log_user_error = pyreports.Executor(error_log)
log_user_error.filter(list(users_in_error))
for line in log_user_error:
    for user in users_in_error:
        myreport.setdefault(user, [])
        myreport[user].append(line)

# Save report: this is a FileManager object
output = pyreports.manager('json', '/home/report/error_login.json')
output.write(myreport)

Report object

import pyreports

# Select source: this is a DatabaseManager object
mydb = pyreports.manager('mysql', host='mysql1.local', database='login_users', user='dba', password='dba0000')
output = pyreports.manager('xlsx', '/home/report/error_login.xlsx', mode='w')

# Get data
mydb.execute('SELECT * FROM site_login')
site_login = mydb.fetchall()

# Create report data
report = pyreports.Report(site_login, title='Site login failed', filters=[400, 401, 403, 404, 500], output=output)
# Filter data
report.exec()
# Save data on file
report.export()

ReportBook collection object

import pyreports

# Select source: this is a DatabaseManager object
mydb = pyreports.manager('mysql', host='mysql1.local', database='login_users', user='dba', password='dba0000')

# Get data
mydb.execute('SELECT * FROM site_login')
site_login = mydb.fetchall()

# Create report data
report_failed = pyreports.Report(site_login, title='Site login failed', filters=[400, 401, 403, 404, 500])
report_success = pyreports.Report(site_login, title='Site login success', filters=[200, 201, 202, 'OK'])
# Filter data
report_failed.exec()
report_success.exec()
# Create my ReportBook object
my_report = pyreports.ReportBook([report_failed, report_success])
# Save data on Excel file, with two worksheet ('Site login failed' and 'Site login success')
my_report.export(output='/home/report/site_login.xlsx')

Tools for dataset

This library includes many tools for handling data received from databases and files. Here are some practical examples of data manipulation.

import pyreports

# Select source: this is a DatabaseManager object
mydb = pyreports.manager('mysql', host='mysql1.local', database='login_users', user='dba', password='dba0000')

# Get data
mydb.execute('SELECT * FROM site_login')
site_login = mydb.fetchall()

# Most common error
most_common_error_code = pyreports.most_common(site_login, 'code')  # args: Dataset, column name
print(most_common_error_code)   # 200

# Percentage of error 404
percentage_error_404 = pyreports.percentage(site_login, 404)    # args: Dataset, filter
print(percentage_error_404)   # 16.088264794 (percent)

# Count every error code
count_error_code = pyreports.counter(site_login, 'code')  # args: Dataset, column name
print(count_error_code)   # Counter({200: 4032, 201: 42, 202: 1, 400: 40, 401: 38, 403: 27, 404: 802, 500: 3})

Official docs

In the following links there is the official documentation, for the use and development of the library.

• Managers: doc
• Executor: doc
• Report: doc
• data tools: doc
• examples: doc
• API: io, core

Open source

pyreports is an open source project. Any contribute, It's welcome.

A great thanks.

For donations, press this

For me

For Telethon

The Telethon Foundation is a non-profit organization recognized by the Ministry of University and Scientific and Technological Research. They were born in 1990 to respond to the appeal of patients suffering from rare diseases. Come today, we are organized to dare to listen to them and answers, every day of the year.

Adopt the future

Acknowledgments

Thanks to Mark Lutz for writing the Learning Python and Programming Python books that make up my python foundation.

Thanks to Kenneth Reitz and Tanya Schlusser for writing the The Hitchhiker’s Guide to Python books.

Thanks to Dane Hillard for writing the Practices of the Python Pro books.

Special thanks go to my wife, who understood the hours of absence for this development. Thanks to my children, for the daily inspiration they give me and to make me realize, that life must be simple.

Thanks Python!

Release notes

1.5.0

Aug 4, 2022

• Added cli module
• Added reports cli
• Added _getitem_ method on Report class
• Added _delitem_ method on Report class
• Added _getitem_ method on ReportBook class
• Added _delitem_ method on ReportBook class
• Added _contains_ on Executor class
• Fix NoSQLManager creation into manager function
• Fix print_data on Report class

1.4.0

Jun 27, 2022

• Added _bool_ method on Report class
• Added _iter_ method on Report class
• Added _bool_ method on ReportBook class
• Added _iter_ method on Connection and File classes
• Added _iter_ method on FileManager class
• Added _iter_ method on DatabaseManager class
• Added _getitem_ on Executor class
• Added _delitem_ on Executor class
• Fix name of attachment on send method of Report class
• Fix write method on LogFile class

1.3.0

Apr 15, 2022

• Added NoSQLManager class; this class extend Manager class on the nosqlapi package
• Added LogFile class; this class load a log file and read method accept regular expression
• Added _bool_ and _repr_ method on File and Connection abstract classes
• Fix documentation API section
• Fix tests package
• Fix CircleCi docker image

1.2.0

Aug 5, 2021

• Added fill_value argument on aggregate function; this value also is callable without arguments
• Added send method on Report class; with this method you send report via email
• Added send method on ReportBook class; with this method you send report via email
• Fix *str* method on Report class

1.1.0

Jun 5, 2021

• Created abstract File class
• Created TextFile class
• Added _str_ method for pretty representation of Executor class
• Added _repr_ method for representation of DatabaseManager class
• Added _repr_ method for representation of FileManager class
• Added _repr_ method for representation of LdapManager class
• Fix documentation for new abstract File class

1.0.0

May 26, 2021

• Created abstract Connection class
• Created *Connection classes
• Created *File classes
• Created FileManager, DatabaseManager and LdapManager classes
• Created Executor class
• Created Report class
• Created ReportBook class
• Created average, most_common, percentage, counter, aggregate, merge, chunks, functions