SnowPy

Current release: SnowPy [v0.1.3]

SnowPy - A Python library to serve machine learning projects as a data gateway

'SnowPy' is a Python library providing a set of user-friendly functions to help upload and download data from different sources including database systems such as Microsoft SQL Server, Snowflake, (more to come), Email servers, data files, images and sound files. Functions of the 'SnowPy' was intially realsed as data tools for the MLToolKit Project (https://mltoolkit.github.io/MLToolKit). Thus, 'SnowPy' shares functions with 'PyMLToolKit' and 'TextLab' being developed under the MLToolKit Project.

  /^^^^^^^\
/-         -\
|   O   O   |  
|_    o    _|  /-----------\
  \   U   /   < Woof! Woof! )  
   |=====|     \-----------/

Introduction

'SnowPy' is a Python library providing a set of user-friendly functions to help upload and download data from different sources including database systems, emails, data files and more to come.

Installation

pip install SnowPy

If the installation failed with dependancy issues, execute the above command with --no-dependencies

pip install SnowPy --no-dependencies

Functions

Usage

import snowpy

Warning: Python Variable, Function or Class names

The Python interpreter has a number of built-in functions. It is possible to overwrite thier definitions when coding without any rasing a warning from the Python interpriter. (https://docs.python.org/3/library/functions.html) Therfore, AVOID THESE NAMES as your variable, function or class names.

If you accedently overwrite any of the built-in function (e.g. list), execute the following to bring built-in defition.

del(list)

Functions

|- MSSQL
|  |- execute_mssql_query
|  |- read_data_mssql
|  |- write_data_mssql
|  |- read_data_mssql_bcp
|  |- write_data_mssql_bcp
|  |- get_database_list
|  |- get_database_usage_report
|- MySQL
|  |- To be integrated...
|- Snowflake
|  |- execute_snowflake_sql_query
|  |- read_data_snowflake
|  |- write_data_snowflake
|- CSV
|  |- read_data_csv
|  |- write_data_csv
|- EMAIL
|  |- read_email_eml
|  |- fetch_emails_ews
|  |- read_email_ews
|  |- save_ews_to_eml
|- DOCUMENTS
|  |- convert_document_file_to_text
|  |- create_document_image_table
|  |- process_docuement_image_table
...

SnowPy Example

# to be posted soon

Data Loading

# to be posted soon

License

Copyright 2019 Sumudu Tennakoon

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Cite as

@misc{SnowPy2019,
  author =  "Sumudu Tennakoon",
  title = "SnowPy: A Python library to upload and download data from database systems",
  year = 2019,
  publisher = "GitHub",
  howpublished = {\url{https://mltoolkit.github.io/SnowPy/}},
  version = "0.1.3"
}

SnowPy Project Timeline

• 2018-07-02 [v0.0.1]: Initialte MLToolKit project.
• 2019-07-02 [v0.0.1]: Snowpy functions first released as etl tools for MLToolKit project (https://mltoolkit.github.io/MLToolKit/).
• 2019-07-02 [v0.1.0]: First Release of SnowPy as seperate package.
• 2019-11-11 [v0.1.1]: Bug Fixes and enhancements.
• 2019-11-16 [v0.1.2]: Bug Fixes, Integrate MSSQL BCP support.
• 2019-12-22 [v0.1.3]: Bug Fixes, Integrate OCR and Email dats extarction functions(from TextLab).

Future Release Plan

• TBD [v0.1.4]: Support Sound data files.
• TBD [v0.1.5]: Integrate Data Exchage Server.
• TBD [v0.1.6]: Integrate MySQL support.
• TBD [v0.1.7]: Comprehensive Documentation

References

• https://pandas.pydata.org/
• https://docs.snowflake.net/manuals/user-guide/python-connector.html
• https://www.numpy.org/
• https://docs.python.org/3.6/library/re.html
• http://json.org/
• https://www.sqlalchemy.org/
• https://docs.microsoft.com/en-us/sql/tools/bcp-utility
• https://github.com/mkleehammer/pyodbc/wiki
• https://github.com/tesseract-ocr/tesseract
• https://github.com/madmaze/pytesseract
• https://www.ghostscript.com/
• https://pillow.readthedocs.io/en/5.1.x/handbook
• https://github.com/ecederstrand/exchangelib
• https://docs.python.org/3/library/email.html