snowpy 0.1.3

A Python library to upload and download data from database systems

SnowPy

Current release: SnowPy [v0.1.3]

SnowPy - A Python library to serve machine learning projects as a data gateway

'SnowPy' is a Python library providing a set of user-friendly functions to help upload and download data from different sources including database systems such as Microsoft SQL Server, Snowflake, (more to come), Email servers, data files, images and sound files. Functions of the 'SnowPy' was intially realsed as data tools for the MLToolKit Project (https://mltoolkit.github.io/MLToolKit). Thus, 'SnowPy' shares functions with 'PyMLToolKit' and 'TextLab' being developed under the MLToolKit Project.

  /^^^^^^^\
/-         -\
|   O   O   |  
|_    o    _|  /-----------\
  \   U   /   < Woof! Woof! )  
   |=====|     \-----------/

Introduction

'SnowPy' is a Python library providing a set of user-friendly functions to help upload and download data from different sources including database systems, emails, data files and more to come.

Installation

pip install SnowPy

If the installation failed with dependancy issues, execute the above command with --no-dependencies

pip install SnowPy --no-dependencies

Functions

Usage

import snowpy

Warning: Python Variable, Function or Class names

The Python interpreter has a number of built-in functions. It is possible to overwrite thier definitions when coding without any rasing a warning from the Python interpriter. (https://docs.python.org/3/library/functions.html) Therfore, AVOID THESE NAMES as your variable, function or class names.

abs	all	any	ascii	bin	bool	bytearray	bytes
callable	chr	classmethod	compile	complex	delattr	dict	dir
divmod	enumerate	eval	exec	filter	float	format	frozenset
getattr	globals	hasattr	hash	help	hex	id	input
int	isinstance	issubclass	iter	len	list	locals	map
max	memoryview	min	next	object	oct	open	ord
pow	print	property	range	repr	reversed	round	set
setattr	slice	sorted	staticmethod	str	sum	super	tuple
type	vars	zip	__import__

If you accedently overwrite any of the built-in function (e.g. list), execute the following to bring built-in defition.

del(list)

Functions

|- MSSQL
|  |- execute_mssql_query
|  |- read_data_mssql
|  |- write_data_mssql
|  |- read_data_mssql_bcp
|  |- write_data_mssql_bcp
|  |- get_database_list
|  |- get_database_usage_report
|- MySQL
|  |- To be integrated...
|- Snowflake
|  |- execute_snowflake_sql_query
|  |- read_data_snowflake
|  |- write_data_snowflake
|- CSV
|  |- read_data_csv
|  |- write_data_csv
|- EMAIL
|  |- read_email_eml
|  |- fetch_emails_ews
|  |- read_email_ews
|  |- save_ews_to_eml
|- DOCUMENTS
|  |- convert_document_file_to_text
|  |- create_document_image_table
|  |- process_docuement_image_table
...

SnowPy Example

# to be posted soon

Data Loading

# to be posted soon

License

Copyright 2019 Sumudu Tennakoon

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Cite as

@misc{SnowPy2019,
  author =  "Sumudu Tennakoon",
  title = "SnowPy: A Python library to upload and download data from database systems",
  year = 2019,
  publisher = "GitHub",
  howpublished = {\url{https://mltoolkit.github.io/SnowPy/}},
  version = "0.1.3"
}

SnowPy Project Timeline

• 2018-07-02 [v0.0.1]: Initialte MLToolKit project.
• 2019-07-02 [v0.0.1]: Snowpy functions first released as etl tools for MLToolKit project (https://mltoolkit.github.io/MLToolKit/).
• 2019-07-02 [v0.1.0]: First Release of SnowPy as seperate package.
• 2019-11-11 [v0.1.1]: Bug Fixes and enhancements.
• 2019-11-16 [v0.1.2]: Bug Fixes, Integrate MSSQL BCP support.
• 2019-12-22 [v0.1.3]: Bug Fixes, Integrate OCR and Email dats extarction functions(from TextLab).

Future Release Plan

• TBD [v0.1.4]: Support Sound data files.
• TBD [v0.1.5]: Integrate Data Exchage Server.
• TBD [v0.1.6]: Integrate MySQL support.
• TBD [v0.1.7]: Comprehensive Documentation

References

• https://pandas.pydata.org/
• https://docs.snowflake.net/manuals/user-guide/python-connector.html
• https://www.numpy.org/
• https://docs.python.org/3.6/library/re.html
• http://json.org/
• https://www.sqlalchemy.org/
• https://docs.microsoft.com/en-us/sql/tools/bcp-utility
• https://github.com/mkleehammer/pyodbc/wiki
• https://github.com/tesseract-ocr/tesseract
• https://github.com/madmaze/pytesseract
• https://www.ghostscript.com/
• https://pillow.readthedocs.io/en/5.1.x/handbook
• https://github.com/ecederstrand/exchangelib
• https://docs.python.org/3/library/email.html