MrSnippets 2.0.1

A complete collection of commonly used code Snippets in Python

MrSnippets

A complete collection of commonly used code Snippets in Python

Getting Started

To get started, install the package using the below command on your machine.

pip install MrSnippets

Sample Usage,

from MrSnippets.utilities import *
print(get_numbers_from_string('1a2b3e'))
>123

Utilities Module

Its a collection of commonly used functions to minimize the coding effort and time.

List of supported functions,

Text Operations

• get_clean_text(string:str)
• get_clean_html(html_chunk)
• get_numbers_from_string(string:str)
• get_alpha_from_string(string:str)
• get_string_from_html(soup)
• compare_two_strings(string_one,string_two)

List Operations

• join_elements(source_list:list, separator:str='')
• get_clean_list(list_x:list)
• compare_two_list(first:list, second:list)
• find_duplicates_from_list(list:list)
• get_unique_values( sequence:list)

Dict Operations

• get_clean_dict(dict:dict)
• modify_jsondata(abbreviations_dict:dict,target_dict:dict)
• combine_dict(dict1, dict2)

Support Functions

• get_current_user_name()
• get_current_platform()
• get_current_script_name()
• get_user_agent(**kwargs)
• generate_uuid_by_digits(digits)
• genearate_hash_key(string)
• generate_fileName(project, ext)
• get_class_variables(class_obj)

Date Operations

• get_current_date(format="ISO")
• get_yearFromDate(self, date_string)
• convert_date_to(input_string, format)

File Operations

• get_base64Image(encoded_data,path_to_store,file_name)
• get_filename(file_name)
• move_to(source,destination,filename)
• copy_to(source,destination,filename)
• list_files(diretory,ext:str='')
• read_text_file(file_name)
• read_xml_file(path,file)
• read_xml_file_as_bs4(path,file_name)
• write_text_file(filepath, filename, content)
• write_csv_file(filepath,filename,content:dict,headers:list)
• get_sizeof(num, suffix='o')
• check_file_exists(file_dir)

Email Functions

Some common functions to send an email from linux servers.

• send_email_with_attachment(projectName, userMessage, attachment, recipients,default_email_recipients='')
• email_notification(notificationType, project, message, recipients,default_email_recipients='')

Data Mining Module

Its a simple implementation of CSS selector using Beautifulsoup. The selectors are straight forward and simple to use.

There is a list of pre-defined selector functions. For example, selecting a single element from a chunk as follows.

from MrSnippets.soup_wrapper import  *
people_name = get_element(html_chunk,'div','class','people_name')
people_name = get_element_by_tag(html_chunk,'<div class="people_name">')

List of supported functions,

Bs4 Object based functions

• get_element(soup, tag="div", attributeName='class', attributeValue='profile')
• get_elements(soup, tag="div", attributeName='class', attributeValue='profiles')
• get_element_by_tag(soup,selector_string:str)
• get_elements_by_tag(soup,selector_string:str)
• get_sibling_text(soup, child:str, sibling:str, contains_string:str, sibling_type="prev|next")

Semi-Automated Functions

• extract_hyper_link(soup_chunk,patterns:list,**kwargs)
• extract_social_links(self, html_source)
• extract_vcard_link(soup_chunk,**kwargs)
• extract_image_link(soup_chunk,**kwargs)
• extract_vcard_data(vcard_text:str)
• extract_meta_data(self, html_source)
• extract_email_addresses(string)
• extract_phone_numbers(html_chunk)
• extract_domain_name_from_url(url)
• extract_date_from_string(date_string: str)
• extract_countries_from_text(input_text)
• extract_url_prefix(url_string: str)

Support Functions

• get_parsed_url(url_string)
• fix_url_format(self, url, prefix)
• validate_url(url)

Data Manipulations

• get_standard_name(name:str)
• split_name(name_string:str,reverse_it:bool=False,**kwargs)

Web Clinet Moudule

Its an collection of commonly used function for interacting on Internet

List of supported functions,

• get_selenium_response(url, timeout)
• get_response(url, response_type, attempt=0, **kwargs)
• download_file(url,dir,file_name,extension)

get_reponse function utilized in many ways, these are the currently supported arguments,

data = kwargs.get('data',{})
params = kwargs.get('params',{})
gateway = kwargs.get('gateway', 'requests')
timeout = kwargs.get('time_out', 60)
verify = kwargs.get('verify', True)
method = kwargs.get('method', None)
domain = kwargs.get('domain', '')
headers = kwargs.get('headers',{})
allow_redirects = kwargs.get('allow_redirects', True)
proxy =  kwargs.get('proxy', True)
stream =  kwargs.get('stream', False)
dom_parser = kwargs.get('dom_parser','html5lib')

Sample Usage:

response = get_response(url,'json',method='post', data = payload, headers=headers, timeout=100, verify=False)

NLP Module

Collection of functions to minimize the code and time for NLP related operations.

List of supported functions,

Main Functions

• sentence_tokenization(raw_text:str)
• get_tokenized(sentence: str, ignore_stopwords: bool = False,use:str='SExpr')
• get_stemming(word_tokens:list)
• get_lemmatize_data(word_tokens: list)
• get_standardize_words(tokens: list, lookup_dict: dict)
• generate_ngrams(tokens: list, n)
• generate_pos_tags(sentence, ignore_stopwords: bool = False)
• generate_named_entity(sentence)
• get_cosine_similarity(text1:str, text2:str)
• sentiment_analysis(sentence,algorithm:str="VADER")
• generate_text_summary(text:str)
• extract_geo_data(text:str,required='country')

Support Functions

• _remove_stopwords(word_tokens:list)
• _normalize_text(text:str)
• _create_frequency_table(text_string)
• _score_sentences(sentences, freqTable)
• _find_average_score(sentenceValue)
• _generate_summary(sentences:str, sentenceValue, threshold)

Image Processing Module

Collection of functions to minimize the code and time for image processing tasks

List of supported functions,

• adjust_brightness(input_image, output_image, factor)
• adjust_contrast(input_image, output_image, factor)
• adjust_sharpness(input_image, output_image, factor)
• resize_image(input_image_path,output_image_path,size)
• scale_image(input_image_path,output_image_path,width=None,height=None)
• black_and_white(input_image_path,output_image_path)
• rotateImage(image_path, degrees_to_rotate, saved_location)
• flipImage(image_path, saved_location,direction)
• cropImage(image_path, coords, saved_location)

Data Support Module

Collection of functions to minimize the code and time for day to day tasks

List of supported functions,

• get_city_list()
• get_state_list()
• get_city_info_obj()

Data Convertion Module

A collection of methods used to convert data files from one format to another. For example, DataFrame to XML

List of supported functions,

• pandas_to_xml(**kwargs) parameters for the functions includes,

dataFrame : pandas data frame object
fileName : output file name
list_variables : Values should considered as list
xml_declaration : True/False by default True

• create_xml_string_from_dict(data_dict:dict)

Mongo Wrapper

Its a collection of commonly used functions to minimize the code and time for mongo related operations.

To Setup, the Client IP do follow the below steps,

1. from MrSnippets.mongo_wrapper import MONGO_CLIENT_IP
2. Now, CTRL + Right Click on MONGO_CLIENT_IP
3. Update this 'xx.xx.xx.xxx' to Your IP (195.16.20.335:27017)

now the line looks something like this,

MONGO_CLIENT_IP = '195.16.20.335:27017'

List of supported functions,

• get_mongo_client(database_name, collection_name)
• list_db()
• list_collections(db:str)
• get_summarize(db:str,collection:str)
• get_sample(db:str,collection:str,query_by:str,value:str,limit:int=1)
• update_record(connection:dict,query_by:str,query_by_value,data:dict)
• create_index(connection:dict,index_attributes:list,ascending:bool=True)
• update_attribute(connection:dict,query_by:str,query_by_value,data:dict,attributes:list)

MySQL Wrapper

Its a collection of commonly used functions to minimize the code and time for MySQL related operations.

List of supported functions,

• get_mysql_client(host_ip, username, pwd,db_name)
• query_actions(connection_obj, query, query_type)

Example usage,

get_mysql_client

from MrSnippets.mysql_wrapper import *
conn = get_mysql_client('190.10.30.160', 'root', 'password', 'profile')

SQL Wrapper

Its a collection of commonly used functions to minimize the code and time for SQL related operations.

List of supported functions,

• get_sql_client(server_ip,database,userName,pwd)
• query_actions(db_object, query, query_type)
• insert_records(connection_obj, table_data:dict, data_dict: dict, unique_columns:list)

Example usage,

sql_insert_records

from MrSnippets.mysql_wrapper import *
db_object = get_sql_client('195.16.40.82', 'SampleDB', 'root', 'password')
sample_data = {'name':'dharan','phone_no':'023456789'}
table_meta = {'db_name':'informationSystem','table_name':'contactInfo'}
insert_records(db_object, table_data:dict, sample_data, ["phone_no"])

query_actions

__author__ = 'dhamodharan.k'
from MrSnippets.sql_wrapper import *
db_object = get_sql_client('xx.xx.xx.xx', 'employeeData', 'root', 'pass')
find_q = 'select emp_id,age from employee_info where emp_id in (4213958, 4213959)'
rows = query_actions(db_object, find_q, 'select')
for row in rows:
    content = str(row[1] + 1)
    update_query = "update employee_info set agePlus = '{}' where emp_id = {}".format(content,row[0])
    query_actions(db_object,update_query,'update')

Prerequisites

Nothing but a basic knowlege of python

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Authors

• Dhamodharan - (https://github.com/dhamodharanrk)

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Required Libraries

Data Extraction

• beautifulsoup4>=4.3.3
• requests>=2.18.4
• html5lib>=1.0b10
• user_agent>=0.1.9
• selenium>= 3.141.0

Database

• PyMySQL>=0.9.3
• pymongo>=3.8.0

Text Pre-Processsing

• ftfy>=5.5.1
• tldextract>=2.2.1
• bleach>=3.1.0
• python-csv>=0.0.11

NLP Operations

• nltk>=3.4.5
• spacy>=2.3.2
• sklearn>=0.0

Utilities

• pytest-shutil>=1.6.0
• Pillow>=2.2.1
• pyodbc>=4.0.26
• fuzzywuzzy>=0.18.0
• pycountry>=19.8
• geopy>=1.21
• python-dateutil>=2.8.1
• urllib3>=1.25
• tldextract>=2.2.2
• geotext>=0.4.0
• dateparser>=1.0.0
• uuid>=1.30
• glob2>=0.7

MrSnippets Change log

Version 1.0.0

• Initial version with base features

Version 1.0.1

• Included MySQL Wrappers with common functions
• Included Mongo Wrapper with common functions
• Improved function operations
• function documents are included

Version 1.1.1

• Included Image Processing functions based on PIL and openCV
• fuzz Module removed (string compare function) which creating issue in installing the package
• all known bugs are fixed
• performance optimisations
• multiple methods are included for web_client
• more functions are added in helper module
• Added SQL wrapper
• Added missing library's in soup module

Version 2.0.0

• Soup module renamed as data_mining module & Added many more functions
• Helper module renamed as utilities module & Added many more functions
• data converter module added which supports data conversion
• Added data_support Module with more data functions
• Now NLP module has more functionalities
• Included doc string for each and every functions
• Many Known bugs are fixed
• Document string added in all the functions
• Improved requirement file

Acknowledgments

• Based on my experience am developing and including more functions to this package.
• In upcoming days am planning to include as many functions as possible
• Any active contributors are welcome. Please free to reach out via email.