Scrap Utils

This is small package that contains some code regularly repeated when scraping.

To install

pip install scrap-utils

Sample code

import scrap_utils as su

response = su.requests_get("https://python.org")
len(response.text)

It has the following functions:

load_json(filepath, encoding=None, errors=None, parse_float=None,
	parse_int=None, parse_constant=None)

dump_json(data, filepath, encoding=None, errors=None, indent=4, skipkeys=False,
	ensure_ascii=False, separators=None, sort_keys=False)

to_csv(dataset, filepath, mode="a", encoding=None, errors=None, newline='',
	header=True, dialect='excel', **fmtparams)

requests_get(url, trials=0, sleep_time=30, max_try=math.inf, **requests_kwargs)

requests_post(url, trials=0, sleep_time=30, max_try=math.inf, **requests_kwargs)

To-do list I'm considering:

• remove print statements
• add unittest
• soup_get()
• driver_get()
• start_firefox()
• read_csv()

Feel free to add your contribution here