Skip to main content

A small package that contains commonly used codes while scraping

Project description

Scrap Utils

This is small package that contains some code regularly repeated when scraping.

To install

pip install scrap-utils

To import in your program

import scrap_utils

It has the following functions:

load_json(filepath, encoding=None, errors=None, parse_float=None,
	parse_int=None, parse_constant=None)

dump_json(data, filepath, encoding=None, errors=None, indent=4, skipkeys=False,
	ensure_ascii=False, separators=None, sort_keys=False)

to_csv(dataset, filepath, mode="a", encoding=None, errors=None, newline='',
	header=True, dialect='excel', **fmtparams)

requests_get(url, trials=0, sleep_time=30, max_try=math.inf, **requests_kwargs)

requests_post(url, trials=0, sleep_time=30, max_try=math.inf, **requests_kwargs)

To-do list I'm considering:

  • remove print statements
  • add unittest
  • soup_get()
  • driver_get()
  • start_firefox()
  • read_csv()

Feel free to add your contribution here

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrap-utils-0.0.2rc1.tar.gz (3.5 kB view hashes)

Uploaded Source

Built Distribution

scrap_utils-0.0.2rc1-py3-none-any.whl (3.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page