A library which does some basic web scraping operations
Project description
easierscrape
A library for basic web scraping.
Overview
easierscrape is a library which helps users do some basic web scraping operations. Oftentimes when doing webscraping code is written and re-written with slightly changed parameters to fit the website to be scraped from. This library is an easy to use tool that can scrape essentials from websites (tables, links, files, etc.). It also has the ability to generate hyperlink trees.
Details
This project is a pure python project using modern tooling. It uses a Makefile
as a command registry, with the following commands:
make
: list available commandsmake develop
: install and build this library and its dependencies usingpip
make build
: build the library usingsetuptools
make lint
: perform static analysis of this library withflake8
andblack
make format
: autoformat this library usingblack
make annotate
: run type checking usingmypy
make test
: run automated tests withpytest
make coverage
: run automated tests withpytest
and collect coverage informationmake dist
: package library for distribution
Basic Usage
Install with pip: pip install easierscrape
Import needed methods from easierscrape
as seen below:
from easierscrape import (
parse_anchors,
parse_files,
parse_images,
parse_tables,
print_tree,
tree_gen,
)
parse_anchors
- Takes in one argument: The
url
to scrape from - Returns a list of anchor tag references
parse_files
- Takes in two arguments:
- The
url
to scrape from - A
List
of file extensions to parse (ex:['pdf', 'txt']
)
- The
- Downloads results to an
easierscrape_downloads
folder in working directory
parse_images
- Takes in one argument: The
url
to scrape from - Downloads results to an
easierscrape_downloads
folder in working directory
parse_tables
- Takes in one argument: The
url
to scrape from - Downloads results to an
easierscrape_downloads
folder in working directory
print_tree
- Takes in an anytree
Node
structure (head of a tree) - Prints the tree
tree_gen
- Takes in two arguments;
- The head
url
to scrape from - The
maxdepth
to scrape until
- The head
- Returns the head anytree
Node
of a generated scrape tree
Command Line Usage
When installed, you can invoke easierscrape from the command-line:
usage: easierscrape [-h] url depth
positional arguments:
url the url to scrape
depth the depth of the scrape tree
optional arguments:
-h, --help show this help message and exit
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.