This library is utility library from digipodium
Project description
A python library which can be used to extraxct data from files, pdfs, doc(x) files, as well as save data into these files. This library can be used to scrape and extract webpage data from websites as well.
Installation Requirements and Instructions
Python versions 3.8 or above should be installed. After that open your terminal: For Windows users:
pip install dputils
For Mac/Linux users:
pip3 install dputils
Files Modules
Functions from dputils.files:
-
get_data:
-
To import, use statement:
from dputils.files import get_data
-
Obtains data from files of any extension given as args(supports text files, binary files, pdf, doc for now, more coming!)
-
sample call:
content = get_data(r"sample.docx") print(content)
-
Returns a string or binary data depending on the output arg
-
images will not be extracted
-
-
save_data:
- save_data can be used to write and save data into a file of valid extension.
- sample call:
from dputils.files import save_data pdfContent = save_data("sample.pdf", "Sample text to insert") print(pdfContent)
- Returns True if file is successfully accessed and modified. Otherwise, False.
Scrape Modules
Data extraction from a page
Here's a basic tutorial to help you get started with the scraper
module.
- Import the required classes and functions:
from dputils.scraper import Scraper, Tag
- Initialize the
Scraper
class with the URL of the webpage you want to scrape:
url = "https://www.example.com"
scraper = Scraper(url)
- Define the tags you want to scrape using the
Tag
class:
title_tag = Tag(name='h1', cls='title', output='text')
price_tag = Tag(name='span', cls='price', output='text')
- Extract data from the page:
data = scraper.get_data_from_page(title=title_tag, price=price_tag)
print(data)
Extracting list of items from a page
For more advanced usage, such as extracting repeated data from lists of items on a page, you can use the following approach:
- Initialize the
Scraper
class:
url = "https://www.example.com/products"
scraper = Scraper(url)
- Define the tags for the target section and the items within that section:
For repeated data extraction, you need to define
Target
anditem
and pass it toget_repeating_data_from_page()
method.- target - defines the
Tag()
for area of the page containing the list of items. - items - defines the
Tag()
for repeated items within the target section. Like a product-card in product grid/list.
- target - defines the
target_tag = Tag(name='div', cls='product-list')
item_tag = Tag(name='div', cls='product-item')
title_tag = Tag(name='h2', cls='product-title', output='text')
price_tag = Tag(name='span', cls='product-price', output='text')
link_tag = Tag(name='a', cls='product-link', output='href')
- Extract repeated data from the page:
products = scraper.get_repeating_data_from_page(
target=target_tag,
items=item_tag,
title=title_tag,
price=price_tag,
link=link_tag
)
for product in products:
print(product)
These functions can used on python versions 3.8 or greater.
References for more help: https://github.com/digipodium/dputils
Thank you for using dputils!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.