dsci524-group29-webscraping 1.1.4

A simple Python toolkit for web scraping

dsci524_group29_webscraping

A Python package for simplified web scraping functionality for data scientists new to web scraping.

Installation

$ pip install dsci524_group29_webscraping

Functions

• fetch_html(url): Retrieves the raw HTML content from the specified URL, handling HTTP requests and potential errors.
• parse_content(html, selector, selector_type): Parses the provided HTML content using CSS selectors or XPath to extract specified data.
• save_data(data, format, destination): Saves the extracted data into the desired format (e.g., TXT, CSV, JSON) at the specified destination path.

Usage

Below are examples demonstrating how to use the main functions in this package:

1. Fetch HTML Content

from dsci524_group29_webscraping import fetch_html

# Fetch the raw HTML content from a webpage
url = "https://example.com"
html_content = fetch_html(url)
print(html_content)  # Outputs the HTML content of the page

2. Parse Content

from dsci524_group29_webscraping import parse_content

# Parse the HTML content to extract specific elements
selector = "h1"  # Example: extract all <h1> elements
selector_type = "css"  # Use CSS selectors
extracted_data = parse_content(html_content, selector, selector_type)
print(extracted_data)  # Outputs a list of the extracted data

3. Save Data

from dsci524_group29_webscraping import save_data

# Save the extracted data to a CSV file
data = [{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]  # Example data
file_path = save_data(data, format="csv", destination="output.csv")
print(f"Data saved to: {file_path}")

This package simplifies the process of fetching, parsing, and saving web data, making it ideal for beginners.

Python Ecosystem

While libraries like BeautifulSoup and Scrapy offer comprehensive web scraping capabilities, dsci524_group29_webscraping aims to provide a more streamlined and beginner-friendly approach. By focusing on three core functions, it abstracts the complexities involved in web scraping, making it accessible for quick tasks and educational purposes.

Similar Packages:

• webscraping: Provides web scraping functions but contains a rich set of functionality that is beyond beginner level.
• webscraping_tools: Offers similar functionalities and many more that in our opinion, places it in the intermediate level.

dsci524_group29_webscraping differentiates itself by offering a simple set of functions that do the job for simple, beginner level needs.

Contributors

• Lixuan Lin
• Hui Tang
• Sienko Ikhabi

Contributing

Interested in contributing? Check out the contributing guidelines.

Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by the specified terms.

License

Package dsci524_group29_webscraping was created by Lixuan Lin, Hui Tang and Sienko Ikhabi for the Master of Data Science, University of British Columbia. It is licensed under the terms of the MIT license.

Credits

This project was created with cookiecutter from the py-pkgs-cookiecutter template.