scrapesome 0.0.9

A Powerful Web Scraper with dynamic rendering support.

ScrapeSome

ScrapeSome is a lightweight, flexible web scraping library with both synchronous and asynchronous support. It includes intelligent fallbacks, JavaScript page rendering, response formatting (HTML → Text/JSON/Markdown), and retry mechanisms. Ideal for developers who need robust scraping utilities with minimal setup.

---

Table of Contents

• 💡 Why Use ScrapeSome?
• 🚀 Features
• ⚖ Comparison with Alternatives
• 📦 Installation
• Playwright Setup
  • Windows
  • Linux (Ubuntu/Debian)
  • macOS
• ⚡ Quick Start
• 🖥️ CLI Usage
• 🧰 Advanced Usage
• 🧪 Testing
• ⚙️ Environment Configuration
• 📄 Output Formats
• 📁 Project Structure
• 🔒 License
• 🤝 Contributions

💡 Why Use ScrapeSome?

• Handles both static and JS-heavy pages out of the box
• Supports both sync and async scraping
• Converts raw HTML into clean text, JSON, or Markdown
• Works with minimal configuration (pip install scrapesome)
• Handles timeouts, retries, redirects, user agents

🚀 Features

• 🔁 Sync + Async scraping support
• 🔄 Automatic retries and intelligent fallbacks
• 🧪 Playwright rendering fallback for JS-heavy pages
• 📝 Format responses as raw HTML, plain text, Markdown, or structured JSON
• ⚙️ Configurable: timeouts, redirects, user agents, and logging
• 🧪 Test coverage with pytest and pytest-asyncio

---

⚖ Comparison with Alternatives

Feature	ScrapeSome ✅	Scrapy	Selenium/UC	Playwright (Raw)
✅ Sync + Async Scraping	✅ Built-in	❌ Async only*	❌ Manual	❌ Manual
🧠 JS Rendering (Fallback)	✅ Seamless	❌ Plugin setup	✅ Full	✅ Full
📝 Output as JSON/Markdown/HTML	✅ Built-in	❌ Requires custom	❌ Manual parsing	❌ Manual parsing
🔁 Retry & Timeout Handling	✅ Built-in	⚠️ Requires config	❌ Manual	❌ Manual
⚡ Minimal Setup (Boilerplate)	✅ Near zero	❌ Needs project	❌ Driver setup	❌ Browser install
🧪 Testable out-of-the-box	✅ Pytest-ready	⚠️ Complex	❌	❌
🛠️ Config via .env or inline	✅ Simple	⚠️ Complex	❌	❌
📦 Install & Run in <1 Min	✅ Yes	❌	❌	❌

📦 Installation

pip install scrapesome

Playwright Setup

ScrapeSome uses Playwright for JavaScript rendering fallback. To enable this, you need to install Playwright and its dependencies.

1. Install Playwright Python package if not installed

pip install playwright

2. Install Playwright browsers

playwright install

3. Install system dependencies

Playwright requires some system libraries to run browsers, which vary by operating system.

For Windows Playwright installs everything you need automatically with playwright install, so no additional setup is usually required.

For Linux (Ubuntu/Debian) Run the following command to install required system libraries:

playwright install-deps

If you don't have playwright CLI available, you can install dependencies manually:

sudo apt-get update
sudo apt-get install -y libwoff1 libopus0 libwebp6 libharfbuzz-icu0 libwebpmux3 \
                        libenchant-2-2 libhyphen0 libegl1 libglx0 libgudev-1.0-0 \
                        libevdev2 libgles2 libx264-160

Note: Package names may vary depending on your distribution and version.

For macOS You can install required libraries using Homebrew:

brew install harfbuzz enchant

After this setup, you should be able to use ScrapeSome with full Playwright rendering support!

⚡ Quick Start

Synchronous Example

from scrapesome import sync_scraper
html = sync_scraper("https://example.com")
html

Asynchronous Example

import asyncio
from scrapesome import async_scraper
html = asyncio.run(async_scraper("https://example.com"))
html

🖥️ CLI Usage

ScrapeSome also includes a powerful CLI for quick and easy scraping from the command line.

📦 Installation with CLI Support

To use the CLI, install with the optional cli extras:

pip install scrapesome[cli]

🔧 Basic Usage

scrapesome scrape --url https://example.com

This performs a synchronous scrape and outputs plain text by default.

⚙️ Available Options

Option	Description	Default
--async-mode	Use asynchronous scraping	False
--force-playwright	Force JavaScript rendering using Playwright	False
--output-format	Choose text, json, markdown, or html	html

Examples

Basic scrape

scrapesome scrape --url https://example.com

Force Playwright rendering

scrapesome scrape --url https://example.com --force-playwright

Get JSON output

scrapesome scrape --url https://example.com --output-format json

Async scrape with markdown output

scrapesome scrape --url https://example.com --async-mode --output-format markdown

🧰 Advanced Usage

Force Rendering (Playwright)

from scrapesome import sync_scraper
content = sync_scraper("https://example.com", force_playwright=True)
content

Custom User Agents

from scrapesome import sync_scraper
content = sync_scraper("https://example.com", user_agents=["MyCustomAgent/1.0"])
content

Control Redirects

from scrapesome import sync_scraper
content = sync_scraper("https://example.com", allow_redirects=False)
content

similarly async_scraper can also be used.

🧪 Testing

Run tests with:

pytest --cov=scrapesome tests/

Target coverage: 75–100%

⚙️ Environment Configuration

ScrapeSome reads from environment variables if a .env file is present.

Example .env

LOG_LEVEL=INFO
OUTPUT_FORMAT=text
FETCH_PLAYWRIGHT_TIMEOUT=10
FETCH_PAGE_TIMEOUT=10
USER_AGENTS=["Mozilla/5.0 (Windows NT 10.0; Win64; x64)......."]

Key	Description
FETCH_PLAYWRIGHT_TIMEOUT	Timeout for Playwright-rendered pages (in seconds)
FETCH_PAGE_TIMEOUT	Timeout for standard page fetch (in seconds)
LOG_LEVEL	Logging verbosity (DEBUG, INFO, WARNING, etc.)
OUTPUT_FORMAT	Default output format (text, markdown, json, html)
USER_AGENTS	Default user agents ("Mozilla/5.0 (Windows NT 10.0; Win64; x64).......")

📄 Output Formats

JSON Example

Get json version

from scrapesome import sync_scraper
content = sync_scraper("https://example.com", output_format_type="json")
content

Output

{
  "title": "Example Domain",
  "description": "This domain is for use in illustrative examples.",
  "url": "https://example.com"
}

Markdown

Convert HTML to Markdown with:

from scrapesome import sync_scraper
content = sync_scraper("https://adenuniversity.us", output_format_type="markdown")
content

Output

# Online Global Masters that boost your global career

**ADEN University** offers students access to professionals who operate in the world of business and administration, who share their knowledge and acumen collaboratively with their students in all **academic programs** offered at ADEN.

[About Us](about-aden-university)


Watch testimonial video 


##### Watch testimonial video

×

[

](https://res.cloudinary.com/cruminott/video/upload/vc_auto,w_auto,q_auto,f_auto/adenu/aden-university-3.mp4)



## ADEN University offers the following academic programs

[![EXECUTIVE MBA. Master of Business Administration](https://adenuniversity.us/files/2016/06/ADENU_miniatura_Emba_900-1-820x400.jpg "EXECUTIVE MBA. Master of Business Administration")](https://adenuniversity.us/academics/executive-mba/  "EXECUTIVE MBA. Master of Business Administration")

##### [EXECUTIVE MBA. Master of Business Administration](https://adenuniversity.us/academics/executive-mba/ "EXECUTIVE MBA. Master of Business Administration")

The ADEN University Executive MBA is designed to strengthen business leaders to manage...

* **37** credits
* **15** months
* **Spanish Only**

[Visit EMBA Course](https://adenuniversity.us/academics/executive-mba/ "EXECUTIVE MBA. Master of Business Administration")

[![GLOBAL MBA. Master of Business Administration](https://adenuniversity.us/files/2016/06/ADENU_miniatura_MBAgl1_900-820x400.jpg "GLOBAL MBA. Master of Business Administration")](https://adenuniversity.us/academics/global-mba/  "GLOBAL MBA. Master of Business Administration")

##### [GLOBAL MBA. Master of Business Administration](https://adenuniversity.us/academics/global-mba/ "GLOBAL MBA. Master of Business Administration")

The Global MBA is designed to prepare business leaders to manage companies in an...

* **36** credits
* **14** months
* **Spanish and English**

similarly async_scraper can also be used.

📁 Project Structure

scrapesome/
├── .gitignore
├── pytest.ini
├── .github/
│   ├── workflows/
│   │   └── deploy.yml
│   ├── ISSUE_TEMPLATE/
│   │   └── index.md
│   ├── PULL_REQUEST_TEMPLATE.md
│   ├── CODE_OF_CONDUCT.md
│   └── SECURITY.md
├── __init__.py
├── cli.py
├── config.py
├── exceptions.py
├── formatter/
│   ├── __init__.py
│   └── output_formatter.py
├── logging.py
├── scraper/
│   ├── __init__.py
│   ├── async_scraper.py
│   ├── sync_scraper.py
│   └── rendering.py
├── docs/
│   ├── index.md
│   ├── getting_started.md
│   ├── usage.md
│   ├── config.md
│   ├── examples.md
│   ├── cli.md
│   ├── about.md
│   └── licence.md
├── tests/
│   ├── __init__.py
│   ├── test_sync_scraper.py
│   ├── test_async_scraper.py
│   └── test_config.py
├── setup.py
├── requirements.txt
├── pyproject.toml
├── LICENSE
└── README.md

🔒 License

MIT License © 2025

🤝 Contributions

Contributions are welcome! Whether it's bug reports, feature suggestions, or pull requests — your help is appreciated.

To get started:

git clone https://github.com/scrapesome/scrapesome.git
cd scrapesome

Documentation & Community

• Contributing Guidelines
• Code of Conduct
• Issue Templates
• Pull Request Templates
• GitHub Discussions