resumesh-scrapers

Enterprise-Grade, Standalone, and Decoupled Web Scraping Engines for Portfolio Building

resumesh-scrapers is a clean, modular, and robust Python library containing standalone scraping services for GitHub, Dev.to, Medium, Substack, Behance, NPM, PyPI, and YouTube platforms. Originally developed as part of the ResuMesh ecosystem, it has been decoupled to serve as a reusable package for any resume, portfolio, or profile aggregator project.

🚀 Key Features

• Robust Core Network Engine: Centralized HTTP request handling with standard retry mechanisms powered by tenacity and structured logging.
• Fully Decoupled Models: Independent, clean data validation using pydantic v2, keeping data structures separated from database constraints.
• Extensible Architecture: Platform scrapers are isolated in a plug-and-play layout under platforms/ allowing quick addition of new integrations (e.g. LinkedIn, GitLab).
• Detailed Exceptions: Standardized exceptions hierarchy inheriting from ScraperError with HTTP status code details.

🛠️ Installation

pip install resumesh-scrapers

For Local Development (Editable mode)

git clone https://github.com/AtaCanYmc/resumesh-scrapers.git
cd resumesh-scrapers
pip install -e .

💡 Quick Start

Here is a simple example showing how to scrape your repository, blog statistics, and YouTube videos:

import asyncio
from resumesh_scrapers import (
    GitHubScraper,
    DevToScraper,
    MediumScraper,
    SubstackScraper,
    BehanceScraper,
    YouTubeScraper,
    GitHubRepositoryModel,
    GitHubCommitModel,
    GitHubUserModel,
    DevToArticleModel,
    MediumEntryModel,
    SubstackEntryModel,
    BehanceProjectModel,
    YouTubeVideoModel,
)

async def main():
    # 1. Scraping GitHub Repositories, README Repos, Commits, and User Lists
    github = GitHubScraper()
    repos: list[GitHubRepositoryModel] = await github.fetch_data(
        username="octocat",
        pat="ghp_...",  # optional PAT token to bypass rate limit
        include_forks=False
    )
    print(f"Fetched {len(repos)} repositories.")

    # Fetch profile README repository details
    readme_repo = await github.fetch_readme_repo("octocat")
    if readme_repo:
        print(f"Profile README Repository: {readme_repo.html_url}")

    # Fetch recent commits (default since: last 7 days)
    commits: list[GitHubCommitModel] = await github.fetch_commits("octocat")
    print(f"Fetched {len(commits)} commits.")

    # Fetch followers and following list
    followers: list[GitHubUserModel] = await github.fetch_followers("octocat", per_page=10)
    print(f"Fetched {len(followers)} followers.")


    # 2. Scraping Dev.to Articles
    devto = DevToScraper()
    devto_posts: list[DevToArticleModel] = await devto.fetch_data("atacanymc")
    print(f"Fetched {len(devto_posts)} Dev.to articles.")

    # 3. Scraping Substack Publications
    substack = SubstackScraper()
    substack_posts: list[SubstackEntryModel] = await substack.fetch_data("atacan")
    print(f"Fetched {len(substack_posts)} Substack posts.")

    # 4. Scraping Behance Projects
    behance = BehanceScraper()
    behance_projects: list[BehanceProjectModel] = await behance.fetch_data(
        username="atacanymc",
        api_key="your_behance_api_key"  # optional, falls back to HTML scraping if not provided
    )
    print(f"Fetched {len(behance_projects)} Behance projects.")

    # 5. Scraping YouTube Video Metadata
    youtube = YouTubeScraper()
    video_data: YouTubeVideoModel = await youtube.fetch_video("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
    print(f"Fetched YouTube video: {video_data.title} ({video_data.view_count:,} views)")

if __name__ == "__main__":
    asyncio.run(main())

🗺️ Clean Architecture

The codebase has been refactored to enforce separation of concerns, decoupling models, platforms, and network layers:

src/resumesh_scrapers/
├── core/                       # Shared network and utility systems
│   ├── client.py               # Central HTTP requester with tenacity retries
│   └── __init__.py
├── platforms/                  # Individual platform scrapers
│   ├── github.py
│   ├── devto.py
│   ├── medium.py
│   ├── substack.py
│   ├── behance.py
│   ├── npm.py
│   ├── pypi.py
│   ├── youtube.py
│   └── __init__.py
└── models/                     # Platform-specific Pydantic validation schemas
    ├── github.py
    ├── devto.py
    ├── medium.py
    ├── substack.py
    ├── behance.py
    ├── npm.py
    ├── pypi.py
    ├── youtube.py
    └── __init__.py

📊 Models & Capabilities

⚠️ Exception Handling

All scraper exceptions inherit from ScraperError to simplify integration errors:

from resumesh_scrapers.exceptions import ScraperError, GitHubScraperError

try:
    repos = await github_scraper.fetch_data("some_username")
except GitHubScraperError as e:
    print(f"GitHub API Error: {e.message} (HTTP {e.status_code})")
except ScraperError as e:
    print(f"Generic Scraping Exception: {e}")

🤝 Contributing

We welcome contributions to add more platforms (such as LinkedIn, GitLab, or Dribbble) or optimize parsers. Please open a Pull Request or file an issue to discuss your ideas!

1. Fork the Project.
2. Create your Feature Branch (git checkout -b feature/AmazingFeature).
3. Commit your Changes (git commit -m 'Add some AmazingFeature').
4. Push to the Branch (git push origin feature/AmazingFeature).
5. Open a Pull Request.

📄 License

Distributed under the Apache 2.0 License. See LICENSE for more information.