vnewsapi 0.1.0

A Python library for crawling Vietnamese financial news from major news websites

vnewsapi

A Python library for crawling Vietnamese financial news from major news websites. Provides RSS feed parsing, sitemap parsing, content extraction, and batch processing capabilities.

Table of Contents

• Features
• Installation
• Quick Start
• Supported News Sources
• API Reference
• Examples
• Configuration
• Requirements
• Contributing
• License
• Links

Features

• RSS Feed Parsing: Fetch and parse RSS feeds from news websites
• Sitemap Parsing: Extract article URLs from XML sitemaps
• Content Extraction: Extract article content using CSS selectors
• Batch Processing: Fetch multiple articles with rate limiting
• Multiple Sources: Support for 10+ Vietnamese news websites
• Easy to Use: Simple, intuitive API
• Error Handling: Comprehensive error handling and logging
• Rate Limiting: Built-in request delay management

Installation

Install from PyPI:

pip install vnewsapi

Or install from source:

git clone https://github.com/Hoanganhvu123/vnewsapi.git
cd vnewsapi
pip install -e .

Quick Start

Basic Usage

from vnewsapi import Crawler

# Initialize crawler for a news site
crawler = Crawler('cafef')

# Get latest articles
articles = crawler.get_articles(limit=10)
print(articles)

# Get article details
details = crawler.get_article_details('https://cafef.vn/article-url')
print(details['title'])
print(details['content'])

Batch Crawling

from vnewsapi import BatchCrawler

# Initialize batch crawler
batch = BatchCrawler('cafef', request_delay=0.5)

# Get article list
crawler = Crawler('cafef')
articles = crawler.get_articles(limit=20)

# Fetch details for all articles
details_df = batch.fetch_details_from_dataframe(articles, url_column='link')
print(details_df)

RSS and Sitemap Parsing

from vnewsapi import RSS, Sitemap

# Parse RSS feed
rss = RSS()
articles = rss.fetch('https://cafef.vn/rss/tin-moi-nhat.rss')
print(articles)

# Parse sitemap
sitemap = Sitemap()
urls_df = sitemap.run('https://cafef.vn/sitemap.xml', limit=100)
print(urls_df)

Supported News Sources

The library supports the following Vietnamese news websites:

• CafeF (cafef) - Financial news
• CafeBiz (cafebiz) - Business news
• VietStock (vietstock) - Stock market news
• VnExpress (vnexpress) - General news
• Bao Dau Tu (baodautu) - Investment news
• Tuoi Tre (tuoitre) - General news
• Thanh Nien (thanhnien) - General news
• Dan Tri (dantri) - General news
• Vietnam Economy (vneconomy) - Economic news
• VietnamNet (vietnamnet) - General news

API Reference

Crawler

Main class for fetching news articles.

crawler = Crawler(site_name, custom_config=None, timeout=30)

Parameters:

• site_name (str): Name of the news site (e.g., 'cafef', 'vietstock')
• custom_config (dict, optional): Custom configuration override
• timeout (int): Request timeout in seconds (default: 30)

Methods:

• get_articles(limit=None, sitemap_url=None, rss_url=None, prefer_rss=True): Get list of articles
  • Returns: pandas DataFrame with columns: title, link, pubDate, description
• get_article_details(url): Get detailed content of a single article
  • Returns: dict with keys: title, content, content_html, short_desc, publish_time, author, url

BatchCrawler

Batch processing for multiple articles.

batch = BatchCrawler(site_name, request_delay=0.5, timeout=30)

Parameters:

• site_name (str): Name of the news site
• request_delay (float): Delay between requests in seconds (default: 0.5)
• timeout (int): Request timeout in seconds (default: 30)

Methods:

• fetch_details_for_urls(urls, show_progress=True): Fetch details for multiple URLs
  • Returns: pandas DataFrame with article details
• fetch_details_from_dataframe(df, url_column='link'): Fetch details from DataFrame
  • Returns: pandas DataFrame merged with article details

RSS

RSS feed parser.

rss = RSS(timeout=30, data_source='CAFEF')
articles = rss.fetch(rss_url)

Parameters:

• timeout (int): Request timeout in seconds (default: 30)
• data_source (str): Data source name for headers (default: 'CAFEF')

Methods:

• fetch(rss_url): Fetch and parse RSS feed
  • Returns: List of dicts with keys: title, link, pubDate, description

Sitemap

Sitemap parser.

sitemap = Sitemap(timeout=30, data_source='CAFEF')
urls_df = sitemap.run(sitemap_url, limit=None)

Parameters:

• timeout (int): Request timeout in seconds (default: 30)
• data_source (str): Data source name for headers (default: 'CAFEF')

Methods:

• run(sitemap_url, limit=None): Fetch and parse sitemap
  • Returns: pandas DataFrame with columns: loc, lastmod

Examples

Example 1: Get Latest Articles

from vnewsapi import Crawler

crawler = Crawler('cafef')
articles = crawler.get_articles(limit=10)

for idx, row in articles.iterrows():
    print(f"{idx+1}. {row['title']}")
    print(f"   URL: {row['link']}")
    print(f"   Date: {row.get('pubDate', 'N/A')}")
    print()

Example 2: Extract Article Content

from vnewsapi import Crawler

crawler = Crawler('vietstock')
articles = crawler.get_articles(limit=1)

if not articles.empty:
    article_url = articles.iloc[0]['link']
    details = crawler.get_article_details(article_url)
    
    print(f"Title: {details['title']}")
    print(f"Author: {details.get('author', 'N/A')}")
    print(f"Publish Time: {details.get('publish_time', 'N/A')}")
    print(f"\nContent:\n{details['content'][:500]}...")

Example 3: Batch Processing

from vnewsapi import Crawler, BatchCrawler

# Get article list
crawler = Crawler('cafef')
articles = crawler.get_articles(limit=5)

# Fetch details in batch
batch = BatchCrawler('cafef', request_delay=0.5)
details_df = batch.fetch_details_from_dataframe(articles, url_column='link')

# Filter successful fetches
successful = details_df[details_df['error'].isna()]
print(f"Successfully fetched {len(successful)} articles")

Example 4: Custom Configuration

from vnewsapi import Crawler

custom_config = {
    'name': 'Custom News Site',
    'base_url': 'https://example.com',
    'rss': {
        'urls': ['https://example.com/rss']
    },
    'sitemap': {
        'url': 'https://example.com/sitemap.xml',
        'pattern_type': 'static'
    },
    'selectors': {
        'title': {'tag': 'h1', 'class': 'article-title'},
        'content': {'tag': 'div', 'class': 'article-content'},
        'short_desc': {'tag': 'div', 'class': 'article-summary'},
        'publish_time': {'tag': 'time', 'class': 'publish-date'},
        'author': {'tag': 'span', 'class': 'author-name'}
    }
}

crawler = Crawler('cafef', custom_config=custom_config)
articles = crawler.get_articles(limit=10)

Configuration

Site Configuration Structure

Each news site requires a configuration dictionary with the following structure:

{
    'name': 'Site Name',
    'base_url': 'https://example.com',
    'rss': {
        'urls': ['https://example.com/rss/feed1.rss']
    },
    'sitemap': {
        'url': 'https://example.com/sitemap.xml',
        'pattern_type': 'static'  # or 'dynamic'
    },
    'selectors': {
        'title': {'tag': 'h1', 'class': 'title'},
        'content': {'tag': 'div', 'class': 'content'},
        'short_desc': {'tag': 'div', 'class': 'summary'},
        'publish_time': {'tag': 'span', 'class': 'time'},
        'author': {'tag': 'span', 'class': 'author'}
    }
}

Selector Configuration

Selectors can be specified in two ways:

1. Using tag and class:

{'tag': 'h1', 'class': 'title'}

2. Using CSS selector directly:

{'selector': 'h1.article-title'}

Requirements

• Python >= 3.10
• requests >= 2.31.0
• beautifulsoup4 >= 4.12.0
• lxml >= 4.9.0
• feedparser >= 6.0.0
• html2text >= 2020.1.16
• pandas >= 2.0.0
• python-dateutil >= 2.8.0

Contributing

Contributions are welcome. Please follow these steps:

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Development Setup

git clone https://github.com/Hoanganhvu123/vnewsapi.git
cd vnewsapi
python -m venv venv
venv\Scripts\activate  # Windows
source venv/bin/activate  # Linux/Mac
pip install -r requirements.txt
pip install -r requirements-dev.txt
pip install -e .

Running Tests

pytest tests/unit/ -v
pytest tests/integration/ -v -m integration

License

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer

This library is for educational and research purposes only. Please respect the terms of service of the news websites you are crawling. Always use appropriate rate limiting and be respectful of server resources. The authors are not responsible for any misuse of this library.

Links

• GitHub Repository: https://github.com/Hoanganhvu123/vnewsapi
• PyPI Package: https://pypi.org/project/vnewsapi/
• Issues: https://github.com/Hoanganhvu123/vnewsapi/issues
• Documentation: https://github.com/Hoanganhvu123/vnewsapi#readme

Acknowledgments

• Inspired by vnstock
• Uses feedparser for RSS parsing
• Uses BeautifulSoup for HTML parsing