malaysian-news-parser 0.1.4

A python library for parsing news articles

Malaysian News Parser

The Malaysian News Parser, is a python library for extracting and parsing news articles from multiple publishers. It supports both static and dynamic content parsing using BeautifulSoup and Selenium. This parser was developed with a focus on Malaysian news publishers, but this may work with other news organizations provided it is allowed.

Table of Contents

• Installation
• Usage
  • Initializing the Config Manager
  • Adding a Publisher
  • Updating a Publisher
  • Removing a Publisher
  • Retrieving Article Data
• Testing
• Contributing
• License

Installation

You can install the library using pip

pip install parser_dev

Usage

Initializing the Config Manager

Before using the library, you need to initialize the `PublisherConfigManager`:

This can be done by importing the manager from `publisher_configs`, and then initializing an instance of the manager

from parser_dev.publisher_config import PublisherConfigManager # Import Manager

# Initialize Manager instance
config_manager = PublisherConfigManager()

This was set as a global variable in the usage tests

Adding a Publisher

There are slight format differences between setting the publisher configuration for static and dynamic sites.

Static Publisher Config Format:

static_publisher_config = {
    'type': 'static', # For static, set this to static
    'headers': { # Only present for website types that are static. Omitted for Dynamic
        'header_name': 'header_value'
    },
    'body_selector': { # Article Body Selector info for beautiful soup
        'type': 'class, id, css'
        'element': '<div>, <article>, etc..., not required for css selector',
        'text': 'selector_text'
    },
    'date_selector': { # Article Date Selector info for beautiful soup
        'type': 'class, id, css',
        'element': '<div>, <article>, etc..., not required for css selector',
        'text': 'selector_text'
    },
    'author_selector': { # Article Author Selector info for beautiful soup
        'type': 'class, id, css',
        'element': '<div>, <article>, etc..., not required for css selector',
        'text': 'selector_text'
    }
}

Dynamic Publisher Config Format:

# Headers omitted, chrome_path and sleep_time added.
dynamic_publisher_config = {
    'type': 'dynamic', # For dynamic, set this to dynamic
    'chrome_path': os.getenv('CHROME_PATH') # your path to your Selenium compatible webdriver (set in your .env file)
        'body_selector': { # Article Body Selector info for beautiful soup
        'type': 'class, id, css'
        'element': '<div>, <article>, etc..., not required for css selector',
        'text': 'selector_text'
    },
    'date_selector': { # Article Date Selector info for beautiful soup
        'type': 'class, id, css',
        'element': '<div>, <article>, etc..., not required for css selector',
        'text': 'selector_text'
    },
    'author_selector': { # Article Author Selector info for beautiful soup
        'type': 'class, id, css',
        'element': '<div>, <article>, etc..., not required for css selector',
        'text': 'selector_text'
    },
    'sleep_time': 10, # set how long the webdriver should wait to render page, dtype: int
}

To add a new publisher configuration:

from parser_dev.publisher_config import PublisherConfigManager # Import Manager

# Initialize Manager instance
config_manager = PublisherConfigManager()

# desired new publisher configuration
new_publisher_config = {
    'type': 'static',
    'headers': {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
        'Accept-Language': 'en-US,en;q=0.9',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'Connection': 'keep-alive',
        'Upgrade-Insecure-Requests': '1',
    },
    'body_selector': {
        'type': 'css',
        'element': 'div',
        'text': '#new-body-selector',
    },
    'date_selector': {
        'type': 'css',
        'element': 'div',
        'text': '#new-date-selector',
    }
}

config_manager.add_publisher('new_publisher', new_publisher_config)

Updating a Publisher

To update an existing publisher configuration:

update_publisher_config = {
    'body_selector': {
        'type': 'css',
        'text': 'body > main > section > div > div > div > section > div:nth-of-type(1) > strong',
    }
}

config_manager.update_publisher('new_publisher', update_publisher_config)

Remove a Publisher

To remove a publisher configuration:

config_manager.remove_publisher('new_publisher')

Retrieving Article Data

The existing publisher config names

Publisher	Name in Config
Astro Awani	astro_awani
Malay Mail	malay_mail
Star	star
Sun Daily	sun_daily
FMT (Free Malaysia Today)	fmt
Sinar Harian	sinar_harian
Berita Harian	berita_harian
Roti Kaya	roti_kaya
New Straits Times	nst
The Vibes	the_vibes
World of Buzz	world_of_buzz
Bernama	bernama
Cili Sos	cili_sos
Coconuts	coconuts
The Edge	the_edge
HMetro	hmetro
Malaysian Gazette	malaysia_gazette
Harakah Daily	harakah_daily
Borneo Post	borneo_post

To retrieve article data from a given URL:

from parser_dev import get_parser

# url of article
url = 'https://www.astroawani.com/berita-malaysia/pelajar-cemerlang-spm-dijamin-tempat-matrikulasi-atau-program-kpt-kpm-477101'

# publisher name
publisher_name = 'astro_awani'

# Get appropriate parser
parser = get_parser(publisher_name)

# Extract and parse article data
article_data = parser.get_article_data(url)

if article_data:
    print(f"Title: {article_data['title']}")
    print(f"Author: {article_data['author']}")
    print(f"Date: {article_data['date']}")
    print(f"Body: {article_data['body'][:1000]}")  # Print the first 1000 characters to keep it concise
else:
    print("Failed to fetch or parse article data.")

Testing

To run the tests, navigate to the project directory and use `unittest`:

python -m unittest discover -s src/tests

Contributing

We welcome contributions! Please follow these steps:

1. Fork the Repository
2. Create a new branch: `git checkout -b feature/your-feature-name
3. Make your changes and commit them: `git commit -m 'Add some feature'
4. Push to the branch: `git push origin feature/your-feature-name`
5. Submit a pull request.

Please make sure to update tests as appropriate.

License

This project is licensed under the MIT License. See the LICENSE file for details.