smlep-news 0.3.1

Custom news scrapper

Essential

This package is a simple gatherer of different news from various APIs.

Currently, it sends all news gathered through e-mails to all the mail addresses from a database.

To send a simple mail to a target without using any database, use the function prepare_mail from mail.

Currently the gathered news are the following:

• Weather from Open Weather Map
• Top products from Product Hunt
• Trending repos from GitHub
• News from Guardian
• News from Le Figaro (in French)
• News from the New York Times
• Exchange rates from exchangeratesapi.io
• Stories from Medium
• Stories from Hacker News

The package can be installed through the official pypi repository with:

pip install smlep-news

Architecture

This project is divided in different packages:

• exchangerates: fetches currency exchange rates from exchangeratesapi.io/.
• figaro: fetches french news from Le Figaro.
• github: fetches trending repos from GitHub.
• guardian: fetches us news from Guardian.
• hackernews: fetches top stories list and stories details from Hacker News.
• mail: calls gatherer from the other packages, format their content and send mail.
• medium: extracts top stories from a specific Medium topic.
• nytimes: fetches news from the New York Times
• product_hunt: fetches top products from Product Hunt.
• runner: gathers e-mail addresses from a database and send the gathered news through e-mails. This package configuration is relative to my own setup, if you want to use this program, you should bring changes to this package (or not use it).
• weather: fetches weather from Open Weather Map. To choose the city you have to provide an ID, a list of city IDs can be downloaded here.

Credentials

This program uses APIs from different sources, and some of them require authentication. You'll have to provide API Keys in environment variables:

• PH_CLIENT_ID/PH_CLIENT_SECRET: Product Hunt Api Client Id/Client Secret.
• WEATHER_KEY: Open Weather Map Api Key.
• GUARDIAN_KEY: Guardian Api Key.
• NYT_KEY: New york times Api key

All these keys can be obtained freely on their respective websites.

To send mails, you need an email account, the default configuration is for gmail, if you need to use another mail server, you should update the code. If you use gmail, you only need to provide some environment variables:

• EMAIL_NAME
• EMAIL_USERNAME
• EMAIL_PASSWORD

Why and how I created this project

I wrote a small post about this project on my GitHub Page, you can read it here.