A telegram bot to stay tuned on real estate ads
Project description
Scraper Bot
This is a bot thought to do periodical scraping of ads from commercial websites.
Found a new ad the bot will send it to you exploiting Apprise channels
Deploy
Pypi
The relative package is available on Pypi
pip install scraper-bot
The package heavily relays on playwright
package, so before start to use the bot you have to install a playwright browser
playwright install --with-deps firefox
You can found further information in the playwright
documentation
(n.b. the bot are not limited to use firefox only)
The scraper-bot
package provide the following command to run the bot
scraper-bot
Container
The CI builds the container for each version and it puts it on the public GitHub registry
ghcr.io/robertobochet/scraper-bot
docker compose
- Create a telegram bot and retrieve its token
- Download
config.example.yaml
and rename it toconfig.yaml
- Change the configuration follow the guidelines
- Download
docker-compose.yaml
- Start the scraper with
docker-compose
docker-compose up
- Wait that the bot does its work!
Kubernetes (Helm chart)
For the deploy of the Scraper Bot is also available a helm chart
You can found the source code in the repo scraper-bot-chart
Helm chart package is available in the github OCI registry
oci://ghcr.io/robertobochet/scraper-bot-chart
You can use it to directly deploy on your kubernetes cluster
- Retrieve the default values file
helm show values oci://ghcr.io/robertobochet/scraper-bot-chart > values.yaml
- Customize the
values.yaml
- Install the scaper bot
helm install oci://ghcr.io/robertobochet/scraper-bot-chart scraper-bot -f values.yaml
Configuration
By default the bot looks for a configuration file in the following path ./config.y(a)ml
and /etc/scaraper-bot/config.y(a)ml
. You cna override this behavior passing via command line the --config
argument followed by the config file path
scraper-bot --config /path/to/scraper-bot-config.yaml
The configuration file has to satisfy the pydantic model which you can find in scraper_bot.settings
.
Furthermore you can get the config json schema from command line with --config-schema
argument
scraper-bot --config-schema
You can also find a configuration example in config.example.yaml
.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for scraper_bot-1.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 45feba5eb3867028ec2809c9695bf8c237d883e22b1e3675f04b57a604411152 |
|
MD5 | c6d3a2e00394401ed4aad47b041cab0f |
|
BLAKE2b-256 | 2e49a122e1c9ad6f03ffc2d71a71b32652bb660386848992ee0bbce45526ca65 |