ek-scraper

Simple scraper for kleinanzeigen.de searches with notifications for new ads.

Installation

Install this package from PyPi in a separate virtual environment using uv.

uv tool install ek-scraper

Usage

For the full usage check the ek-scraper --help command

Create a configuration file using

ek-scraper create-config <path/to/config.json>

The example configuration file will look like this:

{
  "filter": {
    "exclude_topads": true,
    "exclude_patterns": []
  },
  "notifications": {
    "pushover": {
        "token": "<your-app-api-token>",
        "user": "<your-user-api-token>",
        "device": []
    },
    "ntfy.sh": {
      "topic": "<your-private-topic>",
      "priority": 3
    },
    "email": {
      "smtp_host": "<smtp.example.com>",
      "smtp_port": "587",
      "username": "<your-account-username>",
      "password": "<your-account-password>",
      "sender": "<sender@example.com>",
      "recipient": ["<receiver@other.example.com>","<receiver@another.example.com>"],
      "bcc": ["<blindreceiver@third.example.com>","<blindreceiver@fourth.example.com>"]
    }
  },
  "searches": [
    {
      "name": "Wohnungen in Hamburg Altona",
      "url": "https://www.kleinanzeigen.de/s-wohnung-mieten/altona/c203l9497",
      "recursive": true
    }
  ]
}

See Configuration for details on all configuration options.

• Configure one or more searches in the searches section of the configuration, see Searches for more details
• Configure notifications in the notifications section of the configuration, see Notifications for details on notification configuration
• (Optional) Configure filters in the filter section of the configuration, see Filter for more details

Run the following command to initialize the data store without sending any notifications:

ek-scraper run --no-notifications path/to/config.json

Afterwards, run

ek-scraper run path/to/config.json

to receive notifications according to your notifications configuration.

Development

Follow the steps below to set up a development environment for this project.

1. Clone this repository

   git clone git@github.com:jonasehrlich/ek-scraper.git
   

2. Change directory into the repository

   cd ek-scraper
   

3. Create a virtual environment using uv

   uv sync
   

4. (Optional) Install pre-commit environment using pre-commit or prek

Configuration

Searches

Searches can be configured in the searches array of the configuration file. Each of the searches can be configured with the following parameters.

Filter

Filters can be configured in the filter section of the configuration file to exclude specific ads from your scrape results on the client side. The following settings can be configured.

Notifications

Notifications can be configured in the notifications section of the configuration file.

Push notifications using Pushover

ek-scraper supports push notifications to your devices using Pushover. For further information on the service check their terms and conditions.

The implementation for Pushover notifications will send a single notification per search, if new ads are discovered.

To configure Pushover for notifications from the scraper, first register at the service and create an application (e.g. ek-scraper). To use the service in ek-scraper, add the pushover object to the notifications object in your configuration file and fill the API tokens. Selection of the devices which will receive the notifications, is supported using the device array.

Push notifications using ntfy.sh

ek-scraper supports push notifications to your devices using ntfy.sh. For further information on the service check their terms and conditions.

The implementation for ntfy.sh notifications will send a single notification per search, if new ads are discovered.

To configure ntfy.sh for notifications from the scraper, define a topic and subscribe to it in the mobile app.

Note that topic names are public, so it's wise to choose something that cannot be guessed easily. This can be done by including a UUID, e.g. by running the following command in your shell:

echo "ek-scraper-$(uuidgen)"

To use the service in ek-scraper, add the ntfy.sh object to the notifications object in your configuration file and add the topic you previously subscribed to.

Push notifications using E-Mail

ek-scraper supports push notifications to your devices using E-Mail.

The implementation for E-Mail notifications will send a single Email containing the new results per search, if new ads are discovered.

To configure E-Mail for notifications from the scraper, you need to add your smtp server details and credentials. To use the service in ek-scraper, add the email object to the notifications object in your configuration file. Currently only supporting STARTTLS.

Running ek-scraper regularly

It should be avoided to run the tool too often to avoid getting your IP address blocked by kleinanzeigen.de

In order to run ek-scraper regularly on a Unix-like system, configure it as a cronjob.

To configure a cronjob, run

crontab -e

Edit the crontab table to run the command you want to run. A handy tool to check schedule configurations for cronjobs is crontab.guru.

For more information on configuring cronjobs use your favorite search engine.