Python web crawler / scraper for WG-Gesucht. Crawls the WG-Gesucht site for new apartment listings and send a message to the poster, based off your saved filters and saved text template.
Installation
$ pip install wg-gesucht-crawler-cli
Or, if you have virtualenvwrapper installed:
$ mkvirtualenv wg-gesucht-crawler-cli $ pip install wg-gesucht-crawler-cli
Use
Can be run directly from the command line with:
$ wg-gesucht-crawler-cli --help
Or if you want to use it in your own project:
from wg_gesucht.crawler import WgGesuchtCrawlerJust make sure to save at least one search filter as well as a template text on your wg-gesucht account.
Free software: MIT license
Documentation: https://wg-gesucht-crawler-cli.readthedocs.org.
Features
Searches https://wg-gesucht.de for new WG ads based off your saved filters
Sends your saved template message and applies to all matching listings
Reruns every ~5 minutes
Run on a RPi or free EC2 micro instance 24/7 to always be one of the first to apply for new listings
Getting Caught with reCAPTCHA
I’ve made the crawler sleep for 5-8 seconds between each request to try and avoid their reCAPTCHA, but if the crawler does get caught, you can sign into your wg-gesucht account manually through the browser and solve the reCAPTCHA, then start the crawler again.
If it continues to happen, you can also increase the sleep time in the get_page() function in wg_gesucht.py
History
Pre-release
1 maintainer
Metadata
1 maintainer
Release files for wg-gesucht-crawler-cli 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| wg-gesucht-crawler-cli-0.2.1.tar.gz | 33.9 kB | Without provenance | Details for wg-gesucht-crawler-cli-0.2.1.tar.gz |
Release files / wg-gesucht-crawler-cli-0.2.1.tar.gz
| Download URL | wg-gesucht-crawler-cli-0.2.1.tar.gz |
|---|---|
| Size | 33.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2a3be453b1ff8ae9660808c1ec28768ac3d5d66cacf490445f30cefcfc6dbab6
|
|
BLAKE2b-256 checksum How to use checksums |
3ac7a5177f8c5cb5fa73e76a95d1709baf2738fdfbe53926496ffe80fcfa60d2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.3 requests-toolbelt/0.8.0 tqdm/4.30.0 CPython/3.6.7
|
Security for wg-gesucht-crawler-cli 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
