TW Rental House Utility for Scrapy

This package is built for crawling Taiwanese rental house related website using Scrapy. As behavior of crawlers may differ from their goal, scale, and pipeline, this package provides only minimum feature set, which allow developer to list and decode a rental house web page into structured data, without knowing too much about detail HTML and API structure of each website. In addition, this package is also designed for extensibility, which allow developers to insert customized callback, manipulate data, and integrate with existing crawler structure.

Although this package provide the ability to crawl rental house website, it's developer's responsibility to ensure crawling mechanism and usage of data. Please be friendly to target website, such as consider using DOWNLOAD_DELAY or AUTO_THROTTLING to prevent bulk requesting.

Requirement

  1. Python 3.10+

Installation

poetry add scrapy-tw-rental-house

No browser needed. 591 renders both its list and its detail pages on the server, so every page is downloaded by plain HTTP.

591 specific

591 replies 403 to the default scrapy user agent, while it serves requests carrying no user agent at all just fine, so this package sets USER_AGENT to None unless the project configures its own.

Tests

Tests run against HTML saved under tests/fixtures, so they neither hit 591 nor need network access. Sockets are blocked while they run, so a test which crawls by accident fails instead of reaching 591.

poetry install --with dev
poetry run pytest

CLI

The package ships a twrh command for manual testing and monitoring — plain HTTP, no database, no Scrapy project needed:

twrh parse <saved-detail.html>   # offline: run the parser on a saved page
twrh detail <house-id>           # fetch + parse one detail page
twrh list 金門縣                  # fetch + parse one list page
twrh survey 花蓮縣                # full city sweep → completeness report
twrh harvest 花蓮縣               # stratified fixture harvest + manifest
twrh probe 花蓮縣                 # ratio assertions + exit code, for nightly checks

probe asserts on ratios over freshly discovered listings (list volume, detail success rate, parse rate, key-field fill rates) and can compare fill rates against a saved survey report via --baseline to catch silent selector drift. City names follow tw_regions.json; most subcommands also accept a raw URL or house id.

Basic Usage

This package currently support 591. Each rental house website is a Scrapy Spider class. You can either crawl entire website using default setting , which will take couple days, or customize the behaviour base on your need.

Note: The 591 list spider retrieves houses sorted by post date (newest first), instead of using 591's default ordering. This ensures consistent crawling behavior and helps track newly posted listings.

The most basic usage would be creating a new Spider class that inherit Rental591Spider:

from scrapy_twrh.spiders.rental591 import Rental591Spider

class MyAwesomeSpider(Rental591Spider):
    name='awesome'

And than start crawling by

scrapy crawl awesome

Please see example for detail usage.

Items

All spiders populates 2 type of Scrapy items: GenericHouseItem and RawHouseItem.

GenericHouseItem contains normalized data field, spirders from different website will decode their data and fit into this schema in best effort.

RawHouseItem contains unnormalized data field, which keep original and structured data in best effort.

Note that both item are super set of schema. It developer's responsibility to check which field is provided when receiving an item. For example, in Rental591Spider, for a single rental house, Scrapy will get:

  1. 1x RawHouseItem + 1x GenericHouseItem during listing all houses, which provide only minimun data field for GenericHouseItem
  2. 1x RawHouseItem + 1x GenericHouseItem during retrieving house detail.

Handlers

All spiders in this package provide the following handlers:

  1. start_list, similiar to start_requests in Scrapy, control how crawler issue search/list request to find all rental houses.
  2. parse_list, similiar to parse in Scrapy, control how crawler handles response from start_list and generate request for detail house info page.
  3. parse_detail, control how crawler parse detail page.

All spiders implements their own default handler, say, default_start_list, default_parse_list, and default_parse_detail, and can be overwrite during __init__. Please see example for how to control spider behavior using handlers.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.4.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ddio from gravatar.com ddio

Credits

Author: ddio

License

MIT License (MIT)

Requires

Python <4.0, >=3.10

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapy_tw_rental_house-2.2.1.tar.gz (45.1 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

scrapy_tw_rental_house-2.2.1-py3-none-any.whl (52.7 kB view details)

Uploaded Python 3

File details

Details for the file scrapy_tw_rental_house-2.2.1.tar.gz.

File metadata

  • Download URL: scrapy_tw_rental_house-2.2.1.tar.gz
  • Upload date:
  • Size: 45.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.7.1 CPython/3.10.12 Linux/6.8.0-138-generic

File hashes

Hashes for scrapy_tw_rental_house-2.2.1.tar.gz
Algorithm Hash digest
SHA256 18cc177b23096788d63e2e45428ee4cf3ef4ce11c209180728739e79fba8e3f7
MD5 a8b3ad32c2bd79d07b10cc484a2352cf
BLAKE2b-256 b6aade26fba38f0092bdfabc3be94c51e70ee6c0508b59ac745213144c2024b4

See more details on using hashes here.

File details

Details for the file scrapy_tw_rental_house-2.2.1-py3-none-any.whl.

File metadata

File hashes

Hashes for scrapy_tw_rental_house-2.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 d457a3f9ae784bc860d304cd2dbf5fcf7d3d5b55896a418aac1dd727a94012c0
MD5 f6c222d8a40212c74fdf47e62232e69e
BLAKE2b-256 25432e0ee69ca5a33349484e739834b687ae6c076d2faa9058c813d5cf7f70c1

See more details on using hashes here.

Release history Release notifications | RSS feed

2.4.0

2 files

2.3.0

2 files

2.2.4

2 files

2.2.3

2 files

2.2.2

2 files

This release

2.2.1 This release

2 files

2.2.0

2 files

2.1.8

2 files

2.1.7

2 files

2.1.6

2 files

2.1.5

2 files

2.1.4

2 files

2.1.3

2 files

2.1.2

2 files

2.1.1

2 files

2.1.0

2 files

2.0.4

2 files

2.0.3

2 files

2.0.1

2 files

2.0.0

2 files

1.5.1

2 files

1.5.0

2 files

1.4.1

2 files

1.4.0

2 files

1.3.7

2 files

1.3.6

2 files

1.3.5

2 files

1.3.4

2 files

1.3.3

2 files

1.3.2

2 files

1.3.1

2 files

1.3.0

2 files

1.2.1

2 files

1.2.0

2 files

1.1.2

2 files

1.1.1

2 files

1.1.0

2 files

1.0.0

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page