YAML based lightweight crawlers
Project description
YAML based lightweight crawlers
Installation
pip install skyscraper
Usage
Each web crawler is defined in a yml file
# the name of the crawler name: Python 3.x docs # the number of parallel thread workers threads: 3 # start urls params: start_url: https://docs.python.org/3/index.html # how/where the results are saved results: type: Json file: "python.json" # on each url labeled "result", results will be extracted using # this scheme result_extractor: fields: - name: title rules: select: h1 text: yes single: true # the first page is labeled "start" and for each extracted url, we label it # accordingly. In this example, we extract the results directly from # the first page steps: - name: start label: start extract: - type: ahrefs label: result rules: select: a.biglink
To run the crawler, execute
skyscraper run examples/python_docs.yaml
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
skyscraper-0.0.5.tar.gz
(4.2 kB
view hashes)
Built Distribution
Close
Hashes for skyscraper-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfd51f80d32e4c7e5803c6d9be2def7b2244141798cebda000ab986a1144e33e |
|
MD5 | 986133cfc77d43187f565c4d11c6be19 |
|
BLAKE2b-256 | 87215939f42b87fb68389b772342468eb0651cba700cc8f69dfa4345a5ad6635 |