Skip to main content
Join the official Python Developers Survey 2018 and win valuable prizes: Start the survey!

YAML based lightweight crawlers

Project description

YAML based lightweight crawlers


pip install skyscraper


Each web crawler is defined in a yml file

# the name of the crawler
name: Python 3.x docs
# the number of parallel thread workers
threads: 3

# start urls

# how/where the results are saved
  type: Json
  file: "python.json"

# on each url labeled "result", results will be extracted using
# this scheme
  - name: title
      select: h1
      text: yes
      single: true

# the first page is labeled "start" and for each extracted url, we label it
# accordingly. In this example, we extract the results directly from
# the first page
- name: start
  label: start
  - type: ahrefs
    label: result
      select: a.biglink

To run the crawler, execute

skyscraper run examples/python_docs.yaml

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
skyscraper-0.0.5-py3-none-any.whl (6.1 kB) Copy SHA256 hash SHA256 Wheel py3 Jul 1, 2018
skyscraper-0.0.5.tar.gz (4.2 kB) Copy SHA256 hash SHA256 Source None Jul 1, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page