keepaneyeon 0.1.0

Monitor URLs for changes

Monitor URLs for changes

Quick Start

keepaneyeon is configured with YAML. The following configuration will download the URLs http://httpbin.org/html and http://httpbin.org/bytes/128. It will store the results in S3.

storage:
  - &aws !storage/s3
    access_key: <...>
    secret_access_key: <...>
    path: 's3://bucket/path'

downloaders:
  - &http !downloader/http {}

targets:
  - !target
    name: static
    downloader: *http
    url: 'http://httpbin.org/html'
    store: *aws

  - !target
    name: dynamic
    downloader: *http
    url: 'http://httpbin.org/bytes/128'
    store: *aws

If config.yaml contains the above configuration, you can execute the download with:

keepaneyeon config.yaml