Skip to main content

scrapy-athlinks: web scraper for race results hosted on Athlinks

License Python 3.8 PyPI

Introduction

scrapy-athlinks provides the RaceSpider class.

This spider crawls through all results pages from a race hosted on athlinks.com, building and following links to each athlete's individual results page, where it collects their split data. It also collects some metadata about the race itself.

By default, the spider returns one race metadata object (RaceItem), and one AthleteItem per participant. Each AthleteItem consists of some basic athlete info and a list of RaceSplitItem containing data from each split they recorded.

How to use this package

Option 1: In python scripts

Scrapy can be operated entirely from python scripts. See the scrapy documentation for more info.

Installation

The package is available on PyPi and can be installed with pip:

pip install scrapy-athlinks

Example usage

An demo script is included in this repo.

from scrapy.crawler import CrawlerProcess
from scrapy_athlinks import RaceSpider, AthleteItem, RaceItem


settings = {
  'FEEDS': {
    # Athlete data. Inside this file will be a list of dicts containing
    # data about each athlete's race and splits.
    'athletes.json': {
      'format':'json',
      'overwrite': True,
      'item_classes': [AthleteItem],
    },
    # Race metadata. Inside this file will be a list with a single dict
    # containing info about the race itself.
    'metadata.json': {
      'format':'json',
      'overwrite': True,
      'item_classes': [RaceItem],
    },
  }
}
process = CrawlerProcess(settings=settings)

# Crawl results for the 2022 Leadville Trail 100 Run
process.crawl(RaceSpider, 'https://www.athlinks.com/event/33913/results/Event/1018673/')
process.start()

Option 2: Command line

Alternatively, you may clone this repo for use like a typical Scrapy project that you might create on your own.

Installation

git clone https://github.com/aaron-schroeder/athlinks-scraper-scrapy
cd athlinks-scraper-scrapy
pip install -r requirements.txt

Example usage

Run a RaceSpider:

cd scrapy_athlinks
scrapy crawl race -a url=https://www.athlinks.com/event/33913/results/Event/1018673 -O out.json

Dependencies

All that is required is Scrapy (and its dependencies).

Testing

make test

License

License

This project is licensed under the MIT License. See LICENSE file for details.

Contact

You can get in touch with me at the following places:

Metadata

Release files for scrapy-athlinks 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scrapy-athlinks 0.0.1
File Size Uploaded Provenance Actions
scrapy-athlinks-0.0.1.tar.gz 10.4 kB Without provenance Details for scrapy-athlinks-0.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scrapy-athlinks 0.0.1
File Interpreter ABI Platform Actions
scrapy_athlinks-0.0.1-py3-none-any.whl Python 3 none any Details for scrapy_athlinks-0.0.1-py3-none-any.whl

Total release size: 20.1 kB

Security for scrapy-athlinks 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.0.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page