Skip to main content

nfl-data-pipeline

CI PyPI version Python 3.12+ License: MIT

An end-to-end data pipeline that combines PFF team grades with Pro Football Reference game and betting data, then produces analysis-ready datasets with rolling averages, rankings, and over/under features.


Table of Contents


Features

  • PFF Scraping — Selenium-based scraper for PFF team grades (requires PFF Premium; manual login on first run, cookies cached for subsequent runs)
  • PFR Scraping — Proxy-rotated scraper for Pro Football Reference boxscores
  • Data Normalization — Standardizes dates and team names across sources
  • Dataset Merging — Inner join on date + team columns
  • Rolling Averages — Pre-game cumulative stat averages per team per season
  • Games Played Tracking — Cumulative games played before each matchup
  • Feature Rankings — Per-date rankings across all teams
  • CLI + Python API — Run the full pipeline or any individual step

Installation

Install from PyPI:

pip install nfl-data-pipeline

Or install from source with Poetry:

git clone https://github.com/thadhutch/nfl-data-pipeline.git
cd nfl-data-pipeline
poetry install

Prerequisites

Requirement Why
Python 3.12+ Runtime
Google Chrome PFF scraper uses Selenium to render client-side data
PFF Premium subscription Authenticates access to PFF team grades
Rotating proxies (CSV) PFR rate-limits aggressively; proxies prevent blocks

Configuration

Create a .env file (see .env.example) or export environment variables directly:

cp .env.example .env
Variable Default Description
NFL_SEASONS 2025 Comma-separated seasons to scrape from PFF
NFL_START_YEAR 2025 First year for PFR boxscore URL collection
NFL_END_YEAR 2025 Last year for PFR boxscore URL collection
NFL_MAX_WEEK 18 Final week to scrape in the last season
NFL_DATA_DIR data Base directory for all output files
NFL_PROXY_FILE proxies/proxies.csv Path to proxy list (address:port:user:password per line)

Usage

CLI

The nfl-pipeline command is available after installation.

Note: The first time you run nfl-pipeline scrape pff, a Chrome window will open for you to log in to PFF manually. After login, cookies are saved locally and reused for future runs.

# Full end-to-end pipeline
nfl-pipeline pipeline

# Scrape from a single source
nfl-pipeline scrape pff          # PFF grades (scrape + date parsing + name normalization)
nfl-pipeline scrape pfr          # PFR game data (URLs + scrape + date/name normalization)

# Run all post-processing steps
nfl-pipeline process all

# Run individual processing steps
nfl-pipeline process merge
nfl-pipeline process over-under
nfl-pipeline process averages
nfl-pipeline process games-played
nfl-pipeline process rankings

# Check installed version
nfl-pipeline --version

Python API

Every pipeline step is importable:

import nfl_data_pipeline

# Scraping
nfl_data_pipeline.scrape_pff_data()
nfl_data_pipeline.collect_boxscore_urls()
nfl_data_pipeline.scrape_all_game_info()

# Processing
nfl_data_pipeline.merge_datasets()
nfl_data_pipeline.process_over_under()
nfl_data_pipeline.compute_rolling_averages()
nfl_data_pipeline.add_games_played()
nfl_data_pipeline.compute_rankings()

Or run an entire pipeline at once:

from nfl_data_pipeline.pipeline import (
    run_full_pipeline,
    run_pff_pipeline,
    run_pfr_pipeline,
    run_processing_pipeline,
)

run_full_pipeline()        # end-to-end
run_pff_pipeline()         # PFF scraping chain only
run_pfr_pipeline()         # PFR scraping chain only
run_processing_pipeline()  # post-processing only

Make Targets

A Makefile is included for common development workflows:

make all            # Full pipeline end-to-end
make pff            # PFF scraping + processing chain
make pfr            # PFR scraping + processing chain
make merge          # Merge PFF and PFR data (runs both chains first)
make rankings       # Full postprocessing through rankings
make test           # Run the test suite
make clean          # Remove all generated data files
make dirs           # Create data directory structure

Pipeline Architecture

PFF Scrape              PFR Scrape
    |                       |
    v                       v
Extract Dates          Normalize Dates
    |                       |
    v                       v
Normalize Names        Normalize Names
    |                       |
    +----------+   +--------+
               |   |
               v   v
              Merge
                |
                v
           Over/Under
                |
                v
         Rolling Averages
                |
                v
          Games Played
                |
                v
            Rankings

Output files are written to NFL_DATA_DIR (default: data/):

Stage Output
PFF scrape data/pff/raw_team_data.csv
PFF normalized data/pff/normalized_team_data.csv
PFR URLs data/pfr/boxscores_urls.txt
PFR normalized data/pfr/final_pfr_odds.csv
Merged data/pff_and_pfr_data.csv
Final dataset data/over-under/v1-dataset-gp-ranked.csv

Project Structure

nfl-data-pipeline/
├── src/nfl_data_pipeline/
│   ├── __init__.py           # Public API re-exports
│   ├── _config.py            # Paths, env vars, logging
│   ├── cli.py                # Click CLI entry point
│   ├── pipeline.py           # Pipeline orchestrators
│   ├── teams.py              # Team name/abbreviation mappings
│   ├── scrapers/
│   │   ├── pff.py            # PFF grades scraper (Selenium)
│   │   ├── pfr.py            # PFR game data scraper
│   │   ├── pfr_urls.py       # PFR boxscore URL collector
│   │   ├── auth.py           # PFF authentication
│   │   └── proxies.py        # Proxy loading utilities
│   ├── parsers/
│   │   ├── pff_dates.py      # PFF date extraction
│   │   ├── pff_teams.py      # PFF team name normalization
│   │   ├── pfr_dates.py      # PFR date normalization
│   │   └── pfr_teams.py      # PFR team name extraction
│   └── processing/
│       ├── merge.py          # Merge PFF + PFR datasets
│       ├── over_under.py     # O/U betting line extraction
│       ├── rolling_averages.py
│       ├── games_played.py
│       └── rankings.py
├── tests/
├── pyproject.toml
├── Makefile
├── LICENSE
└── README.md

Development

# Clone and install with dev dependencies
git clone https://github.com/thadhutch/nfl-data-pipeline.git
cd nfl-data-pipeline
poetry install

# Run the test suite
poetry run pytest -v

# Run a specific test file
poetry run pytest tests/test_rolling_averages.py -v

CI runs automatically on every push to master and on pull requests via GitHub Actions. Releases are published to PyPI through Trusted Publishers.

Contributing

Contributions are welcome! To get started:

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/my-feature)
  3. Make your changes and add tests where appropriate
  4. Run the test suite (poetry run pytest -v)
  5. Commit your changes (git commit -m "Add my feature")
  6. Push to your fork (git push origin feature/my-feature)
  7. Open a Pull Request

Please make sure all existing tests pass before submitting a PR.

Known Limitations

  • PFF scraping is DOM-dependent. The scraper relies on XPath selectors tied to PFF's frontend. If PFF changes their page structure, the selectors in scrapers/pff.py will need updating.
  • PFR scraping requires rotating proxies. Without them, requests will be rate-limited and blocked.
  • The PFF scraper is slow by design. It drives a real Chrome browser via Selenium because PFF renders data client-side.
  • PFF login requires manual interaction on first run. A Chrome window will open for you to log in. Cookies are cached afterward, so subsequent runs are fully automated.
  • Data files are not tracked in git. Run the pipeline to generate them, or bring your own data in the expected CSV format.

License

This project is licensed under the MIT License.


PyPI · Issues · CI Status

Metadata

Release files for nfl-data-pipeline 1.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for nfl-data-pipeline 1.1.1
File Size Uploaded Provenance Actions
nfl_data_pipeline-1.1.1.tar.gz 22.1 kB With provenance Details for nfl_data_pipeline-1.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for nfl-data-pipeline 1.1.1
File Interpreter ABI Platform Actions
nfl_data_pipeline-1.1.1-py3-none-any.whl Python 3 none any Details for nfl_data_pipeline-1.1.1-py3-none-any.whl

Total release size: 50.6 kB

Release files / nfl_data_pipeline-1.1.1.tar.gz

Download URL nfl_data_pipeline-1.1.1.tar.gz
Size 22.1 kB
Tags Source
SHA-256 checksum
How to use checksums
fc849b935b53e3ca6d18af52440e0e87a7d9acf51c16fe7101d73d695c4bdb89
BLAKE2b-256 checksum
How to use checksums
55df9acae00ed948ad72fbcc4ccfaaf337318d10c9696044fb7962b3432156f1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 8, 2026.

Transparency log

Release files / nfl_data_pipeline-1.1.1-py3-none-any.whl

Download URL nfl_data_pipeline-1.1.1-py3-none-any.whl
Size 28.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2d989d15cd00bdc2cd96759f60954f64311998d8aa47e52c380f8c3a94681ea2
BLAKE2b-256 checksum
How to use checksums
7daee436fa928ba5fa91018bba5560d57ef270928c992d82e5a2cecfb11ce2c0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 8, 2026.

Transparency log

Security for nfl-data-pipeline 1.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for nfl_data_pipeline-1.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Feb 8, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page