goodreads-user-scraper 2.1.0

Scrape Goodreads User Data: Profile, Book Shelves, Books, Authors

Goodreads User Scraper

Scrape Goodreads User Data: Profile, Book Shelves, Books, Authors

   

Contents

• Usage
  • Install once, then run
  • Run once without installing
• Arguments
  • --user_id
  • --output_dir
  • --skip_user_info
  • --skip_shelves
  • --skip_authors
  • --cookie
  • --cookie_file
• Authentication
  • Getting your session cookie
  • Passing the cookie
• Troubleshooting
• Development
• Publishing

Usage

Install once, then run

Best for repeat use. Installs the CLI into an isolated environment and adds the goodreads-user-scraper command to your shell.

Using pipx:

pipx install goodreads-user-scraper
goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Or with uv:

uv tool install goodreads-user-scraper
goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Run once without installing

Best for one-off use. Downloads and runs the CLI in a temporary environment: no install step, no $PATH changes.

Using pipx:

pipx run goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Or with uvx (ships with uv):

uvx goodreads-user-scraper --user_id <your id> --output_dir goodreads-data

Arguments

--user_id

• Description: The user whose data should be scraped. Find your user id using these directions.
• Required: Yes

--output_dir

• Description: The directory where all scraped data will be output.
• Required: No
• Default: goodreads-data

--skip_user_info

• Description: If passed, skip scraping user information.
• Required: No

--skip_shelves

• Description: If passed, skip scraping shelves.
• Required: No

--skip_authors

• Description: If passed, skip scraping authors.
• Required: No

--cookie

• Description: Your Goodreads session cookie (the full Cookie: request-header value). Required for shelf scraping — see Authentication.
• Required: No
• Default: None

--cookie_file

• Description: Path to a text file containing your Goodreads session cookie. Used only if --cookie is not passed and GOODREADS_COOKIE is not set.
• Required: No
• Default: None

Authentication

Shelf scraping requires authentication — Goodreads hides shelf data behind login, and the tool gathers books and authors from your shelves. So without a cookie you get a profile only; a cookie adds shelves, books, and authors, and works on your own private profile too.

Getting your session cookie

1. Sign in to Goodreads in your browser.
2. Open DevTools (Cmd/Ctrl+Shift+I) and switch to the Network tab.
3. Refresh the page, then click any goodreads.com request in the list.
4. In the request Headers, find the Cookie: header and copy its full value.

Passing the cookie

In order of precedence (first one set wins):

1. --cookie "<cookie string>"
2. GOODREADS_COOKIE environment variable
3. --cookie_file <path-to-file>

Cookies typically last several weeks. If you see a "Cookie appears invalid or expired" error, re-grab the cookie from your browser.

If no cookie is provided, shelf scraping is skipped with a warning. Pass --skip_shelves to suppress the warning.

Troubleshooting

Missing profile or shelf data?

• Your own account: pass your session cookie (see Authentication). Every request is authenticated, so your profile, shelves, and books all scrape even when your profile is private — there's no need to change any Goodreads privacy settings.
• Another user's account: you can only see what your logged-in account is allowed to, so their profile must be publicly viewable. Shelves still require your cookie.

Development

1. Clone the GitHub repository

   git clone https://github.com/YashTotale/goodreads-user-scraper.git
   

2. Run the install script

   bash scripts/install.sh
   

3. Make changes

4. Run the unit tests

   pytest
   

   These run against saved Goodreads HTML in tests/fixtures/ — no network, no cookie. This is the CI gate on every push and PR.

   When Goodreads changes its markup, refresh the fixtures with scripts/capture_fixtures.py (reads your cookie from .goodreads-cookie if present), then re-run pytest.

5. Optionally run the live smoke test

   bash scripts/test.sh
   

   This scrapes the real Goodreads site end to end. To include shelf scraping, save your Goodreads cookie to a gitignored .goodreads-cookie file in the repo root — the test script picks it up automatically. CI runs this monthly (see integration.yml) to catch Goodreads markup changes.

Publishing

Publishing is automated via GitHub Actions using PyPI Trusted Publishing (OIDC) — no API tokens are stored.

Run the publish script with the version bump type:

bash scripts/publish.sh <patch|minor|major>

The script bumps the version, creates a git tag, and pushes it. The publish workflow builds the distribution and uploads it to PyPI on any pushed v* tag.