trustpilot-scraper
Turn a company's public Trustpilot reviews into clean Python dicts, CSV or JSON, and fail loudly instead of returning an empty list when Trustpilot says no.
Heads-up (October 2026): Trustpilot currently puts an AWS WAF bot-verification page in front of its review pages, and plain HTTP clients such as
requestsusually get HTTP 403. We re-checked this on 2026-10-31. When it happens the library raisesTrustpilotBlockedErrorrather than quietly returning nothing. It does not try to get around the check. Instead, save the page from your browser and parse it withparse_reviews_html(), as the quickstart shows.
Why
Review scrapers usually break in one of two quiet ways: CSS selectors rot after a redesign, or a
block page comes back and the script happily reports zero reviews. This library reads the JSON
Trustpilot embeds in every review page (__NEXT_DATA__) instead of the visual markup, and it
raises a specific exception for blocks, missing pages and malformed data. The same parser works
on HTML you fetched yourself, so a page saved from your browser is a first-class input.
Features
- Reads the page's embedded
__NEXT_DATA__JSON instead of brittle CSS selectors. - Paginates automatically and stops at the real last page (
filters.pagination.totalPages) or atmax_pages. - Returns
Date,Author,Body,Heading,RatingandLocation. Withextended=Trueit also returnsReviewId,ExperienceDate,Verified,Likes,Language,ReplyandReplyDate. - De-duplicates by review id and keeps page order.
- Polite defaults: 2 s between pages, a 30 s timeout, and an honest
User-Agentthat links back to this project. - Accepts your own
requests.Session(proxies, cookies, connection reuse, or a test adapter), and headers you set on it are not overridden. - Raises clear exceptions:
TrustpilotBlockedError(withstatus_codeandretry_after),TrustpilotParseErrorandTrustpilotError. - Accepts only real Trustpilot hosts, so a typo cannot send requests to a look-alike domain.
- A
trustpilot-scrapercommand that writes JSON, JSONL or CSV. - Typed (
py.typed, checked with mypy) and light: onlyrequestsandbeautifulsoup4.
Install
pip install trustpilot_scraper
# or
uv add trustpilot_scraper
# optional: pandas, for working with results as a DataFrame
pip install "trustpilot_scraper[pandas]"
Requires Python 3.10 to 3.13.
Quickstart
Try the live site, and fall back to a page saved from your browser (File > Save Page As) when Trustpilot blocks the request:
from pathlib import Path
from trustpilot_scraper import (
TrustpilotBlockedError,
normalize_review,
parse_reviews_html,
scrape_trustpilot_reviews,
)
try:
reviews = scrape_trustpilot_reviews("www.amazon.com", max_pages=1)
except TrustpilotBlockedError as exc:
print(f"Blocked (HTTP {exc.status_code}); using a page saved from the browser")
html = Path("page.html").read_text(encoding="utf-8")
reviews = [normalize_review(r) for r in parse_reviews_html(html)]
for review in reviews[:2]:
print(review)
Output from a run on 2026-10-31. The live request was blocked, and page.html was one of the
synthetic sample pages from
examples/data/:
Blocked (HTTP 403); using a page saved from the browser
{'Date': '2026-10-28', 'Author': 'Sample Reviewer 1', 'Body': 'Parcel arrived a day early and the tracking was spot on.', 'Heading': 'Arrived a day early', 'Rating': 5, 'Location': 'GB'}
{'Date': '2026-10-28', 'Author': 'Sample Reviewer 2', 'Body': 'Fast delivery, though I got five text messages about it.', 'Heading': 'Fast but noisy updates', 'Rating': 4, 'Location': 'NL'}
When the live request does succeed, reviews has exactly the same shape.
Command line
trustpilot-scraper www.example.com --max-pages 3 -o reviews.csv
trustpilot-scraper https://www.trustpilot.com/review/www.example.com --extended -o reviews.jsonl
trustpilot-scraper www.example.com --all-pages --delay 5 -o reviews.json
trustpilot-scraper www.example.com # first page, JSON to stdout
python -m trustpilot_scraper --version
When a request is blocked or fails, the error goes to stderr and the exit code is 1.
How it works
flowchart TD
input["Domain or review URL"] --> norm["normalize_url()<br/>validates the Trustpilot host"]
norm --> fetch["_fetch(url?page=N)<br/>requests or your Session, headers, timeout"]
fetch -- "HTTP 403 / 429" --> blocked["TrustpilotBlockedError<br/>status_code, retry_after"]
fetch -- "404 / other errors" --> err["TrustpilotError"]
fetch -- "200 HTML" --> parse["parse_next_data()<br/>BeautifulSoup: script#__NEXT_DATA__"]
saved["HTML saved from a browser"] --> parse
parse -- "verification page" --> blocked
parse -- "missing or bad JSON" --> perr["TrustpilotParseError"]
parse --> props["props.pageProps<br/>reviews + filters.pagination.totalPages"]
props --> dedupe["de-duplicate by review id"]
dedupe --> normrev["normalize_review()<br/>flat dict, optional extended fields"]
normrev --> out["iter_trustpilot_reviews() yields<br/>scrape_trustpilot_reviews() returns a list<br/>CLI writes JSON / JSONL / CSV"]
props -- "more pages: sleep(delay)" --> fetch
Key design decisions:
- Data, not markup. Trustpilot is a Next.js app, and the review list arrives as JSON inside
<script id="__NEXT_DATA__">. That JSON changes far less often than the CSS classes. - Errors are values you can act on. Blocks, missing pages and parse failures raise distinct
exceptions, so a scheduled job can back off on
TrustpilotBlockedError.retry_afterinstead of writing an empty file. - One parser for every source.
iter_trustpilot_reviewsandparse_reviews_htmlshare the same parsing and normalisation, so offline HTML and live pages produce identical dicts. - Lazy by default.
iter_trustpilot_reviewsfetches a page only when you ask for its reviews;scrape_trustpilot_reviewsis justlist(iter_...). - Bring your own transport. Every network call goes through
requests.getor theSessionyou pass, which makes the library easy to proxy, rate-limit further, or test offline.
Use cases and recipes
The offline examples use synthetic saved pages for two invented companies, so they never contact Trustpilot. The test suite runs them.
| Script | What it shows | Needs |
|---|---|---|
complaint_themes.py |
Support triage: rating distribution, reply coverage, themes in 1-2 star reviews, oldest unanswered complaint | offline |
competitor_benchmark.py |
Two companies side by side: average rating, 1-star share, verified share, reply rate, median reply time | offline |
monthly_trend_pandas.py |
pandas.DataFrame(reviews), monthly trend, per-country ratings, CSV export |
offline, [pandas] |
offline_pipeline_test.py |
Test your pipeline offline: a requests transport adapter serves saved pages to the real pagination loop, including a simulated 403 |
offline |
live_scrape.py |
Polite live scrape that respects retry_after and explains the save-from-browser fallback |
network |
python examples/competitor_benchmark.py
metric brightparcel swiftcart
reviews 24 16
avg rating 3.42 3.31
1-star share 25% 12%
verified share 79% 81%
reply rate 54% 31%
median reply (days) 2.0 3.0
More snippets: extended fields, explicit error handling, your own Session
These need network access and will usually raise TrustpilotBlockedError while Trustpilot's
verification page is active.
import time
import requests
from trustpilot_scraper import (
TrustpilotBlockedError,
TrustpilotError,
iter_trustpilot_reviews,
)
session = requests.Session()
session.headers["User-Agent"] = "acme-research/1.0 (data-team@acme.example)" # kept as-is
try:
for review in iter_trustpilot_reviews(
"www.example.com", max_pages=3, delay=5.0, extended=True, session=session
):
print(review["Date"], review["Rating"], review["Verified"], bool(review["Reply"]))
except TrustpilotBlockedError as exc:
print(f"blocked (HTTP {exc.status_code})")
if exc.retry_after:
time.sleep(exc.retry_after) # back off before any later attempt
except TrustpilotError as exc:
print("request failed:", exc)
Configuration
There are no environment variables. Everything is a function argument or CLI flag.
| Option | Default | CLI flag | Meaning |
|---|---|---|---|
max_pages |
None (all) |
--max-pages N (default 1), --all-pages |
Stop after this many pages |
delay |
2.0 |
--delay S |
Seconds to sleep between page requests |
timeout |
30.0 |
--timeout S |
Per-request timeout in seconds |
extended |
False |
--extended |
Add ReviewId, ExperienceDate, Verified, Likes, Language, Reply, ReplyDate |
session |
None |
A requests.Session to use; headers you set on it take precedence |
|
| output | stdout JSON | -o FILE |
.csv, .jsonl or .json (anything else is written as JSON) |
trustpilot_scraper.scraper.DEFAULT_HEADERS holds the default User-Agent, Accept and
Accept-Language.
API overview
| Name | Purpose |
|---|---|
scrape_trustpilot_reviews(base_url, *, max_pages=None, delay=2.0, extended=False, session=None, timeout=30.0) |
Fetch every page (or max_pages) and return a list of dicts |
iter_trustpilot_reviews(...) |
Same arguments, as a lazy generator that fetches page by page |
get_reviews_from_page(url, *, session=None, timeout=30.0) |
Fetch one page and return the raw Trustpilot review objects |
parse_reviews_html(html) |
Raw review objects from an HTML string (for example a saved page) |
parse_next_data(html) |
The whole decoded __NEXT_DATA__ JSON |
normalize_review(raw, *, extended=False) |
A raw review object as the flat dict shown above |
normalize_url(company) |
"www.example.com" becomes "https://www.trustpilot.com/review/www.example.com"; non-Trustpilot hosts raise ValueError |
TrustpilotError |
Base exception (a RuntimeError) |
TrustpilotBlockedError |
HTTP 403/429 or a verification page; has status_code and retry_after |
TrustpilotParseError |
The page has no usable review data |
Upgrading from 0.x
from trustpilot_scraper.scraper import scrape_trustpilot_reviewsstill works, and the output keys are unchanged.- Version 0.10 on PyPI shipped a wheel without the Python package in it. 1.0.0 fixed the packaging.
- Errors raise exceptions instead of returning
[]. pandasis no longer a required dependency.
When to use it
- Use it to collect public reviews for research, sentiment analysis or benchmarking at a gentle pace, or to parse review pages you saved yourself.
- Use Trustpilot's official APIs if you manage the business profile or have API access. They are the sanctioned route for production integrations.
- Do not use it to scrape at volume, to collect personal data beyond what you need, or to get around Trustpilot's access controls. The library deliberately does none of that.
Responsible use
Only scrape public pages, and keep the request rate low (the default delay is 2 s). Follow Trustpilot's terms of use and the data-protection law that applies to you. Review texts and reviewer names are personal data in many jurisdictions. If Trustpilot blocks you, stop and use a saved page or the official API rather than trying to evade the block.
Roadmap
- Parse business-level fields (
businessUnit: TrustScore, review count) into a helper. - Optional language and star-rating filters mapped to Trustpilot's query parameters.
- Retry helper that waits for
retry_afterbefore trying again, off by default.
Contributing, security, citation, license
- Pull requests are welcome. See CONTRIBUTING.md
and the Code of Conduct.
The dev loop is
pip install -e ".[dev]", thenruff check . && ruff format --check . && mypy trustpilot_scraper && pytest --cov=trustpilot_scraper. Tests run offline against saved HTML fixtures. - Report vulnerabilities as described in SECURITY.md.
- To cite this project, use CITATION.cff.
- Release notes are in CHANGELOG.md.
- MIT licensed. See LICENSE.
Metadata
Release files for trustpilot-scraper 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| trustpilot_scraper-1.1.0.tar.gz | 35.3 kB | Without provenance | Details for trustpilot_scraper-1.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| trustpilot_scraper-1.1.0-py3-none-any.whl | Python 3 | none | any | Details for trustpilot_scraper-1.1.0-py3-none-any.whl |
Total release size: 50.4 kB
Release files / trustpilot_scraper-1.1.0.tar.gz
| Download URL | trustpilot_scraper-1.1.0.tar.gz |
|---|---|
| Size | 35.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8a4f0f58fbcc0a7a1170b6452009979ca22c55286cdcfa48662513b5dc6cba7f
|
|
BLAKE2b-256 checksum How to use checksums |
d37f63e119553c412eba4d160c5113ec2f0e5344dfab1bf9301c4f7792276a73
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.7
|
Release files / trustpilot_scraper-1.1.0-py3-none-any.whl
| Download URL | trustpilot_scraper-1.1.0-py3-none-any.whl |
|---|---|
| Size | 15.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d0c10259a33ea3a6af1b0328857125e7eba3322e2c344ec2fc992c6e96a5a7d9
|
|
BLAKE2b-256 checksum How to use checksums |
5d58e9eac8c07fe620d79b3fa8b40031d2144d851dbea8e5c6a83f244711e31d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.7
|
Security for trustpilot-scraper 1.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.