nextflight

Extract JSON data from any Next.js (App Router) page in Python.

nextflight parses the React Server Components ("Flight") payloads that Next.js embeds in server-rendered HTML — the <script>self.__next_f.push([...])</script> blocks, or the raw RSC response you get back from a request sent with an RSC: 1 header — and turns them into clean, searchable Python dicts and lists. It works on any Next.js 13+ App Router site out of the box, with no per-site configuration, which makes it a natural fit for web scraping, crawling, and structured data extraction with Scrapy, requests, httpx, or the stdlib alone.

Next.js pages don't put their data in one obvious place — it's spread across dozens of numbered chunks, cross-referenced with $-sigils, and reshuffled every time the site redeploys. Hardcoding array paths like data[3]["children"][0][3]... breaks the moment that happens. nextflight resolves those references for you and lets you search for the shape of data you want instead — page.find_by_keys({"price", "title"}) instead of a brittle index chain.

Install

pip install nextflight

No required dependencies — stdlib only, so it drops into any existing Scrapy/Zyte project without touching your dependency tree. A few optional extras unlock extra features automatically if installed; see Optional dependencies.

Quick start

Two steps: see what's on the page, then fetch the shape of data you want.

from nextflight import extract

page = extract(html_text)      # a string, bytes, or response object

page.keys()                    # ['0', '1', '3f', '20', ...] -- what's here
page["3f"]                     # the resolved JSON for one specific chunk

# In practice, chunk ids are arbitrary per build (they change on
# redeploy), so search for the shape of data you want instead:
listing = page.find_by_keys({"price", "title"})       # first match
listings = page.find_all_by_keys({"price", "title"})  # every match
products = page.find_by_type("Product")               # by @type
everything = page.resolve_all()                        # everything, dereferenced

Usage

In a Scrapy spider

import scrapy
from nextflight import extract

class MySpider(scrapy.Spider):
    name = "my_spider"

    def parse(self, response):
        page = extract(response.text)
        for item in page.find_all_by_keys({"price", "title"}):
            yield {
                "title": item.get("title"),
                "price": item.get("price"),
                "url": response.url,
            }

Fetching a URL directly (no Scrapy needed)

from nextflight import FlightExtractor

page = FlightExtractor.from_url("https://example.com/product/123")
product = page.find_by_keys({"price", "title"})

from_url uses only the stdlib, for quick exploration or lightweight crawling. For anything needing retries, proxies, JS rendering, or robots.txt handling, fetch the page with your own HTTP client and pass response.text to extract(...) instead.

Raw RSC fetches (no HTML at all)

Sending a request with an RSC: 1 header — the way Next.js's own client-side navigation does — returns the raw Flight row stream directly as the response body, with no HTML wrapper. extract() detects and parses this automatically, same as the HTML-embedded form:

from nextflight import FlightExtractor

# Sets RSC:1 and Next-Url for you, and best-effort auto-discovers a
# build-specific _rsc=<id> from the page's own prefetch links
page = FlightExtractor.from_rsc_url("https://example.com/car/search?page=2")

# Or bring your own client:
import requests
resp = requests.get(
    "https://example.com/car/search",
    params={"page": "2", "_rsc": "1p28d"},   # a build-specific cache key
    headers={"RSC": "1", "Next-Url": "/en/car/search"},
)
page = FlightExtractor(resp.text)

If a site also requires a Next-Router-State-Tree header, grab it once from a real browser's network tab and reuse it — it's stable for every request to the same route regardless of query params, so it doesn't need to be regenerated per request.

Pages Router support

Older or mixed Next.js deployments use the Pages Router's __NEXT_DATA__ blob instead of Flight — already plain JSON, no $-refs to resolve:

from nextflight import extract, find_next_data, detect_next_router

router = detect_next_router(html_text)   # "app" | "pages" | "both" | "unknown"

if router == "app":
    data = extract(html_text).find_by_keys({"price", "title"})
else:
    data = find_next_data(html_text)["props"]["pageProps"]

Monitoring a page over time

diff_pages compares two crawls of the same URL and reports what changed — handy for a price or stock watcher:

from nextflight import FlightExtractor, diff_pages

old_page = FlightExtractor.from_url(url)
# ...re-fetch later...
new_page = FlightExtractor.from_url(url)

diff_pages(old_page, new_page)
# {"added": {...}, "removed": {...}, "changed": {"path.to.price": (100, 90)}}

For a list of records with a stable id, pass id_key — otherwise inserting one new item shifts every later index and makes everything after it look changed even though it didn't:

diff_pages(old_page, new_page, id_key="listing_id")
# {"changed": {"items[listing_id=7165546].price": (929900, 899900)}, ...}

Or from the command line, polling continuously (--rsc for the lighter-weight RSC payload instead of full HTML each poll):

nextflight https://example.com/product/123 --watch 60
nextflight https://example.com/car/search --rsc --watch 60

Exporting to a DataFrame or CSV

page = extract(html_text)

df = page.to_dataframe(required_keys={"id", "price"})       # requires pandas
page.to_csv("listings.csv", required_keys={"id", "price"})  # works either way

Command line

nextflight page.html --keys sections,meta
nextflight https://example.com/product/123 --type Product
nextflight page.html --tree                 # shape summary, no full values
nextflight page.html --all > everything.json

API reference

extract(html) -> FlightExtractor

Shorthand constructor. html accepts a plain string, bytes, or a response-like object (Scrapy's Response, requests.Response, etc.) — pass response straight from a parse() method.

FlightExtractor(html, *, strict=False)

strict=True raises FlightParseError on a row that's neither valid JSON nor a recognizable $-reference, instead of keeping it as a raw string. Useful while developing a new scraper; leave off in production so a handful of odd rows never take down extraction of everything else.

Exploring a page

Resolving data (dereferencing $-refs)

"3f" in page and for k in page also work, like a dict.

Searching (schema-free, works across redeploys)

Pass include_source=True on any find_* method to get (node, chunk_id) tuples instead of bare nodes, so you can trace a match back to where it came from. find_all/find_one/find_by_keys resolve chunks lazily and stop the moment max_results is hit — they don't pay to resolve chunks after a match is already found.

Fetching

from_url/from_rsc_url transparently decompress gzip/deflate/br responses even if the server ignores the default Accept-Encoding: identity request.

Diffing and exporting

Module-level functions

• find_json_ld(html, type_=None) -> list — parse <script type="application/ld+json"> blocks, optionally filtered by @type. Often more stable across redesigns than Flight data — worth trying first for product/article/breadcrumb structured data.
• find_next_data(html) -> dict | None — parse a Pages Router __NEXT_DATA__ blob. None if the page doesn't have one.
• detect_next_router(html) -> str — "app", "pages", "both", or "unknown". Run this first if you're not sure which extractor to use.
• diff_pages(old, new, id_key=None) -> dict — module-level form of .diff().

Command-line reference

nextflight <file-or-url>
  [--keys a,b | --all-by-keys a,b | --any-keys a,b
   | --type Product | --text PATTERN | --get path.to.value
   | --json-keys | --html-keys | --tree
   | --router | --next-data | --stats | --watch SECONDS | --all]
  [--rsc] [--redact] [--save out.json]

• --tree prints a .shape() summary instead of full values.
• --router / --next-data cover Pages Router pages.
• --rsc fetches the raw RSC payload instead of full HTML (URL sources only) — lighter weight, also works with --watch.
• --watch SECONDS polls a URL and prints only what changed since the last poll.
• --redact best-effort scrubs email/phone-shaped strings from output, for sharing debug dumps.

Optional dependencies

Nothing below is required to install or use nextflight — each is used automatically if already present in your environment, and raises a clear ImportError only if you call the one method that needs it.

Or pip install nextflight[all] for all three.

How it works

Flight payloads aren't newline-delimited JSON — text rows (id:T<hexByteLen>,<raw bytes>) are byte-length-prefixed and can contain literal newlines or run straight into the next row with no separator, and module/preload rows (id:I[...], id:HL[...]) need bracket-aware parsing. nextflight implements the actual row grammar rather than splitting on \n, so it holds up on both well-formed pages and payloads truncated mid-chunk (e.g. by a proxy that cuts a response short).

It also doesn't assume one <script>self.__next_f.push(...)</script> call is one complete, self-contained set of rows. On large real-world pages, Next.js's own streaming buffer can flush mid-string, splitting a single row's raw text across two or more separate push() calls with no separator between the pieces — all push() payloads are reassembled into one continuous stream before being split into rows, so this doesn't silently corrupt chunk ids on pages large enough to trigger it (confirmed against production pages where over half of all push() calls turned out to be mid-row continuations).

Chunk ids aren't guaranteed unique, either — Next.js deliberately emits preload (HL) rows with a completely empty id (:HL["/path.css","style"]) since nothing ever needs to $-ref them individually, and real pages have had dozens of these sharing the same empty id. Rather than the later ones silently overwriting the earlier ones, only the first occurrence of a duplicated id keeps its real id; later ones get a synthesized, clearly distinguishable key ("id#2", "id#3", ...) so nothing gets lost.

Parsing and resolution are both designed to scale roughly linearly with page size: rows are split cheaply up front, each chunk's JSON is decoded lazily on first access rather than all at once, and searches (find_one/find_by_keys) stop resolving chunks the moment a match is found instead of resolving the whole page first.

License

MIT