nextflight

A general-purpose parser for the data Next.js (App Router) embeds in <script>self.__next_f.push([...])</script> tags — the React Server Components "Flight" wire format. Works on any Next.js 13+ App Router site, not just one particular project.

Instead of hardcoding array indices like data[3]["children"][0][3]..., which break the moment a site's component tree reshuffles on redeploy, nextflight resolves the $-sigil references Next.js uses internally and lets you search for the shape of data you want.

Install

pip install nextflight

Quick start

The core workflow is two steps: hand it any HTML, see what keys are on the page, then fetch the resolved JSON for whichever key you want.

from nextflight import extract

# Step 1: send any HTML, get the list of keys (one per __next_f.push chunk)
page = extract(html_text)
print(page.keys())          # e.g. ['0', '1', '3f', '20', ...]

# Step 2: fetch the resolved JSON for a specific key
data = page["3f"]           # same as page.resolve_chunk("3f")

Chunk ids are arbitrary per build though (a redeploy can renumber them), so in practice you'll usually skip straight to searching for the shape of data you want instead of a specific id:

from nextflight import extract

page = extract(html_text)

# Find the first object anywhere in the page that has all of these keys,
# wherever this build's component tree happened to put it:
listing = page.find_by_keys({"sections", "meta"})

# Find every node with a given @type (or any custom key):
products = page.find_by_type("Product")

# Or search with a fully custom predicate:
items = page.find_all(lambda n: isinstance(n, dict) and "price" in n)

# Or grab everything, fully dereferenced, and inspect by hand:
everything = page.resolve_all()

Command line

For quick, no-script exploration of a page you've already saved (or a live URL):

nextflight page.html --keys sections,meta
nextflight https://example.com/product/123 --type Product
nextflight page.html --all > everything.json

In a Scrapy / Zyte spider

import scrapy
from nextflight import extract

class MySpider(scrapy.Spider):
    name = "my_spider"

    def parse(self, response):
        page = extract(response.text)

        items = page.find_all(
            lambda n: isinstance(n, dict) and "price" in n and "title" in n
        )
        for item in items:
            yield {
                "title": item.get("title"),
                "price": item.get("price"),
                "url": response.url,
            }

Fetching a URL directly (no Scrapy needed)

from nextflight import FlightExtractor

page = FlightExtractor.from_url("https://example.com/product/123")
product = page.find_by_keys({"price", "title"})

(from_url uses only the stdlib for quick one-off exploration. For production crawling — retries, proxies, JS rendering, robots.txt — fetch the page with your own HTTP client / Scrapy / Zyte and pass response.text to FlightExtractor(...) / extract(...) instead.)

Raw RSC fetches (no HTML at all)

Sending a request with an RSC: 1 header — the way Next.js's own client-side navigation does it — gets back the raw Flight row stream directly as the response body, with no surrounding HTML and no self.__next_f.push(...) wrapper. extract() detects and parses this automatically, exactly the same as the HTML-embedded form:

from nextflight import FlightExtractor

# Convenience constructor: sets the RSC header for you
page = FlightExtractor.from_rsc_url("https://example.com/car/search?page=2")

# Or bring your own client (requests, httpx, Scrapy, ...):
import requests
resp = requests.get(
    "https://example.com/car/search",
    params={"page": "2", "_rsc": "1p28d"},  # build-specific cache key from the page's own JS
    headers={"RSC": "1", "Next-Url": "/en/car/search"},
)
page = FlightExtractor(resp.text)

Some deployments require extra headers to serve the RSC payload instead of redirecting to the full HTML page or rejecting the request — a matching Next-Router-State-Tree header, a next-url header pointing at the page itself, or a build-specific _rsc=<id> query parameter (grab it from the page's own client-side JS/network tab; it changes across deploys). Copy whatever a real browser sends for that specific site if the bare RSC: 1 header alone doesn't work.

Pages Router support

Not every Next.js site uses the App Router. Older / mixed deployments often use the Pages Router's __NEXT_DATA__ blob instead, which is already plain JSON:

from nextflight import extract, find_next_data, detect_next_router

router = detect_next_router(html_text)  # "app" | "pages" | "both" | "unknown"

if router == "app":
    page = extract(html_text)
    data = page.find_by_keys({"price", "title"})
else:
    data = find_next_data(html_text)["props"]["pageProps"]

Monitoring a page over time

diff_pages compares two crawls of the same URL and reports what changed, by dotted path — handy for a price/stock watcher:

from nextflight import FlightExtractor, diff_pages

old_page = FlightExtractor.from_url(url)
# ...re-fetch later...
new_page = FlightExtractor.from_url(url)

changes = diff_pages(old_page, new_page)
# {"added": {...}, "removed": {...}, "changed": {"path.to.price": (100, 90)}}

Or from the command line, polling continuously:

nextflight https://example.com/product/123 --watch 60

Exporting to a DataFrame / CSV

page = extract(html_text)

df = page.to_dataframe(required_keys={"id", "price"})   # requires pandas
page.to_csv("listings.csv", required_keys={"id", "price"})  # works either way

API

• extract(html) -> FlightExtractor — shorthand constructor. html accepts a plain string, bytes, or a response-like object (Scrapy's Response, requests.Response, etc.) — pass response straight from a parse() method without writing response.text yourself.
• FlightExtractor(html, *, strict: bool = False)
  • .keys() -> list[str] — every chunk id found on the page, in order.
  • .json_keys() -> list[str] — chunk ids whose raw value is structured JSON (a dict or list), i.e. the ones you almost always want.
  • .html_keys() -> list[str] — chunk ids from Flight text (T) rows that look like an HTML fragment (contain a tag) — suspense fallbacks, error boundaries, inlined SVGs, and other raw markup Next.js streams outside the JSON chunks.
  • .text_keys() -> list[str] — every chunk id from a text (T) row, HTML-looking or not (plain copy, translated strings, etc).
  • .kind(chunk_id) -> str | None — which Flight row kind a chunk came from ("json", "text", "module", "preload"), or None if the id doesn't exist. This is what json_keys() / html_keys() / text_keys() filter on.
  • .find_any_keys(any_keys, root=None, include_source=False) -> list — like find_all_by_keys but matches a dict containing ANY of any_keys rather than requiring all of them.
  • .find_by_key_pattern(pattern, root=None, include_source=False) -> list — find every dict with at least one key matching a regex pattern (e.g. r"^price_" to catch price_usd, price_aed, ...).
  • include_source=True on any find_* method returns (node, chunk_id) tuples instead of bare nodes, so you can trace a match back to roughly where it came from (or re-fetch just that chunk on a future crawl).
  • .iter_resolved() -> Iterator[(chunk_id, value)] — like resolve_all() but lazy, one chunk at a time, for bailing out early on very large pages.
  • .shape(chunk_id=None, max_depth=3) -> Any — a compact summary of the resolved data's structure (key names + value types, lists collapsed to their first element) instead of full values — for getting a feel for an unfamiliar site fast. Omit chunk_id to summarize every chunk.
  • .diff(other_page) -> dict / diff_pages(old, new) -> dict — compare two crawls of the same URL and report {"added", "removed", "changed"} by dotted path — handy for price/stock-monitoring pipelines. See the caveat about chunk ids reshuffling across redeploys in the docstring.
  • .to_dataframe(records=None, required_keys=None) / .to_csv(path, ...) — build a pandas DataFrame or write a CSV from a list of dict records (or run find_all_by_keys(required_keys) for you first). to_csv falls back to the stdlib csv module if pandas isn't installed.
  • .from_url_async(url, ...) (async classmethod) — async counterpart to from_url, for concurrent multi-page crawls with asyncio.gather(...). Requires httpx (optional).
  • .from_rsc_url(url, headers=None, cookies=None, ...) — fetch a raw RSC payload directly (sets the RSC: 1 header for you) instead of the full HTML page. See "Raw RSC fetches" above.
  • page["3f"] / .resolve_chunk("3f") — the resolved JSON for one specific chunk id (page[...] raises KeyError if it doesn't exist; resolve_chunk returns None). "3f" in page and for k in page also work, like a dict.
  • .resolve_all() -> dict — every chunk, fully dereferenced.
  • .resolve_json() -> dict — only the chunks whose raw value is structured JSON (see json_keys()). Skips top-level text/HTML-only chunks that aren't referenced from any JSON chunk, so it's cheaper than resolve_all() on pages with a lot of raw markup/text rows.
  • .resolve_html() -> dict / .resolve_text() -> dict — the mirror image: resolve only html_keys() / text_keys().
  • .find_all(predicate, root=None, max_results=None) -> list — walk the resolved tree and collect every node matching predicate. With no root, chunks are resolved lazily one at a time (via iter_resolved()), so max_results stops resolving further chunks the moment enough matches are found, not just stops searching.
  • .find_one(predicate, root=None) -> Any | None
  • .find_by_keys(required_keys, root=None) -> dict | None — find the first dict containing all of required_keys.
  • .find_all_by_keys(required_keys, root=None) -> list — like find_by_keys but returns every match, for pages with repeated cards/listings that share the same shape.
  • .find_by_type(type_value, key="@type", root=None) -> list — find every dict whose key field equals type_value.
  • .find_text(pattern, root=None) -> list — regex-search every string value on the page and return the distinct whole values that contain a match (emails, prices, phone numbers, SKUs, ...) without needing to know which object they live on.
  • .get("path.to.value", default=None) -> Any — tolerant dotted-path lookup into the resolved page (dict keys and/or list indices), once you already know roughly where something lives on this site.
  • .stats() -> dict — quick diagnostic snapshot (chunk count, ids, value type counts, Flight row kind counts, json/html chunk counts, page size) for exploring a new site.
  • .to_json(path=None, indent=2) -> str | None — dump the fully resolved page to a file, or return it as a JSON string.
  • .from_url(url, timeout=15.0, headers=None) -> FlightExtractor (classmethod) — fetch and parse a URL using only the stdlib.
  • strict=True raises FlightParseError on a row that's neither valid JSON nor a recognizable $-reference marker, instead of silently keeping it as a raw string (useful while developing a new scraper; leave off in production so a handful of odd rows never take down extraction of everything else on the page).
• find_json_ld(html, type_=None) -> list — parse any <script type="application/ld+json"> blocks on the page, optionally filtered by @type. Also accepts response-like objects.
• find_next_data(html) -> dict | None — parse the Pages Router's __NEXT_DATA__ JSON blob (Next.js's pre-App-Router data mechanism). Already plain JSON, no $-ref resolution needed, so there's no extractor class for it, just this function. Returns None if the page doesn't have one (e.g. it's an App Router page — use extract() there instead).
• detect_next_router(html) -> str — best-effort guess at which router rendered a page: "app", "pages", "both" (rare, e.g. mid-migration sites), or "unknown". Run this first if you're not sure which of extract() / find_next_data() to reach for.
• diff_pages(old, new) -> dict — module-level version of .diff(), same thing.
• CLI: nextflight <file-or-url> [--keys a,b | --all-by-keys a,b | --any-keys a,b | --type Product | --text PATTERN | --get path.to.value | --json-keys | --html-keys | --tree | --router | --next-data | --stats | --watch SECONDS | --all] [--redact] [--save out.json]
  • --tree prints .shape() instead of full values.
  • --router / --next-data cover Pages Router pages (see above).
  • --watch SECONDS polls a URL and prints only what changed since the last poll (via diff_pages) — a quick way to eyeball whether a site's data-monitoring pipeline is worth building before you build it.
  • --redact best-effort scrubs email/phone-shaped strings from output, for sharing debug dumps.

No required runtime dependencies — stdlib only (json, re, urllib, argparse) — so it's safe to drop into any existing Scrapy/Zyte project without touching the rest of your dependency tree. Optional accelerators/ integrations, used automatically if already installed, otherwise skipped or raising a clear ImportError only if you call the specific method that needs them:

(or pip install nextflight[all] for all three.)

Upgrading from nextjs-flight-extractor / NextFlightExtractor

The old names still work but emit a DeprecationWarning:

Performance

Parsing is designed to be roughly linear in the size of the page, even on pages with hundreds or thousands of Flight rows:

• Text (T) rows no longer re-encode the rest of the payload on every row. Earlier versions read a text row's body with payload[body_start:].encode("utf-8")[:hex_len], which re-encodes the entire remainder of the payload for every single text row. A page with many text rows (translated copy, repeated card fragments, inlined SVGs) made parsing effectively O(n²). It now reads only the bytes each row actually needs, so parsing is O(n) again — roughly a 10x–20x speedup on pages with thousands of text rows, and the gap grows with page size.
• No more O(n) string copies for bare/ref rows. Locating the end of an unbracketed value (a bare $-ref marker, number, etc.) used to slice payload[i:] — copying the rest of the payload — before searching it. It now searches in place with a pos argument instead.
• Optional orjson acceleration. If orjson is already installed in your environment (common in scraping stacks), nextflight will use it automatically for JSON decoding — no configuration needed, and it's not a required dependency. Falls back to the stdlib json module otherwise.
• find_one/find_by_keys stop resolving, not just searching, at the first match. These used to build resolve_all() — resolving every chunk on the page — before the search even started, so max_results=1 only cut short the walk, not the resolution work leading up to it. They now resolve chunks lazily (via iter_resolved()) and stop the moment a match is found. On a page with thousands of unrelated chunks and the match near the front, this measured 300x+ faster in testing (0.056s → 0.0002s on a 3,000-chunk synthetic page) — the gap scales with how many chunks come after the match and how expensive they are to resolve.
• resolve_json() / resolve_html() / resolve_text() for when you only care about one kind of chunk and don't want to pay to resolve (and allocate copies of) everything else on the page via resolve_all().

Why not just str.split('\n')?

Two of the Flight row kinds break that assumption:

• Text rows (id:T<hexByteLen>,<raw text>) are byte-length-prefixed blobs, not newline-terminated, and can contain literal newlines or run directly into the next row's id with zero separator.
• Module / preload rows (id:I[...] / :HL[...]) need bracket-aware parsing.

nextflight implements the real row grammar, quote/escape aware, so it holds up on both well-formed and truncated payloads (e.g. from a proxy that cuts a response off mid-chunk).

License

MIT