pydj - Dynamic JS File Extractor (Python)

中文 | English

pydj statically analyses a website's HTML and JavaScript to find the JS files that are loaded dynamically — webpack chunks, import() lazy loading, Vite preloads, Module Federation remotes, Next.js Flight chunks, microfrontend sub-applications — then locates the source maps and restores the original source code.

A Python port of the Go project ejfkdev/dj, keeping the same CLI surface and output formats.

Scope: CLI and library only. There is no HTTP API and no MCP server.

Why this is useful

A plain curl of a page gets you the entry bundle and nothing else. The other 300 chunk files exist only as strings assembled at runtime by the bundler's chunk loader. pydj recognises those loaders and reconstructs the URLs — which is what makes the whole application graph, and its source maps, reachable.

$ pydj https://example.com
## Summary
- **JS files**: 74
- **Source maps**: 12 (found)
- **Restored sources**: 1204 files (restored)

Installation

pipx install pydj              # isolated CLI install (recommended)
uvx pydj --help                # run without installing
pip install pydj               # library use

This pulls in curl-cffi for browser TLS impersonation, so there is no extra to remember. Prebuilt abi3 wheels cover macOS, Linux (glibc and musl) and Windows, so no compiler is normally needed. If it cannot be installed, pydj still runs with a plain fingerprint — see TLS fingerprints.

git clone https://github.com/ejfkdev/pydj && cd pydj
pip install -e '.[dev]'

CLI usage

pydj <url>                    # scan a website (same as: pydj scan <url>)
pydj scan <url>               # canonical subcommand form
pydj <url> -f json            # structured output
pydj <url> -f text            # bare URL list

Options

Exit codes are 0 for success and 1 for usage or runtime errors.

Examples

# Basic scan
pydj https://example.com

# JSON output, piped into jq
pydj -f json https://example.com | jq -r '.jsURLs[]'

# Fresh scan (ignore the cache), export artefacts, use a SOCKS5 proxy
pydj --no-cache -o ./output -x socks5://127.0.0.1:1080 -t 60 https://example.com

# Bypass a Cloudflare challenge
pydj --cookie "cf_clearance=xxx" --useragent "Mozilla/5.0 ..." https://example.com

# Custom headers (later value wins for duplicate keys)
pydj -H 'Referer: https://google.com' -H 'X-Token: abc' https://example.com

# Keep JS bodies off disk; export only the URL list and restored sources
pydj --no-js-cache -f json https://example.com

# Proxy from the environment, with a bypass list
HTTPS_PROXY=http://127.0.0.1:7890 NO_PROXY=localhost pydj https://example.com

Library usage

scan is a facade over a set of composable pieces. Use it for the whole job, or drop to the pieces when you want to control part of it.

The one-call version

from pydj import scan

result = scan("https://example.com")

print(result.summary.js_count)      # 74
print(result.summary.source_count)  # 1204 restored source files
print(result.cache_dirs.sources)    # /tmp/ejfkdev/dj/https_example.com/sources

for url in result.js_urls:          # ScanResult iterates over JS URLs
    print(url)

# Structured data
data = result.to_dict()
json_text = result.to_json()

# Rendered report, matching the CLI byte for byte
print(scan("https://example.com", format="md").text)

ScanResult carries both the structured OutputResult (result.result) and the rendered report (result.text).

scan() parameters

Bringing your own HTTP layer

pydj never touches the network directly. Every request — the landing page, JS files, source maps, .map probes, Next.js RSC payloads — goes through a transport: any object with two methods.

class HttpTransport(Protocol):
    def get(self, url, *, headers=None) -> HttpResponse: ...
    def head(self, url, *, headers=None) -> HttpResponse: ...

That is the entire contract, and it is what makes TLS handling your choice rather than pydj's. Implement it to plug in a different TLS library, an existing authenticated session, a caching proxy, or a request budget you own:

import curl_cffi
from pydj import scan


class ImpersonatingTransport:
    """curl-cffi with a fingerprint pydj does not ship."""

    def __init__(self, impersonate: str = "chrome124") -> None:
        self._session = curl_cffi.requests.Session(impersonate=impersonate)

    def get(self, url, *, headers=None):
        return self._session.get(url, headers=dict(headers or {}), timeout=30)

    def head(self, url, *, headers=None):
        return self._session.head(url, headers=dict(headers or {}), timeout=30)


result = scan("https://example.com", transport=ImpersonatingTransport("safari17_0"))
print(result.js_urls)

Your return value does not have to be an HttpResponse. pydj coerces whatever comes back, so requests.Response, httpx.Response and curl-cffi responses work as-is — anything with status_code, content, headers and url:

import requests
from pydj import scan


class SessionTransport:
    def __init__(self):
        self.session = requests.Session()
        self.session.headers["User-Agent"] = "MyBot/1.0"

    def get(self, url, *, headers=None):
        return self.session.get(url, headers=headers or {}, timeout=30)

    def head(self, url, *, headers=None):
        return self.session.head(url, headers=headers or {}, timeout=30)


result = scan("https://example.com", transport=SessionTransport())

The fine-grained API

Everything scan does is also available as a standalone function. None of them hold state, read configuration or write to disk, so you can assemble your own pipeline:

from pydj import fetch_content, analyze_content, fetch_source_map, restore_sources

# 1. Fetch a page.
page = fetch_content(transport, "https://example.com/")
if page is None:
    raise SystemExit("fetch failed")

# 2. Analyse it. No network, no disk — just plugins over bytes.
for result in analyze_content("https://example.com/", page.content, "html",
                              headers=page.headers):
    for found in result.urls:
        print(result.from_plugin, "found", found.url)

# 3. Fetch a JS file and find what it loads.
app = fetch_content(transport, "https://example.com/js/app.js")
for result in analyze_content("https://example.com/js/app.js", app.content, "js"):
    print(result.urls)

# 4. Recover original sources from its map.
map_bytes = fetch_source_map(transport, "https://example.com/js/app.js")
for source in restore_sources(map_bytes, minified_content=app.content):
    print(source.path, source.mode)

discover_js_urls is the interesting one: the whole crawl with none of the side effects, so summary.source_map_count and summary.source_count stay zero and nothing is written anywhere.

JS on disk

JS bodies dominate the disk usage of a scan and are usually the least useful artefact afterwards — you wanted the URL list and the sources. Three ways to control that:

# Default: bodies land in the cache's js/ directory.
scan("https://example.com")

# Keep them off disk entirely. The URL list, source maps and restored
# sources are unaffected.
scan("https://example.com", cache_js=False)

# Or send them somewhere specific, flat and without a per-origin level,
# so several sites can share one folder.
scan("https://example.com", js_cache_dir="./js-bodies")

With cache_js=False, meta.json records no local_path for JS entries and cache_dirs.js is empty — the metadata never claims a file that is not there.

Runnable examples

examples/ holds six scripts covering the whole library surface, each runnable offline against a bundled fixture site:

cd examples
python 01_basic_scan.py
python 04_custom_transport.py

# or without installing pydj at all
PYTHONPATH=../src python 05_custom_plugin.py

To develop against your local checkout so edits take effect immediately:

pip install -e .          # then `python examples/01_basic_scan.py`
pipx install --editable . # same, but the CLI is on PATH too

See examples/README.md for the fixture's contents and the wheel / uvx variants.

Custom plugins

from pydj import PluginRegistry, scan
from pydj.types import AnalyzeInput, ContentType, Result, DiscoveredJS


class MyBundlerPlugin:
    def name(self) -> str:
        return "MyBundlerPlugin"

    def precheck(self, input: AnalyzeInput) -> bool:
        return input.content_type is ContentType.JS and b"myBundler" in input.content

    def analyze(self, input: AnalyzeInput) -> Result:
        result = Result()
        result.urls.append(
            DiscoveredJS(url="https://example.com/my-chunk.js", from_url=input.source_url)
        )
        return result


registry = PluginRegistry()
registry.register(MyBundlerPlugin())

result = scan("https://example.com", registry=registry)

A plugin implements three methods:

• name() — identifier reported as from_plugin in provenance output.
• precheck(input) — cheap gate; only if it returns True is analyze called.
• analyze(input) — returns a Result with urls (finished URLs), probe_targets (path fragments needing resolution), intermediates (config files to download), inline_scripts, rsc_probes, public_paths and prepend_urls.

Use pydj.plugins.build_registry() to start from the built-ins and add yours.

Supported patterns

26 built-in plugins.

How it works

1. Download the landing page HTML.
2. If meta.json exists for that origin, restore the previous run's JS list and sources from cache and stop — no network requests at all.
3. Otherwise dispatch every plugin against the payload. Each result is one of: a finished URL (queued), a path fragment (probed later), an intermediate config file (downloaded and re-analysed), or a follow-up request.
4. Probe path fragments: webpack chunk paths have no origin, so candidates are assembled from known-good JS directories, the source file's directory, known CDN prefixes and every known domain, then de-duplicated.
5. For every JS file, HEAD-probe the sibling .map and download it if real.
6. Restore sources: prefer sourcesContent from the map; fall back to re-assembling fragments via mappings (VLQ) when it is absent.
7. Write meta.json recording every JS URL, its provenance, its source map and the restored sources.

Cache reuse

Caching is on by default and lives under the system temp directory:

<temp>/ejfkdev/dj/<normalized-origin>/
├── js/           downloaded JS
├── source_map/   .map files
├── sources/      restored original sources (directory tree preserved)
├── html/         landing page (web.html)
└── meta.json     site metadata

The second run on a site reuses all of it. --no-cache forces a network scan while still writing what it downloads.

With -o/--output every artefact is also written to that directory without the origin level, giving a flat export:

<output_dir>/
├── js/
├── source_map/
├── sources/
├── html/
└── meta.json

Output formats

text (bare URL list)

https://example.com/js/main.js
https://example.com/js/chunk-abc123.js

--- Summary ---
JS files: 2
Source maps: 1 (found)
Restored sources: 15 files (restored)

md (default)

A summary, the JS URL list, per-JS provenance, cache directories and any HTML entries discovered through multi-page pivoting.

json

{
  "summary": { "jsCount": 3, "sourceMapCount": 1, "sourceCount": 15 },
  "jsURLs": ["https://example.com/js/main.js"],
  "jsDetails": [
    {
      "url": "https://example.com/js/main.js",
      "from_url": "https://example.com/",
      "from_plugin": "HTMLScriptPlugin"
    }
  ],
  "cacheBase": "/tmp/ejfkdev/dj/example.com",
  "cacheDirs": {
    "js": "/tmp/ejfkdev/dj/example.com/js",
    "sourceMap": "/tmp/ejfkdev/dj/example.com/source_map",
    "source": "/tmp/ejfkdev/dj/example.com/sources",
    "html": "/tmp/ejfkdev/dj/example.com/html/web.html"
  }
}

Verified against real sites

Numbers below come from scans of live sites, one per supported framework family.

Two things worth reading off that table:

• A URL count is not a file count. chat.z.ai's 359 URLs are 183 distinct files, 176 of which are served from the site's own origin and its CDN. Dedupe by path when you want the file inventory and not every route to it.
• Cross-origin bundles are followed. A chunk resolved from a CDN host keeps its own absolute URL, so the output records where a file actually lives instead of assuming the scanned origin.

Totals drift between runs on sites that generate chunks on demand -- VitePress builds *.md.<hash>.lean.js per request, so vuejs.org lands anywhere between 56 and 63. Treat these as approximate.

Three real defects were found this way and fixed:

• Source maps for cross-origin bundles were silently dropped. A script hosted on a CDN has no site-relative path, and the map writer skipped it -- so the sources were restored (1012 files on react.dev) while the summary reported zero maps. Now stored under a host-qualified name.
• The cache root was computed two different ways. The summary rebuilt it from base_dir + origin, which differs from the cache's own root when the scanned URL has no trailing slash (https_x vs https_x_), so the summary looked in a directory nothing had written to.
• A global request lock serialised every download. It was added on the assumption that curl-cffi's session is not thread-safe; measuring showed 12 threads sharing one session give a 4.7x speedup with no errors. Removing it took a 390-request scan from 185s to 35s.

TLS fingerprints

Modern bot detection fingerprints the TLS ClientHello, not just the headers, so a plain Python client is refused by Cloudflare-era sites while a browser is let through. pydj presents a real browser fingerprint using curl-cffi (curl-impersonate) — installed by default, picking a random browser per scan and honouring --no-random-tls to pin Chrome.

The dependency is declared rather than optional because the fingerprint is the tool's entire value proposition: without it, a Cloudflare-protected site yields an empty result while still "succeeding". It is still non-fatal when it cannot be installed, because some platforms have no wheel and a toolchain may not be available — refusing to run would be worse than running with a weaker fingerprint.

If you need a fingerprint pydj does not offer, or your own session, pass a custom transport.

Development

git clone https://github.com/ejfkdev/pydj
cd pydj
pip install -e '.[dev]'
python -m pytest

Python 3.10 is the supported floor. See CONTRIBUTING.md for what a change needs to pass, and CHANGELOG.md for what has changed. Security reports go through GitHub's private advisory flow, described in SECURITY.md -- please do not open a public issue for those.

Continuous integration

.github/workflows/ci.yml runs on every push and pull request:

Python 3.10 is the declared floor and is tested as such; the newer interpreters are there because a decompression bug once appeared only on 3.14.

ruff check src tests     # what CI runs
ruff check src tests --fix

Releasing

Publishing is tag-driven — pushing a tag is the only action required:

# 1. Bump the version in pyproject.toml and src/pydj/__init__.py
# 2. Commit, then tag and push
git tag -a v0.2.0 -m "v0.2.0"
git push origin v0.2.0

.github/workflows/release.yml then:

1. verify — runs the suite and fails if the tag, pyproject.toml and __init__.py versions disagree. PyPI does not allow re-uploading a version, so a mismatch has to be caught before publishing, not after.
2. build — python -m build plus twine check.
3. pypi — publishes with trusted publishing (OIDC), so there is no long-lived token in the repository. Falls back to a PYPI_API_TOKEN secret if one is set.
4. github-release — creates the GitHub Release from the annotated tag, with an install snippet in the run summary.

One-time setup for trusted publishing, at https://pypi.org/manage/account/publishing/:

The environment name must match the environment: pypi in the workflow; create it under the repository's Settings → Environments (adding required reviewers there is worth doing, and gives you a manual gate on publication).

License

MPL-2.0, matching the original project.