httpcrabber 1.4.0

Network-level traffic interceptor for reverse-engineering and debugging web APIs

Captures on the wire and saves everything to disk — no extensions, no injected code, the page runs untouched.

English · Русский · Español · 中文

Features · Install · Quick start · Command line · Session output · Working with a session · How it works · Security

⚡ 30-second start

pipx install httpcrabber          # or: pip install httpcrabber inside a venv
httpcrabber

Answer three prompts, browse as usual, close Chrome — every request, response, WebSocket frame and script of the session is in LOGS/<name>/.

Why

DevTools is great for a quick look, but it is a poor recorder: the log lives only as long as the tab, large bodies and WebSocket frames are awkward to export, scripts are scattered across requests, and a HAR file is one huge snapshot of one tab.

httpcrabber records a whole session to disk. It is an HTTPS proxy based on mitmproxy: every request, response, WebSocket frame and script is captured on the wire and written to a per-session folder as line-by-line JSONL — ready for grep, jq, diffs and scripts. Nothing is installed into the browser or injected into the page, so the site runs exactly as it does on a normal visit, and any browser or device that can use a proxy works.

Features

Requirements

• Python 3.11+
• Google Chrome or Chromium (optional with --no-browser)

Set HTTPCRABBER_BROWSER=/path/to/chrome to override discovery on any OS.

Install

pipx install httpcrabber
httpcrabber --version

Why pipx: mitmproxy pins exact versions of its dependencies (cryptography, h2, …), so installing into your global Python can downgrade packages that other tools rely on. pipx keeps httpcrabber in its own environment; pip install httpcrabber inside a venv works just as well.

Or from a clone, for development:

git clone https://github.com/web3daemon/httpcrabber-client.git
cd httpcrabber-client
python -m venv .venv && source .venv/bin/activate     # .venv\Scripts\Activate.ps1 on Windows
pip install -e ".[dev]"

No install at all? python run.py works straight from the clone.

Quick start

httpcrabber

The tool asks for three things and handles the rest:

1. Language — Russian or English
2. Upstream proxy — paste it in any format, or press Enter to go direct
3. Session name — e.g. TARGET RECON

It shows a session brief, installs the CA certificate if needed, starts Chrome through the proxy, and streams everything into the session folder. Browse normally — every request, response, WebSocket frame and script is captured. Close Chrome (or press Ctrl+C) to finish; you get an analytics panel and the folder path.

Command line

Every prompt has a flag. Pass them all and httpcrabber asks nothing — handy for scripts.

httpcrabber --lang en --session "target recon" --proxy socks5://user:pass@1.2.3.4:1080
httpcrabber -l en -s quick --direct --no-browser          # use your own browser / device
httpcrabber --no-anim                                      # plain output, no animations
httpcrabber -s api --include '*.target.com' --sourcemaps  # only the target + original sources

Proxy formats

Every common notation is accepted. Press Enter with an empty input for a direct connection.

host:port
host:port:user:pass
user:pass@host:port
socks5://user:pass@host:port
http://host:port
https://user:pass@host:port

Schemes: http, https, socks5, socks5h, socks4. Passwords are masked in the interface.

Session output

Each session is a self-contained folder:

LOGS/
└── target_recon/
    ├── target_recon.jsonl                     # network dump
    └── js/
        ├── index.json                         # manifest: url, file, sha256, size, hits
        ├── cdn.target.com/
        │   ├── main.a3f1c8d4.js               # external scripts
        │   ├── maps/main.js.9c1d2e3f.map      # source maps
        │   └── sources/app/src/…              # original sources unpacked from maps
        └── target.com/
            └── inline/inline_0001.e5f6a7b8.js # inline <script> blocks

If a session name already exists, _2, _3 … is appended. Nothing is ever overwritten.

Network dump

One JSON object per line, with an event field: request · response · ws_open · ws_msg · ws_close · error. Each record is flushed to disk immediately, so a crash or a hard kill loses nothing.

Every record carries an id: a request, its response, its error and the WebSocket frames of one connection share it, even when the same URL is fetched in parallel. Responses add duration_ms and size. Headers are written twice — headers (a dict, as before) and headers_raw (a list of pairs that keeps repeated headers such as several Set-Cookie). WebSocket frames have a type, text or binary; binary frames are stored like binary bodies.

Captured JavaScript

• Scripts are stored complete. Bodies inside the .jsonl are truncated at 200 KB, but files in js/ are the full source — a 5 MB minified bundle is saved whole.
• Duplicates collapse by SHA-256. One bundle requested a hundred times is stored once, with hits: 100 in the manifest.
• Inline scripts are extracted from HTML. Tags with src= are skipped (they arrive as their own request), as are application/ld+json and text/template — those are not code.
• Filenames carry a short content hash, so different builds of the same app.js never overwrite each other.

Source maps

Browsers download source maps only while DevTools is open, so normally they never cross the wire. httpcrabber unpacks every map it does see — inline data: maps and any .map response — into js/<host>/sources/. With --sourcemaps it also requests the maps that scripts reference, through its own proxy and your upstream, so they land in the dump too, marked fetched_by: "sourcemap". Many production sites don't publish maps; when one does, you get the original project tree instead of a minified bundle.

Working with a session

Everything below works on a finished session — pass its folder or its .jsonl.

httpcrabber ls                                         # saved sessions, newest first
httpcrabber browse                                     # explore the newest one in the terminal
httpcrabber openapi LOGS/target_recon                  # → LOGS/target_recon/openapi.json
httpcrabber openapi LOGS/target_recon -o api.yaml --host 'api.*'
httpcrabber export har LOGS/target_recon               # → LOGS/target_recon/target_recon.har
httpcrabber export curl LOGS/target_recon -m /graphql -X POST
httpcrabber redact LOGS/target_recon                   # masked copy for sharing

The spec describes only what was observed: an endpoint you never called is missing, and a field seen in only some responses is optional. It is a head start, not a contract.

How it works

flowchart LR
    B["🌐 Chrome"] -->|HTTPS| M["🦀 mitmproxy<br/><sub>capture</sub>"]
    M --> P["🔌 pproxy bridge<br/><sub>scheme adapter</sub>"]
    P -->|"socks5 · http"| U["🧅 your upstream proxy"]
    U --> T["🎯 target"]
    M -.-> F[("📁 LOGS/session<br/><sub>JSONL + JS</sub>")]

    classDef hop fill:#0b0f0c,stroke:#39ff14,color:#d6ded6,stroke-width:1.5px
    classDef core fill:#0b0f0c,stroke:#ff2fd0,color:#ffffff,stroke-width:2px
    classDef store fill:#0b0f0c,stroke:#00e5ff,color:#d6ded6,stroke-width:1.5px
    class B,P,U,T hop
    class M core
    class F store

mitmproxy natively supports only http/https upstream proxies. To make SOCKS5 work transparently, httpcrabber starts a local pproxy bridge that speaks HTTP to mitmproxy and any scheme to your proxy. Both hops are on loopback, so the overhead is negligible.

src/httpcrabber/
  cli.py       arguments, prompts, main()
  session.py   orchestration: bridge → mitmproxy → CA → browser → live loop
  capture.py   mitmproxy addon: JSONL dump, JS collector, live stats
  proxy.py     upstream proxy parser        bridge.py   pproxy bridge
  ca.py        CA install per OS            browser.py  Chrome discovery per OS
  ui.py        animations, panels, feed     i18n.py     UI strings
  commands.py  redact / export / openapi    dump.py     reads a recorded session
  export.py    HAR and curl                 openapi.py  OpenAPI inference
  sourcemaps.py unpacking source maps       redact.py   masking secrets
  browse.py    terminal session browser     replay.py   request replay

⚠️ Security

Captured traffic contains live credentials. Session dumps routinely include Cookie, Set-Cookie, Authorization headers, API keys and tokens for every site you visited during the session.

• LOGS/ and *.jsonl are excluded by .gitignore — keep it that way.
• To share a session, make a masked copy: httpcrabber redact LOGS/target_recon → LOGS/target_recon_redacted/. Auth headers, cookies and tokens in URLs, forms and JSON become [REDACTED]; scripts are copied as is, the original session stays untouched.
• Never commit, upload or share a session dump before reviewing it.
• Treat a session folder as if it were your password manager export. Because effectively, it is.
• The tool installs a locally generated root CA. SECURITY.md explains how to remove it when you are done.

Responsible use

This is a tool for security research, API debugging, and interoperability work on systems you own or are authorized to test. You are responsible for complying with applicable law, the terms of service of the sites you access, and the privacy of any third-party data you encounter. Do not use it to access systems without permission.

Contributing

Issues and pull requests are welcome — see CONTRIBUTING.md for setup, conventions and how to add a language. Security reports go through SECURITY.md.

ruff check src tests run.py && pytest

License

GNU General Public License v3.0 — see the license file for the full text.