ParquetScope

See how DuckDB reads Parquet over HTTP: requests, byte ranges, column chunks, which files of a set it touches, and the effect of a different file layout.

ParquetScope captures one SELECT against one local Parquet file, or against an explicit, bounded set of files, serves them through a controlled HTTP endpoint, and maps the observed reads to each file's physical structure. Compare queries or file layouts, inspect linked requests and chunks, and share a capture that others can open without the source data.

Install and run

Install from PyPI with uv (the release page also carries the wheel):

uv tool install --python 3.13 parquetscope
parquetscope setup
parquetscope serve --demo

Open http://127.0.0.1:8400 and choose Run layout demo. The demo creates two equivalent Parquet layouts and measures the same query against both.

Python 3.10+ is required. The wheel includes the browser UI; Node.js is needed only for frontend development. setup downloads the signed httpfs extension for DuckDB 1.5.6 once. Subsequent local captures and offline viewing need no external internet, account, or cloud credentials.

To run from source instead:

uv sync --frozen
uv run parquetscope setup
uv run parquetscope serve --demo

Inspect your data

parquetscope serve /path/to/events.parquet

Use the input view in the SQL workspace. For example, on a file containing customer_id and amount_cents:

SELECT customer_id, sum(amount_cents)
FROM input
WHERE customer_id BETWEEN 1000 AND 1099
GROUP BY customer_id

In Range lab, select a GET to see its response interval, server writes, and intersecting column chunks. Select a chunk under Structure to find related requests. Zoom to selected range expands the selected interval; Whole file restores context. Compare runs shows each run's query, input identity, settings, and measured totals.

The source file is read-only. Each capture uses a fresh worker and executes one fully consumed SELECT. ParquetScope is a local diagnostic tool for trusted files and SQL, rather than a sandbox for hostile workloads.

Compare HTTP server profiles

Use SQL workspace → HTTP profile or explicit CLI options to compare fewer requests against additional bytes under controlled server conditions:

parquetscope capture events.parquet --sql 'SELECT sum(amount_cents) FROM input' \
  --response-delay-ms 40 --body-bytes-per-second 2097152 --deadline 60 \
  --output shaped.parquetscope.json

The default is zero delay and unrestricted body writes. Delay applies once before each prepared HEAD/GET response's headers. The body rate is shared by all GET responses and all files in one capture. A token bucket permits max(1, min(16384, ceil(rate / 10))) burst bytes. Low rates can reach the existing execution deadline. Profile changes apply to the next run. Comparison and offline viewers retain requested/effective profiles, burst and timing evidence; older artifacts explicitly have no recorded profile.

This models server waits and body writes, not transport RTT or a cloud provider. Successful writes mean acceptance into the server response stream, not confirmed DuckDB receipt. See the measurement contract and the reproducible HTTP profile lab.

Inspect a set of files

List the files in a manifest and give it to serve or capture:

parquetscope serve --manifest hive.manifest.json
parquetscope capture --manifest hive.manifest.json \
  --sql 'SELECT count(*) FROM input WHERE month = 3' --output set.parquetscope.json

The Files view lists every file by its location and partition segments with its requests, bytes and observed access (GET, HEAD only, or no observed requests). Inspect opens that file in Range lab. Offsets always belong to one file. Compare runs matches files by content hash, so two layouts of identical bytes line up row by row, and lists read options, versions and the limits of the comparison. Binding the view is part of the capture. Requests are labelled by the phase in which they started.

The manifest is explicit: no directory walking, globbing or discovery, a fixed budget (256 files, 16 GiB) and one schema per set. See the partition-pruning lab for a recorded flat versus Hive comparison of identical bytes.

Save and share

parquetscope capture events.parquet \
  --sql 'SELECT sum(amount_cents) FROM input WHERE customer_id BETWEEN 1000 AND 1099' \
  --output run.parquetscope.json

parquetscope serve --artifact run.parquetscope.json
parquetscope viewer run.parquetscope.json --output run.viewer.html

A capture contains the query, file identity, layout, HTTP observations, and derived metrics. It omits result preview rows and the original source path. SQL, column names, and headers remain visible, as are the relative paths and partition values of a file set, so check the contents before sharing.

The standalone HTML viewer opens directly as a file, without Python, a server, or network access. It can import additional captures and compare them offline. The UI's Export JSON and Offline viewer buttons provide the same exports.

Explore a real dataset

The NYC TLC case study measures the same DECIMAL aggregate over 3,475,226 Yellow Taxi records in three layouts:

• As delivered: 59,158,238-byte file; 14 GET + 2 HEAD; 11,387,877 server-written body bytes.
• Repacked in original order: 53,370,794-byte file; 118 GET + 2 HEAD; 11,446,545 body bytes.
• Sorted by pickup location: 59,417,539-byte file; 10 GET + 2 HEAD; 795,241 body bytes.

All three repetitions of each layout produced those totals and identical query results. A smaller file needed more requests, while the sorted file needed fewer body bytes. These are observations for one workload, not guaranteed request sequences or proportional speedups.

The case study includes nine captures, three standalone viewers, a report, checksums, and reproduction instructions. To build its static interactive lab from the repository:

# Install the development environment first; see CONTRIBUTING.md.
pnpm --dir frontend build:lab
.venv/bin/python -m http.server 8418 --bind 127.0.0.1 --directory .runtime/pages

Open http://127.0.0.1:8418/parquetscope/. The lab browses recorded captures and runs no SQL. It loads neither the source dataset nor an execution backend.

Read the measurements correctly

Server-written bytes count successful writes to the response stream. They do not establish how many bytes DuckDB received, decoded, or used. Unique coverage is the union of confirmed response intervals; repeated coverage is their summed length minus that union. Overlap is not automatically waste.

SQL success and capture completeness are separate. Incomplete traces retain available observations and mark totals as an observed subset. A fresh worker resets the client, but OS and disk caches remain uncontrolled. Loopback measurements do not measure S3 latency or cloud costs.

See Understanding measurements for definitions and limits.

Documentation

• User guide: CLI workflows, the viewer, settings, and troubleshooting.
• Documentation index: user and developer references.
• Artifact formats: portable v1 and v2 captures and validation rules.
• File-set manifest and the partition-pruning lab.
• NYC TLC case study: results, downloads, and reproduction.
• Contributing: development environment, architecture, and checks.
• Testing and compatibility and release guide.

Community and maintenance

Read Contributing before submitting a change. Use the repository's issue forms for bugs, feature requests, and documentation issues. Participation follows the code of conduct. Report security concerns through the process in SECURITY.md, and see CHANGELOG.md for release history and changes on main.

ParquetScope supports DuckDB 1.5.6 (and 2.0.0 for file-set captures, in an isolated environment prepared with parquetscope setup --engine 2.0.0), one file or one explicit set of identically structured files per run. It does not capture production S3 traffic or listing/globbing, merge schemas, run engines other than DuckDB, or predict query costs. The project is licensed under Apache-2.0.