wayback2warc

PyPI - Version

A CLI tool that exports captures from the Wayback Machine and packs them into WARC files.

usage: wayback2warc [-h] [-c COLLAPSE] [-f FILTER] [-m] [-p PROXY] [-t THREADS] [-w WARC_SIZE] url [prefix]

positional arguments:
  url                   *.example.org or example.org/*
  prefix                output path prefix

options:
  -h, --help            show this help message and exit
  -c, --collapse COLLAPSE
                        dedupe records based on returned key
  -f, --filter FILTER   filter records based on returned boolean
  -m, --meta            dump cdx metadata into stdout
  -p, --proxy PROXY     url formatted proxy, can be a single value or a file
  -t, --threads THREADS
                        concurrent downloads (default: 2)
  -w, --warc-size WARC_SIZE
                        max size of produced warc files in MB (default: 1024)

Install

pip install wayback2warc

Usage

Download all subdomains, all pages, and all captures for a domain:

wayback2warc '*.example.org'

Download all pages and all captures for a prefix:

wayback2warc 'example.org/*'

Download all captures for a single page:

wayback2warc 'example.org'

Download all pages but only a yearly capture of each:

wayback2warc 'example.org/*' --collapse 'lambda m: (m.urlkey, m.timestamp[:4])'

Filtering and collapsing

This tool implements filtering/collapsing functionality on the client-side rather than relying on the CDX server to do so. This is a necessity due to the way CDX paging works, but on the bright side it allows for much more granular queries.

Instead of a query language, lambda functions are used to narrow down captures for downloading. A CaptureMetadata instance is passed to both functions, and the returned value is used as a key for collapsing, or as a boolean for filtering.

Collapse captures by URL excluding query: --collapse 'lambda m: m.urlkey.split("?")[0]'

Filter captures before 2020 and exclude images: --filter 'lambda m: m.timestamp < "2020" and "image/" not in m.mimetype'

WARC records

The generated WARC files use per-record GZIP compression. Request records are not included. Response record payloads are always saved uncompressed with a Content-Length header, any Content-Encoding or Transfer-Encoding headers are stripped. Headers and status-lines are reconstructed on a best-effort basis, but the capitalization and order of headers may differ from the original capture. The HTTP version is always set to HTTP/1.1.

Development Status
- 5 - Production/Stable
Environment
- Console
Intended Audience
- System Administrators
License
- OSI Approved :: MIT License
Natural Language
- English
Operating System
- OS Independent
Programming Language
- Python
Topic
- Utilities

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

wayback2warc-0.1.4.tar.gz (6.2 kB view details)

Uploaded Jul 5, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

wayback2warc-0.1.4-py3-none-any.whl (7.3 kB view details)

Uploaded Jul 5, 2026 Python 3

File details

Details for the file wayback2warc-0.1.4.tar.gz.

File metadata

Download URL: wayback2warc-0.1.4.tar.gz
Upload date: Jul 5, 2026
Size: 6.2 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for wayback2warc-0.1.4.tar.gz
Algorithm	Hash digest
SHA256	`373b26829aeb656926058f59c54919cf9b87c4f6251f73dae86a0196c5e3039d`
MD5	`e460ed0f2e264a493f337014a6bbf8a0`
BLAKE2b-256	`76ead8d1903125dbc25a7d0de80a5ef3301e9cda75af4175602735b5bb77babd`

See more details on using hashes here.

Provenance

The following attestation bundles were made for wayback2warc-0.1.4.tar.gz:

Publisher: publish.yml on tmctmt/wayback2warc

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: wayback2warc-0.1.4.tar.gz
- Subject digest: 373b26829aeb656926058f59c54919cf9b87c4f6251f73dae86a0196c5e3039d
- Sigstore transparency entry: 2081928888
- Sigstore integration time: Jul 5, 2026
Source repository:
- Permalink: tmctmt/wayback2warc@0d27ed86b614a4c2c70beb14db17550569a21b50
- Branch / Tag: refs/tags/0.1.4
- Owner: https://github.com/tmctmt
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yml@0d27ed86b614a4c2c70beb14db17550569a21b50
- Trigger Event: release