Skip to main content
Help improve PyPI by participating in a 5-minute user interface survey!

Command line tools and libraries for handling and manipulating WARC files (and HTTP contents)

Project Description

dependencies
setuptools
unittest2
python 2.6

hanzo warc tools:

warcvalid.py
returns 0 if the arguments are all valid arc/warc files
non zero on error

warcdump.py - writes human readable summary of warcfiles:
usage: python warcdump.py foo.warc foo.warc.gz
autodetects input format when filenames are passed
i.e recordgzip vs plaintext, warc vs arc

assumes uncompressed warc on stdin if no args

warcfilter.py
python warcfilter.py pattern file file file
searches all headers for regex pattern
use -i to invert search
use -U to constrain to url
use -T to constrain to record type
use -C to constrain to content-type

autodetects and stdin like warcdump

prints out a warc format by default.

warc2warc.py:
python warc2warc <input files>

autodetects compression on file
args, assumes uncompressed stdin if none

use -Z to write compressed output

i.e warc2warc -Z input > input.gz

should ignore buggy records in input

arc2warc.py
creates a crappy warc file from arc files on input
a handful of headers are preserved
use -Z to write compressed output
i.e arc2warc -Z input.arc > input.warc.gz

warcindex.py
spits out an index like this:
#WARC filename offset warc-type warc-subject-uri warc-record-id content-type content-length
warccrap/mywarc.warc 1196018 request /images/slides/hanzo_markm__wwwoh.pdf <urn:uuid:fd1255a8-d07c-11df-b125-12313b0a18c6> application/http;msgtype=request 193
warccrap/mywarc.warc 1196631 response http://www.hanzoarchives.com/images/slides/hanzo_markm__wwwoh.pdf <urn:uuid:fd2614f8-d07c-11df-b125-12313b0a18c6> application/http;msgtype=response 3279474
not great, but a start

notes:

arc2warc uses the conversion rules from the earlier arc2warc.c
as a starter for converting the headers

I haven't profiled the code yet (and don't plan to until it falls over)

warcvalid barely skirts some of the iso standard:
missing things: strict whitespace, required headers check...
mime quoted printable header encoding
treating headers as utf8

things left to do (in no order):
lots more testing.
supporting pre 1.0 warc files
add more documentation
support more commandline options for output and filenames
s3 urls


-- tef thomas.figg@hanzoarchives.com

Release history Release notifications

This version
History Node

4.10.0

History Node

4.9.0

History Node

4.7

History Node

4.8.1-

History Node

4.8-

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
warctools-4.10.0.tar.gz (24.6 kB) Copy SHA256 hash SHA256 Source None Sep 2, 2016

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page