lockss-warcread 0.4.0

Top-Level Program

The top-level executable alone does not perform any action or default to a given command:

Usage: warcread [-h] {copyright,ext,extract,license,rep,report,version} ...
warcread: error: the following arguments are required: {copyright,ext,extract,license,rep,report,version}

extract (ext)

The extract (or alternatively ext) command can be used to look for the WARC response record for a given target URL in a set of WARC files, and extract the WARC record’s headers (--warc-headers/--wh/-A), the HTTP response’s headers (--http-headers/--hh/-H), or the HTTP response’s payload (--http-payload/--hp/-P):

Usage: warcread extract [-h] [-w WARC [WARC ...]] [-W WARCS [WARCS ...]] [-t TARGET_URL] [-H] [-P] [-A]

Required Arguments:
  -t, --target-url TARGET_URL
                        target URL

Optional Arguments:
  -w, --warc WARC [WARC ...]
                        (WARCs) add one or more WARC files to the set of WARC files to process (default: [])
  -W, --warcs WARCS [WARCS ...]
                        (WARCs) add the WARC files listed in one or more files to the set of WARC files to process (default: [])
  -H, --hh, --http-headers
                        (action) extract HTTP headers for target URL (default: False)
  -P, --hp, --http-payload
                        (action) extract HTTP payload for target URL (default: False)
  -A, --wh, --warc-headers
                        (action) extract WARC headers for target URL (default: False)

Help:
  -h, --help            show this help message and exit

The command needs:

• One or more WARC files, from the WARC File Options (--warc/-w options, --warcs/-W options).

• A target URL, from the --target-url/-t option.

report (rep)

The report (or alternatively rep) command can be used to produce a tabular (tab-separated) report over a set of WARC files, listing one or more columns of information about each:

Usage: warcread report [-h] [-w WARC [WARC ...]] [-W WARCS [WARCS ...]] [-c] [-n] [-d] [-p] [-r] [-s] [-m] [-u] [-D] [-F]

Optional Arguments:
  -w, --warc WARC [WARC ...]
                        (WARCs) add one or more WARC files to the set of WARC files to process (default: [])
  -W, --warcs WARCS [WARCS ...]
                        (WARCs) add the WARC files listed in one or more files to the set of WARC files to process (default: [])
  -c, --content-type    (column) output HTTP Content-Type (e.g. text/xml; charset=UTF-8) (default: False)
  -n, --http-code       (column) output HTTP response code (e.g. 404) (default: False)
  -d, --http-date       (column) output HTTP Date (default: False)
  -p, --http-protocol   (column) output HTTP protocol (e.g. HTTP/1.1) (default: False)
  -r, --http-reason     (column) output HTTP reason (e.g. Not Found) (default: False)
  -s, --http-status     (column) output HTTP status (e.g. HTTP/1.1 404 Not Found) (default: False)
  -m, --media-type      (column) output media type of HTTP Content-Type (e.g. text/xml) (default: False)
  -u, --url             (column) output URL of WARC record (default: False)
  -D, --warc-date       (column) output date of WARC record (default: False)
  -F, --warc-file       (column) output name of WARC file of origin (default: False)

Help:
  -h, --help            show this help message and exit

The command needs:

• One or more WARC files, from the WARC File Options (--warc/-w options, --warcs/-W options).

• One or more column options, chosen among --content-type/-c, --http-code/-n, --http-date/-d, --http-protocol/-p, --http-reason/-r, --http-status/-s, --media-type/-m, --url/-u, --warc-date/-D, and --warc-file/-F. Note that currently --url/-u is not always on.