warc3-wet-clueweb09

Note: This is a fork of the original (now dead) warc repository.

Updated to handle problems with the ClueWeb09 files.

Changes are based on this repository (which only supports python2)

WARC (Web ARChive) is a file format for storing web crawls.

This warc library makes it very easy to work with WARC files.:

import warc
with warc.open("test.warc") as f:
    for record in f:
        print(record['WARC-Target-URI'], record['Content-Length'])

And WET files.:

import warc
with warc.open("test.warc.wet") as f:
    for record in f:
        print(record['WARC-Target-URI'], record['Content-Length'])

Documentation

The documentation of the warc library is available at http://warc.readthedocs.org/.

Apart from the install from pip, which will not work for this warc3 version, the interface as described there is unchanged.

This software is licensed under GPL v2. See LICENSE file for details.

Original Python2 Versions:

Python3 Port:

Modification * Willian Zhang

0.2.5 replace utf8 errors in headers

0.2.4 support ClueWeb09

0.2.3 Support seeking in WARC/WET

0.2.2 Allow WET parse

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution for warc3-wet-clueweb09 0.2.5
File	Size	Uploaded
warc3-wet-clueweb09-0.2.5.tar.gz	17.9 kB	Dec 7, 2020	Details

Download URL	warc3-wet-clueweb09-0.2.5.tar.gz
Size	17.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`3054bfc07da525d5967df8ca3175f78fa3f78514c82643f8c81fbca96300b836`
BLAKE2b-256 checksum How to use checksums	`9fc1dd817bf57e0274dacb10e0ac868cb6cd70876950cf361c41879c030a2b8b`
Upload date	Dec 7, 2020
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/3.2.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/50.3.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.7.9`