Skip to main content

Saved webpage index and search

Project description

WebHist indexes a collection of saved webpages and provides an interface to search the index.

WebHist can handle the following archive file types:

  • MAFF files generated by Mozilla Archive Format, with MHT and Faithful Save

  • HTML files generated by Save Page WE


Package is uploaded on PyPI.

You can install it with pip:

$ pip install webhist


Create an index of archived webpages

i = webhist.Index("/path/to/index")

Index a single file


A file will not be re-indexed unless explicitly requested. Files are tracked by the path string passed to the add() function, so an absolute path and a relative path will be considered two different files.

The code below will update the file in the index

i.add("/path/to/file", update=True)

Add all files in a specified directory (note that it does not search within subdirectories)


Again, you can specify update=True to re-index files. You can also specify verbose=True to print information about whether or not files were indexed

i.add_path("/path/to/directory", verbose=True)

The output will look something like:

- file2 (already in index)
- file3 (exception type: error message)

In the example output above:

  • file1 was indexed correctly

  • file2 was already in the index, and was not re-indexed

  • file3 had a problem and was not indexed (python exception message shown)

After adding files, the changes to the index need to be committed


You can also cancel the changes


Once an index has been populated, you can run search queries against it. The syntax follows the Whoosh default query language. More information can be found here.

The code below searches for webpage archives that contain “webhist” and “installation”

results ="webhist installation")

The field searched by default is the content field. The following fields are indexed and searchable:

  • title (title of page)

  • content (content of page)

  • url (full URL of page)

  • fqdn (fully qualified domain name, e.g.

  • dn (domain name, e.g.

  • date (the date the webpage archive was saved)

For example, you can search the title field for webpages saved from

results ="title:webhist")

Shell Interface

A simple shell interface to a WebHist index is provided in examples/ You can clone the webhist repo and run it from the repo root:

$ python examples/ /path/to/archive -i /path/to/index

The -i parameter is optional. The default index location is /path/to/archive/index.

Run a search query:

webhist> search title:webhist

The output will look something like:

0: [2010-01-02 12:30:01] Title of page
1: [2011-02-03 16:20:25] Another page
2: [2013-06-12 00:00:01] Yet another page

To open page #2 from the search results:

webhist> open 2

To get more help:

webhist> help

To exit the shell:

webhist> exit


WebHist is released under the GNU Lesser General Public License, Version 3.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

webhist-1.0.0.tar.gz (4.8 kB view hashes)

Uploaded source

Built Distribution

webhist-1.0.0-py2-none-any.whl (20.8 kB view hashes)

Uploaded py2

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page