A URL scanner, maintainer, and validator
Project description
- webval is a system that will scan documents for fully-qualified
HTTP URLs, keeping its database fresh with newly-seen URLs. It can then be requested to validate the URLs, whereby it will attempt to access each URL via an HTTP request and record the response code; it maintains a list of the most recent codes that have been retrieved. Response codes are classified as “good” (URL is correct and a valid page is there) and “bad” (URL is invalid or outdated). By default any code other than a 2xx code is considered bad, but this can be changed (e.g., to ignore 3xx redirection codes).
webval can then be used in report mode where it will scan documents for URLs as before, but will report invalid URLs (that is, URLs in the database which have a number of “bad” codes exceeding a certain threshhold). These are then printed to stderr in a format that shows the file and line number the URLs were seen in so that they can be corrected.
