Software Heritage Datastore Scrubber
Project description
Tools to periodically checks data integrity in swh-storage and swh-objstorage, reports errors, and (try to) fix them.
This is a work in progress; some of the components described below do not exist yet (cassandra storage checker, objstorage checker, recovery, and reinjection)
The Scrubber package is made of the following parts:
Checking
Highly parallel processes continuously read objects from a data store, compute checksums, and write any failure in a database, along with the data of the corrupt object.
There is one “checker” for each datastore package: storage (postgresql and cassandra), journal (kafka), and objstorage.
The journal is “crawled” using its native streaming; others are crawled by range, reusing swh-storage’s backfiller utilities, and checkpointed from time to time to the scrubber’s database (in the checked_range table).
Storage
For the storage checker, a checking configuration must be created before being able to spawn a number of checkers.
A new configuration is created using the swh scrubber check init tool:
$ swh scrubber check init --object-type snapshot --nb-partitions 65536 --name chk-snp
Created configuration chk-snp [2] for checking snapshot in datastore storage postgresql
One (or more) checking worker can then be spawned by using the swh scrubber check storage command:
$ swh scrubber check storage chk-snp
[...]
Recovery
Then, from time to time, jobs go through the list of known corrupt objects, and try to recover the original objects, through various means:
Brute-forcing variations until they match their checksum
Recovering from another data store
As a last resort, recovering from known origins, if any
Reinjection
Finally, when an original object is recovered, it is reinjected in the original data store, replacing the corrupt one.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for swh.scrubber-2.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c763de8af6a2b23a8282567194dda4a36517592cad83e6333a70aff8fad9a8d |
|
MD5 | 53205917efa54c96e4285d219d7587e3 |
|
BLAKE2b-256 | 2c842c586a72eda7567968809df0ebcfe05ecae429c97864c8b52ac2d900334b |