Tools to periodically checks data integrity in swh-storage, swh-objstorage and swh-journal, reports errors, and (try to) fix them.
The Scrubber package is made of the following parts:
Checking
Highly parallel processes continuously read objects from a data store, compute checksums, and write any failure in a database, along with the data of the corrupt object.
There is one “checker” for each datastore package: storage (postgresql and cassandra), journal (kafka), and object storage (any backends).
The journal is “crawled” using its native streaming; others are crawled by range, reusing swh-storage’s backfiller utilities, and checkpointed from time to time to the scrubber’s database (in the checked_range table).
Storage
For the storage checker, a checking configuration must be created before being able to spawn a number of checkers.
A new configuration is created using the swh scrubber check init tool:
$ swh scrubber check init storage --object-type snapshot --nb-partitions 65536 --name chk-snp
Created configuration chk-snp [2] for checking snapshot in datastore storage postgresqlOne (or more) checking worker can then be spawned by using the swh scrubber check run command:
$ swh scrubber check run chk-snp
[...]Object storage
As with the storage checker, a checking configuration must be created before being able to spawn a number of checkers.
A new configuration is created using the swh scrubber check init tool:
$ swh scrubber check init objstorage --object-type content --nb-partitions 65536 --name check-contents
Created configuration check-contents [3] for checking content in datastore objstorage remoteBy default, an object storage checker detects missing and corrupted contents. To disable detection of missing contents, use the --no-check-references option of the swh check init command. To disable detection of corrupted contents, use the --no-check-hashes option of the swh check init command.
One (or more) checking worker can then be spawned by using the swh scrubber check run command:
if the content ids must be read from a storage instance
$ swh scrubber check run check-contents
[...]if the content ids must be read from a kafka content topic of swh-journal
$ swh scrubber check run check-contents --use-journal
[...]Journal
As with the other checkers, a checking configuration must be created before being able to spawn a number of checkers.
A new configuration is created using the swh scrubber check init tool:
$ swh scrubber check init journal --object-type directory --name check-dirs-journal
Created configuration check-dirs-journal [4] for checking directory in datastore journal kafkaOne (or more) checking worker can then be spawned by using the swh scrubber check run command:
$ swh scrubber check run check-dirs-journal
[...]Recovery
Then, from time to time, jobs go through the list of known corrupt objects, and try to recover the original objects, through various means:
Brute-forcing variations until they match their checksum
Recovering from another data store
As a last resort, recovering from known origins, if any
Reinjection
Finally, when an original object is recovered, it is reinjected in the original data store, replacing the corrupt one.
Metadata
Release files for swh.scrubber 4.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| swh_scrubber-4.0.0.tar.gz | 70.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| swh_scrubber-4.0.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 150.3 kB
Release files / swh_scrubber-4.0.0.tar.gz
| Download URL | swh_scrubber-4.0.0.tar.gz |
|---|---|
| Size | 70.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8726d5c2750a3d26495a4670f3fa0a1783f92c2925cbfcce3bf7a9ebc7bcdfb8
|
|
BLAKE2b-256 checksum How to use checksums |
f6f1264eef7ad43cf5dd17bcdf2aa7510f69423437eea7ff77f8569190610a17
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.9
|
Release files / swh_scrubber-4.0.0-py3-none-any.whl
| Download URL | swh_scrubber-4.0.0-py3-none-any.whl |
|---|---|
| Size | 80.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2c09455be99eb371da336249ae2d616a7f81bc0769a3cbba6d619157ed5218f5
|
|
BLAKE2b-256 checksum How to use checksums |
b3f333a3f208734f53279803e737e176447db127186c46f38af3983820863a9a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.11.9
|
