Software Heritage indexer
Project description
Tools to compute multiple indexes on SWH’s raw contents:
content:
mimetype
fossology-license
metadata
origin:
metadata (intrinsic, using the content indexer; and extrinsic)
An indexer is in charge of:
looking up objects
extracting information from those objects
store those information in the swh-indexer db
There are multiple indexers working on different object types:
content indexer: works with content sha1 hashes
revision indexer: works with revision sha1 hashes
origin indexer: works with origin identifiers
Indexation procedure:
receive batch of ids
retrieve the associated data depending on object type
compute for that object some index
store the result to swh’s storage
Current content indexers:
mimetype (queue swh_indexer_content_mimetype): detect the encoding and mimetype
fossology-license (queue swh_indexer_fossology_license): compute the license
metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta vocabulary)
Current origin indexers:
metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta and ForgeFed vocabularies)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for swh.indexer-3.5.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 058b3a160eef217a8c9d934e5f87d3c6d2cb10a79aa8a57f53ca1bbec7a7f75e |
|
MD5 | 2862724debe6f996d25aa9a4cb6ebaa0 |
|
BLAKE2b-256 | 6622627be8798779f33f66cfeb66c2139ecfe7cbdbba49768d331775ed30fda5 |