swh.indexer 4.8.3

Software Heritage indexer

Tools to compute multiple indexes on SWH’s raw contents:

• content:

  • mimetype

  • fossology-license

  • metadata

• origin:

  • metadata (intrinsic, using the content indexer; and extrinsic)

An indexer is in charge of:

• looking up objects

• extracting information from those objects

• store those information in the swh-indexer db

There are multiple indexers working on different object types:

• content indexer: works with content sha1 hashes

• revision indexer: works with revision sha1 hashes

• origin indexer: works with origin identifiers

Indexation procedure:

• receive batch of ids

• retrieve the associated data depending on object type

• compute for that object some index

• store the result to swh’s storage

Current content indexers:

• mimetype (queue swh_indexer_content_mimetype): detect the encoding and mimetype

• fossology-license (queue swh_indexer_fossology_license): compute the license

• metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta vocabulary)

Current origin indexers:

• metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta and ForgeFed vocabularies)

Custom indexers and metadata mappings can be added as plugins.