Skip to main content

Software Heritage - Indexer

Tools to compute multiple indexes on SWH’s raw contents:

  • content:

    • mimetype

    • fossology-license

    • metadata

  • origin:

    • metadata (intrinsic, using the content indexer; and extrinsic)

An indexer is in charge of:

  • looking up objects

  • extracting information from those objects

  • store those information in the swh-indexer db

There are multiple indexers working on different object types:

  • content indexer: works with content sha1 hashes

  • revision indexer: works with revision sha1 hashes

  • origin indexer: works with origin identifiers

Indexation procedure:

  • receive batch of ids

  • retrieve the associated data depending on object type

  • compute for that object some index

  • store the result to swh’s storage

Current content indexers:

  • mimetype (queue swh_indexer_content_mimetype): detect the encoding and mimetype

  • fossology-license (queue swh_indexer_fossology_license): compute the license

  • metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta vocabulary)

Current origin indexers:

  • metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta and ForgeFed vocabularies)

Custom indexers and metadata mappings can be added as plugins.

Metadata

Release files for swh.indexer 5.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for swh.indexer 5.0.0
File Size Uploaded Provenance Actions
swh_indexer-5.0.0.tar.gz 191.7 kB Without provenance Details for swh_indexer-5.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for swh.indexer 5.0.0
File Interpreter ABI Platform Actions
swh_indexer-5.0.0-py3-none-any.whl Python 3 none any Details for swh_indexer-5.0.0-py3-none-any.whl

Total release size: 438.4 kB

Release files / swh_indexer-5.0.0.tar.gz

Download URL swh_indexer-5.0.0.tar.gz
Size 191.7 kB
Tags Source
SHA-256 checksum
How to use checksums
02bf8c09ec84a4ba8566504b64b1f77db72399cca6efa8e048d3157cac32bf87
BLAKE2b-256 checksum
How to use checksums
9ee2aba49d7dbee74e6504fb9f9d17aba5915693eed1a0866cf4f7944f7e2e80
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Release files / swh_indexer-5.0.0-py3-none-any.whl

Download URL swh_indexer-5.0.0-py3-none-any.whl
Size 246.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2ec9858887973a3ddbcd56dd8e97fc88d13300bd859c53ec0c39aadc579240b9
BLAKE2b-256 checksum
How to use checksums
b2a6f21d211b2a400e0691fb50d4ce341f96508d7fd39c7485f713a91715ef69
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Security for swh.indexer 5.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

5.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page