Tools to compute multiple indexes on SWH’s raw contents:

  • content:

    • mimetype

    • fossology-license

    • metadata

  • origin:

    • metadata (intrinsic, using the content indexer; and extrinsic)

An indexer is in charge of:

  • looking up objects

  • extracting information from those objects

  • store those information in the swh-indexer db

There are multiple indexers working on different object types:

  • content indexer: works with content sha1 hashes

  • revision indexer: works with revision sha1 hashes

  • origin indexer: works with origin identifiers

Indexation procedure:

  • receive batch of ids

  • retrieve the associated data depending on object type

  • compute for that object some index

  • store the result to swh’s storage

Current content indexers:

  • mimetype (queue swh_indexer_content_mimetype): detect the encoding and mimetype

  • fossology-license (queue swh_indexer_fossology_license): compute the license

  • metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta vocabulary)

Current origin indexers:

  • metadata: translate file from an ecosystem-specific formats to JSON-LD (using schema.org/CodeMeta and ForgeFed vocabularies)

Custom indexers and metadata mappings can be added as plugins.

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

Owner

PyPI data
Data sourced directly from PyPI's database.

Credits

Author: Software Heritage developers

License

GNU General Public License v3 (GPLv3)

Requires

Python >=3.11

Provides Extra

testing

Classifiers

Report project as malware

Metadata

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

Owner

PyPI data
Data sourced directly from PyPI's database.

Credits

Author: Software Heritage developers

License

GNU General Public License v3 (GPLv3)

Requires

Python >=3.11

Provides Extra

testing

Classifiers

Report project as malware

Release files for swh.indexer 4.12.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for swh.indexer 4.12.1
File Size Uploaded
swh_indexer-4.12.1.tar.gz 191.7 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for swh.indexer 4.12.1
File Interpreter ABI Platform
swh_indexer-4.12.1-py3-none-any.whl 246.6 kB Python 3 none any Details

Total release size: 438.3 kB

Release files / swh_indexer-4.12.1.tar.gz

Download URL swh_indexer-4.12.1.tar.gz
Size 191.7 kB
Tags Source
SHA-256 checksum
How to use checksums 		77ffc577b3bf2f37ead21865e3420314052df1d4c1fb69a925f4189d50e94e22
BLAKE2b-256 checksum
How to use checksums 		e685d4b5d382576f7db2d82e5f71b75fa1533b26cc224ee6b56a648c7be7354e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/7.0.0 CPython/3.11.15

Release files / swh_indexer-4.12.1-py3-none-any.whl

Download URL swh_indexer-4.12.1-py3-none-any.whl
Size 246.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums 		c2679a6f67f2f828d951687ce420c915f1b7c668c557ea56401541a29a127b56
BLAKE2b-256 checksum
How to use checksums 		ecf5350469a0c54726d23016ad073ff46f74436a79e60a026f9608e7a3de23b9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/7.0.0 CPython/3.11.15

Release history Release notifications | RSS feed

This release

4.12.1 This release

2 release files

4.12.0

2 release files

4.11.0

2 release files

4.10.0

2 release files

4.9.0

2 release files

4.8.3

2 release files

4.8.2

2 release files

4.8.1

2 release files

4.8.0

2 release files

4.7.0

2 release files

4.6.0

2 release files

4.5.2

2 release files

4.5.1

2 release files

4.5.0

2 release files

4.4.1

2 release files

4.3.1

2 release files

4.3.0

2 release files

4.2.2

2 release files

4.2.1

2 release files

4.1.0

2 release files

4.0.0

2 release files

3.9.1

2 release files

3.9.0

2 release files

3.8.1

2 release files

3.7.0

2 release files

3.6.2

2 release files

3.6.1

2 release files

3.6.0

2 release files

3.5.0

2 release files

3.4.0

2 release files

3.3.0

2 release files

3.2.0

2 release files

3.1.0

2 release files

3.0.0

2 release files

2.12.0

2 release files

2.11.1

2 release files

2.11.0

2 release files

2.10.0

2 release files

2.9.4

2 release files

2.9.3

2 release files

2.9.1

2 release files

2.9.0

2 release files

2.8.0

2 release files

2.7.3

2 release files

2.7.2

2 release files

2.7.1

2 release files

2.6.0

2 release files

2.5.0

2 release files

2.4.4

2 release files

2.4.3

2 release files

2.4.2

2 release files

2.4.1

2 release files

2.4.0

2 release files

2.3.0

2 release files

2.2.2

2 release files

2.2.1

2 release files

2.2.0

2 release files

2.1.0

2 release files

2.0.2

2 release files

2.0.1

2 release files

2.0.0

2 release files

1.9.4

2 release files

1.2.0

2 release files

1.1.0

2 release files

1.0.0

2 release files

0.8.2

2 release files

0.8.1

2 release files

0.8.0

2 release files

0.7.0

2 release files

0.6.4

2 release files

0.6.3

2 release files

0.6.2

2 release files

0.6.1

2 release files

0.6.0

2 release files

0.5.0

2 release files

0.4.2

2 release files

0.4.1

2 release files

0.4.0

2 release files

0.3.0

2 release files

0.2.4

2 release files

0.2.3

2 release files

0.2.2

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.1

2 release files

0.1.0

2 release files

0.0.171

2 release files

0.0.170

2 release files

0.0.169

2 release files

0.0.168

2 release files

0.0.167

2 release files

0.0.166

2 release files

0.0.165

2 release files

0.0.164

2 release files

0.0.163

2 release files

0.0.162

2 release files

0.0.161

2 release files

0.0.160

2 release files

0.0.159

2 release files

0.0.158

2 release files

0.0.157

2 release files

0.0.156

2 release files

0.0.155

2 release files

0.0.154

2 release files

0.0.153

2 release files

0.0.152

2 release files

0.0.151

2 release files

0.0.150

2 release files

0.0.149

2 release files

0.0.148

2 release files

0.0.147

2 release files

0.0.146

2 release files

0.0.145

2 release files

0.0.144

2 release files

0.0.143

2 release files

0.0.142

2 release files

0.0.141

2 release files

0.0.140

2 release files

0.0.139

2 release files

0.0.138

2 release files

0.0.137

2 release files

0.0.136

2 release files

0.0.135

2 release files

0.0.134

2 release files

0.0.133

2 release files

0.0.132

2 release files

0.0.131

2 release files

0.0.129

2 release files

0.0.128

2 release files

0.0.127

2 release files

0.0.126

2 release files

0.0.125

2 release files

0.0.124

2 release files

0.0.123

2 release files

0.0.121

2 release files

0.0.120

2 release files

0.0.118

2 release files

0.0.68

2 release files

0.0.67

2 release files

0.0.65

2 release files

0.0.60

2 release files

0.0.59

2 release files

0.0.57

2 release files

0.0.56

2 release files

0.0.55

2 release files

0.0.54.post3

2 release files

0.0.52

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page