Skip to main content

Software Heritage lister

Project description

Collection of listers for source code distribution places like development forges, FOSS distributions, package managers, etc. Each lister is in charge to enumerate the software origins (e.g., VCS, packages, etc.) available at a source code distribution place.

A lister is a component from the Software Heritage stack aims to produce listings of software origins and their urls hosted on various public developer platforms or package managers. As these operations are quite similar, this package provides a set of Python modules abstracting common software origins listing behaviors.

It also provides several lister implementations, contained in the Python swh.lister.* modules. See this documentation for the list of supported listers.

Dependencies

All required python dependencies can be found in the requirements*.txt files located at the root of the repository.

In order to be able to run all the listers (and thus execute the tests), some tools must be available on your system, namely:

  • opam

  • tar

  • psql

On a Debian-like system, you may use:

Local deployment

Lister configuration

Each lister implemented so far by Software Heritage (bitbucket, cgit, cran, debian, gitea, github, gitlab, gnu, golang, launchpad, npm, packagist, phabricator, pypi, tuleap, maven) must be configured by following the instructions below (please note that you have to replace <lister_name> by one of the lister name introduced above).

Preparation steps

  1. mkdir ~/.config/swh/

  2. create configuration file ~/.config/swh/listers.yml

Configuration file sample

Minimalistic configuration shared by all listers to add in file ~/.config/swh/listers.yml:

scheduler:
  cls: 'remote'
  args:
    url: 'http://localhost:5008/'

credentials: {}

Note: This expects scheduler (5008) service to run locally

Executing a lister

Once configured, a lister can be executed by using the swh CLI tool with the following options and commands:

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister <lister_name> [lister_parameters]

Examples:

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister bitbucket

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister cran

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitea url=https://codeberg.org/api/v1/

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitlab url=https://salsa.debian.org/api/v4/

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister npm

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister pypi

Licensing

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

See top-level LICENSE file for the full text of the GNU General Public License along with this program.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

swh_lister-6.9.4.tar.gz (1.4 MB view details)

Uploaded Source

Built Distribution

swh_lister-6.9.4-py3-none-any.whl (1.4 MB view details)

Uploaded Python 3

File details

Details for the file swh_lister-6.9.4.tar.gz.

File metadata

  • Download URL: swh_lister-6.9.4.tar.gz
  • Upload date:
  • Size: 1.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.11.9

File hashes

Hashes for swh_lister-6.9.4.tar.gz
Algorithm Hash digest
SHA256 6e2beb86a075950c9f06a77cafbee16f1e1d91f6f01be9d085395512ee119939
MD5 bedf2c17c0397a35ffa1f178dcf55836
BLAKE2b-256 9f7190f124e9f37f4b2bc7d620c14a51cf260ec84edc1e32385919094dc406b7

See more details on using hashes here.

File details

Details for the file swh_lister-6.9.4-py3-none-any.whl.

File metadata

  • Download URL: swh_lister-6.9.4-py3-none-any.whl
  • Upload date:
  • Size: 1.4 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.11.9

File hashes

Hashes for swh_lister-6.9.4-py3-none-any.whl
Algorithm Hash digest
SHA256 99a3826ea14ea67eb51a8b25c1806b8336dea1e706e349d61f78293db7b5759d
MD5 3669ce30fcd248229ad0100d6227610c
BLAKE2b-256 1f0e8aed1aef8d4fc3d612f26145caeaa65c6c54b57790b465dbd744af66f938

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page