Software Heritage storage manager
Abstraction layer over the archive, allowing to access all stored source code artifacts as well as their metadata.
See the documentation for more details.
Python tests for this module include tests that cannot be run without a local Postgresql database, so you need the Postgresql server executable on your machine (no need to have a running Postgresql server). They also expect a cassandra server.
$ sudo apt install libpq-dev postgresql-11 cassandra
Non Debian-like host
The tests expects the path to
cassandra to either be unspecified, it is then
looked up at
/usr/sbin/cassandra, either specified through the environment
Optionally, you can avoid running the cassandra tests.
(swh) :~/swh-storage$ tox -- -m 'not cassandra'
It is strongly recommended to use a virtualenv. In the following, we
consider you work in a virtualenv named
swh. See the
developer setup guide
for a more details on how to setup a working environment.
You can install the package directly from pypi:
(swh) :~$ pip install swh.storage [...]
Or from sources:
(swh) :~$ git clone https://forge.softwareheritage.org/source/swh-storage.git [...] (swh) :~$ cd swh-storage (swh) :~/swh-storage$ pip install . [...]
Then you can check it's properly installed:
(swh) :~$ swh storage --help Usage: swh storage [OPTIONS] COMMAND [ARGS]... Software Heritage Storage tools. Options: -h, --help Show this message and exit. Commands: rpc-serve Software Heritage Storage RPC server.
The best way of running Python tests for this module is to use tox.
(swh) :~$ pip install tox
From the sources directory, simply use tox:
(swh) :~/swh-storage$ tox [...] ========= 315 passed, 6 skipped, 15 warnings in 40.86 seconds ========== _______________________________ summary ________________________________ flake8: commands succeeded py3: commands succeeded congratulations :)
Note: it is possible to set the
JAVA_HOME environment variable to specify the
version of the JVM to be used by Cassandra. For example, at the time of writing
this, Cassandra does not support java 14, so one may want to use for example
(swh) :~/swh-storage$ export JAVA_HOME=/usr/lib/jvm/java-14-openjdk-amd64/bin/java (swh) :~/swh-storage$ tox [...]
The storage server can be locally started. It requires a configuration file and a running Postgresql database.
A typical configuration
storage.yml file is:
storage: cls: local db: "dbname=softwareheritage-dev user=<user> password=<pwd>" objstorage: cls: pathslicing root: /tmp/swh-storage/ slicing: 0:2/2:4/4:6
which means, this uses:
a local storage instance whose db connection is to
the objstorage uses a local objstorage instance whose:
rootpath is /tmp/swh-storage,
slicing scheme is
0:2/2:4/4:6. This means that the identifier of the content (sha1) which will be stored on disk at first level with the first 2 hex characters, the second level with the next 2 hex characters and the third level with the next 2 hex characters. And finally the complete hash file holding the raw content. For example: 00062f8bd330715c4f819373653d97b3cd34394c will be stored at 00/06/2f/00062f8bd330715c4f819373653d97b3cd34394c
Note that the
root path should exist on disk before starting the server.
Starting the storage server
If the python package has been properly installed (e.g. in a virtual env), you should be able to use the command:
(swh) :~/swh-storage$ swh storage rpc-serve storage.yml
This runs a local swh-storage api at 5002 port.
(swh) :~/swh-storage$ curl http://127.0.0.1:5002 <html> <head><title>Software Heritage storage server</title></head> <body> <p>You have reached the <a href="https://www.softwareheritage.org/">Software Heritage</a> storage server.<br /> See its <a href="https://docs.softwareheritage.org/devel/swh-storage/">documentation and API</a> for more information</p>
And then what?
storage: cls: remote url: http://localhost:5002/
You could directly define a local storage with the following snippet:
storage: cls: local db: service=swh-dev objstorage: cls: pathslicing root: /home/storage/swh-storage/ slicing: 0:2/2:4/4:6
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size swh.storage-0.21.0-py3-none-any.whl (246.3 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size swh.storage-0.21.0.tar.gz (279.1 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for swh.storage-0.21.0-py3-none-any.whl