Software Heritage - Object storage
Content-addressable object storage for the Software Heritage project.
Quick start
The easiest way to try the swh-objstorage object storage is to install it in a virtualenv. Here, we will be using virtualenvwrapper but any virtual env tool should work the same.
In the example below we will create a new objstorage using the pathslicer backend.
~/swh$ mkvirtualenv -p /usr/bin/python3 swh-objstorage
[...]
(swh-objstorage) ~/swh$ pip install swh.objstorage
[...]
(swh-objstorage) ~/swh$ cat >local.yml <<EOF
objstorage:
cls: pathslicing
root: /tmp/objstorage
slicing: 0:2/2:4/4:6
EOF
(swh-objstorage) ~/swh$ mkdir /tmp/objstorage
(swh-objstorage) ~/swh$ swh objstorage -C local.yml rpc-serve -p 15003
INFO:swh.core.config:Loading config file local.yml
======== Running on http://0.0.0.0:15003 ========
(Press CTRL+C to quit)
Now we have an API listening on http://0.0.0.0:15003 we can use to store and retrieve objects from. In an other terminal, you can import all the files from a local directory in this objstorage:
~/swh$ workon swh-objstorage
(swh-objstorage) ~/swh$ cat >remote.yml <<EOF
objstorage:
cls: remote
url: http://127.0.0.1:15003
EOF
(swh-objstorage) ~/swh$ swh objstorage -C remote.yml import .
INFO:swh.core.config:Loading config file remote.yml
Imported 1369 files for a volume of 722837 bytes in 2 seconds
Winery developer’s check-list
Working on Winery, the production backend, requires a slightly longer set-up.
First ensure your virtualenv contains the correct dependencies:
pip install -e .[winery]
Then create a postgres DB, called winery:
swh db create -d winery objstorage.backends.winery
swh db init -d winery objstorage.backends.winery
Prepare a container folder:
mkdir /home/martin/objstores/winery
And set it in a configuration file we’ll call localwinery.yml:
objstorage:
cls: winery
# boolean (false (default): allow writes, true: only allow reads)
readonly: false
rw_idle_timeout: 300
database:
# string: PostgreSQL connection string for the object index and read-write shards
db: "dbname=winery"
# string: PostgreSQL application name for connections (unset by default)
application_name: localwinery
shards_pools:
- ## Settings for a directory pool storing swh-shards
# Shards are stored in `{base_directory}/{pool_name}`
type: directory
base_directory: /home/martin/objstores/winery
pool_name: the-shards
# integer: threshold in bytes above which shards get packed. Can be
# overflowed by the max allowed object size.
shard_max_size: 100_000_000 # 100MB
shard_active_pool: the-shards
packer:
# Whether the packer should create shards in the shard pool, or defer to
# the pool manager (default: true, the packer creates images)
create_images: true
Note that you have to run a packer and a cleaner process separately. You might want to use a smaller max_size to trigger the packer more frequently.
Now you’ll need a few terminal splits/tabs because we’ll start 3 services
# Main service (winery writer) listens on 0.0.0.0:15003
swh objstorage -C localwinery.yml rpc-serve -p 15003
# Winery Packer Service
swh objstorage -C localwinery.yml winery packer
# optional, relevant later: RW Shard Cleaner
swh objstorage -C localwinery.yml winery rw-shard-cleaner
To import contents we’ll use the swh objstorage import, with the remote.yml configuration created in the Quick Start section in order to use the RPC server we’ve just started:
swh objstorage -C remote.yml import ~/swh-environment/
Test dependencies
Some tests do require non-python dependencies to be installed on the machine.
Azurite
The azurite tool is needed for Azure backend tests. Since it’s a npm package, you can install it using:
~/swh$ npm install azuriteand run tests with:
~/swh$ AZURITE_EXE=$HOME/node_modules/azurite/dist/src/blob/main.js tox
Ceph
Some Winery tests suites really manipulate RBD images, but this requires the ceph binary, configured to access a realistic cluster, and an explicit environment variable to flag you really want to run this suite (it takes a few minutes). Otherwise these tests are skipped.
To run these tests, on a developer’s machine you can install MicroCeph with snap. But it has permissions issues so you also need binaries from the Debian package and a hacky configuration. So we wrapped it in a Bash script. You need to pre-install the snap and ceph-common Debian packages (or your distribution’s equivalent), then run:
(swh) ~/swh-environment/swh-objstorage$ ./bin/test_winery_with_microceph.sh
This script uses the $PYTEST_FLAGS environment variable, that you can set for example to -v or -x.
Metadata
Release files for swh.objstorage 8.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| swh_objstorage-8.0.0.tar.gz | 141.1 kB | Without provenance | Details for swh_objstorage-8.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| swh_objstorage-8.0.0-py3-none-any.whl | Python 3 | none | any | Details for swh_objstorage-8.0.0-py3-none-any.whl |
Total release size: 299.5 kB
Release files / swh_objstorage-8.0.0.tar.gz
| Download URL | swh_objstorage-8.0.0.tar.gz |
|---|---|
| Size | 141.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4ff6397a2c90c858d971f9bf9e1ac853414e25bb38f4df5a16a8792f4dcb30fd
|
|
BLAKE2b-256 checksum How to use checksums |
faee9d1f05e367c2c76b28a1159a0b976da5e7916a2787e2b7b474400f393e8b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.15
|
Release files / swh_objstorage-8.0.0-py3-none-any.whl
| Download URL | swh_objstorage-8.0.0-py3-none-any.whl |
|---|---|
| Size | 158.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b4135bdc313ebaa051b8b9daa60dac6248f6f5e70e2567ae9492c2c5ef1a987f
|
|
BLAKE2b-256 checksum How to use checksums |
e28627477033cf3456202a4c3e9fae753e50b642324f1f51ea26ebd762786842
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.15
|
Security for swh.objstorage 8.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.