Skip to main content

Software Heritage - Object storage

Content-addressable object storage for the Software Heritage project.

Quick start

The easiest way to try the swh-objstorage object storage is to install it in a virtualenv. Here, we will be using virtualenvwrapper but any virtual env tool should work the same.

In the example below we will create a new objstorage using the pathslicer backend.

~/swh$ mkvirtualenv -p /usr/bin/python3 swh-objstorage
[...]
(swh-objstorage) ~/swh$ pip install swh.objstorage
[...]
(swh-objstorage) ~/swh$ cat >local.yml <<EOF
objstorage:
  cls: pathslicing
  root: /tmp/objstorage
  slicing: 0:2/2:4/4:6
EOF
(swh-objstorage) ~/swh$ mkdir /tmp/objstorage
(swh-objstorage) ~/swh$ swh objstorage -C local.yml rpc-serve -p 15003
INFO:swh.core.config:Loading config file local.yml
======== Running on http://0.0.0.0:15003 ========
(Press CTRL+C to quit)

Now we have an API listening on http://0.0.0.0:15003 we can use to store and retrieve objects from. In an other terminal, you can import all the files from a local directory in this objstorage:

~/swh$ workon swh-objstorage
(swh-objstorage) ~/swh$ cat >remote.yml <<EOF
objstorage:
  cls: remote
  url: http://127.0.0.1:15003
EOF
(swh-objstorage) ~/swh$ swh objstorage -C remote.yml import .
INFO:swh.core.config:Loading config file remote.yml
Imported 1369 files for a volume of 722837 bytes in 2 seconds

Winery developer’s check-list

Working on Winery, the production backend, requires a slightly longer set-up.

First ensure your virtualenv contains the correct dependencies:

pip install -e .[winery]

Then create a postgres DB, called winery:

swh db create -d winery objstorage.backends.winery
swh db init -d winery objstorage.backends.winery

Prepare a container folder:

mkdir /home/martin/objstores/winery

And set it in a configuration file we’ll call localwinery.yml:

objstorage:
  cls: winery

  # boolean (false (default): allow writes, true: only allow reads)
  readonly: false
  rw_idle_timeout: 300

  database:
    # string: PostgreSQL connection string for the object index and read-write shards
    db: "dbname=winery"

    # string: PostgreSQL application name for connections (unset by default)
    application_name: localwinery

  shards_pools:
    - ## Settings for a directory pool storing swh-shards
      # Shards are stored in `{base_directory}/{pool_name}`
      type: directory
      base_directory: /home/martin/objstores/winery
      pool_name: the-shards
      # integer: threshold in bytes above which shards get packed. Can be
      # overflowed by the max allowed object size.
      shard_max_size: 100_000_000  # 100MB
  shard_active_pool: the-shards

  packer:
    # Whether the packer should create shards in the shard pool, or defer to
    # the pool manager (default: true, the packer creates images)
    create_images: true

Note that you have to run a packer and a cleaner process separately. You might want to use a smaller max_size to trigger the packer more frequently.

Now you’ll need a few terminal splits/tabs because we’ll start 3 services

# Main service (winery writer)  listens on 0.0.0.0:15003
swh objstorage -C localwinery.yml rpc-serve -p  15003
# Winery Packer Service
swh objstorage -C localwinery.yml winery packer
# optional, relevant later: RW Shard Cleaner
swh objstorage -C localwinery.yml winery rw-shard-cleaner

To import contents we’ll use the swh objstorage import, with the remote.yml configuration created in the Quick Start section in order to use the RPC server we’ve just started:

swh objstorage -C remote.yml import ~/swh-environment/

Test dependencies

Some tests do require non-python dependencies to be installed on the machine.

Azurite

The azurite tool is needed for Azure backend tests. Since it’s a npm package, you can install it using:

~/swh$ npm install azurite

and run tests with:

~/swh$ AZURITE_EXE=$HOME/node_modules/azurite/dist/src/blob/main.js tox

Ceph

Some Winery tests suites really manipulate RBD images, but this requires the ceph binary, configured to access a realistic cluster, and an explicit environment variable to flag you really want to run this suite (it takes a few minutes). Otherwise these tests are skipped.

To run these tests, on a developer’s machine you can install MicroCeph with snap. But it has permissions issues so you also need binaries from the Debian package and a hacky configuration. So we wrapped it in a Bash script. You need to pre-install the snap and ceph-common Debian packages (or your distribution’s equivalent), then run:

(swh) ~/swh-environment/swh-objstorage$ ./bin/test_winery_with_microceph.sh

This script uses the $PYTEST_FLAGS environment variable, that you can set for example to -v or -x.

Metadata

Release files for swh.objstorage 9.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for swh.objstorage 9.0.0
File Size Uploaded Provenance Actions
swh_objstorage-9.0.0.tar.gz 146.2 kB Without provenance Details for swh_objstorage-9.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for swh.objstorage 9.0.0
File Interpreter ABI Platform Actions
swh_objstorage-9.0.0-py3-none-any.whl Python 3 none any Details for swh_objstorage-9.0.0-py3-none-any.whl

Total release size: 311.5 kB

Release files / swh_objstorage-9.0.0.tar.gz

Download URL swh_objstorage-9.0.0.tar.gz
Size 146.2 kB
Tags Source
SHA-256 checksum
How to use checksums
f4622ac0a9be41724f37f9303c1046d2be1fb958e904590028a9e959a2c348da
BLAKE2b-256 checksum
How to use checksums
a11a0b40d4d760281647ec0a9d3ec99df7bc319293a5a9ce10734bf1654b0bff
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Release files / swh_objstorage-9.0.0-py3-none-any.whl

Download URL swh_objstorage-9.0.0-py3-none-any.whl
Size 165.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
abcea89cbc0f78f883ca70353cbdd5ef2a27df4e909d0981a656e4570c10711e
BLAKE2b-256 checksum
How to use checksums
cdefd6b38b3aee733267025fcd2647ef6ae02cc3d7f305750a33f53b99977a5b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.15

Security for swh.objstorage 9.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

9.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page