Skip to main content

Software Heritage Directory Loader

Project description

SWH-loader-dir

The Software Heritage Directory Loader is a tool and a library.

Its sole purpose is to walk a local directory and inject into the SWH dataset all unknown contained files from that directory structure.

Configuration

The loader needs a configuration file in {/etc/softwareheritage | ~/.config/swh | ~/.swh}/loader/dir.yml.

This file should be similar to this (adapt according to your needs):

storage:
  cls: remote
  args:
    url: http://localhost:5002/

Run

To run the loader, you can use either:

  • python3's toplevel
  • celery

Toplevel

Load directory directly from code or toplevel:

dir_path = '/home/storage/dir/'

# Fill in those
origin = {'url': 'some-origin', 'type': 'dir'}
visit_date = 'Tue, 3 May 2017 17:16:32 +0200'
revision = {
    'author': {'name': 'some', 'fullname': 'one', 'email': 'something'},
    'committer': {'name': 'some', 'fullname': 'one', 'email': 'something'},
    'message': '1.0 Released',
    'date': None,
    'committer_date': None,
    'type': 'tar',
    'metadata': {}
}
import logging
logging.basicConfig(level=logging.DEBUG)

from swh.loader.dir.tasks import LoadDirRepository
l = LoadDirRepository()
l.run_task(dir_path=dir_path, origin=origin, visit_date=visit_date,
           revision=revision, release=None, branch_name='master')

Celery

To use celery, add the following entries in the {/etc/softwareheritage | ~/.config/swh | ~/.swh}/worker.yml` file:

task_modules:
  - swh.loader.dir.tasks
task_queues:
  - swh_loader_dir

cf. swh-core's documentation for more details.

You can then send the following message to the task queue:

from swh.loader.dir.tasks import LoadDirRepository

# Fill in those
origin = {'url': 'some-origin', 'type': 'dir'}
visit_date = 'Tue, 3 May 2017 17:16:32 +0200'
release = None
revision = {}
occurrence = {}

# Send message to the task queue
LoaderDirRepository().run(('/path/to/dir', origin, visit_date, revision, release, [occurrence]))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

swh.loader.dir-0.0.35.tar.gz (9.4 kB view details)

Uploaded Source

Built Distribution

swh.loader.dir-0.0.35-py3-none-any.whl (24.4 kB view details)

Uploaded Python 3

File details

Details for the file swh.loader.dir-0.0.35.tar.gz.

File metadata

  • Download URL: swh.loader.dir-0.0.35.tar.gz
  • Upload date:
  • Size: 9.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.5.3

File hashes

Hashes for swh.loader.dir-0.0.35.tar.gz
Algorithm Hash digest
SHA256 cc36bd04f86b0cb6d1693b8f87c78750c91e6d37dcf6c4084f0530deadc804ce
MD5 a685a5ac2d1f6f5eeddefadeb830bcb0
BLAKE2b-256 b3f3bcaa6af8ecd803e9e58cd7e0022ad280c0fa0dc7115b364321fc7448f974

See more details on using hashes here.

File details

Details for the file swh.loader.dir-0.0.35-py3-none-any.whl.

File metadata

  • Download URL: swh.loader.dir-0.0.35-py3-none-any.whl
  • Upload date:
  • Size: 24.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.5.3

File hashes

Hashes for swh.loader.dir-0.0.35-py3-none-any.whl
Algorithm Hash digest
SHA256 6a22c2cd2413ff2efdaa4bfab13993b669b2abbe44ebf953a74045bdad6a4d84
MD5 3fcf9523111f47ce5f4ffbe360338d2c
BLAKE2b-256 f7e5c9f7d11b03b013c4ca3232d855c87cd62ce8cc49901baafc4ee73e44fc4a

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page