Software Heritage Directory Loader
Project description
SWH-loader-dir
The Software Heritage Directory Loader is a tool and a library.
Its sole purpose is to walk a local directory and inject into the SWH dataset all unknown contained files from that directory structure.
Configuration
The loader needs a configuration file in {/etc/softwareheritage | ~/.config/swh | ~/.swh}
/loader/dir.yml.
This file should be similar to this (adapt according to your needs):
storage:
cls: remote
args:
url: http://localhost:5002/
Run
To run the loader, you can use either:
- python3's toplevel
- celery
Toplevel
Load directory directly from code or toplevel:
dir_path = '/home/storage/dir/'
# Fill in those
origin = {'url': 'some-origin', 'type': 'dir'}
visit_date = 'Tue, 3 May 2017 17:16:32 +0200'
revision = {
'author': {'name': 'some', 'fullname': 'one', 'email': 'something'},
'committer': {'name': 'some', 'fullname': 'one', 'email': 'something'},
'message': '1.0 Released',
'date': None,
'committer_date': None,
'type': 'tar',
'metadata': {}
}
import logging
logging.basicConfig(level=logging.DEBUG)
from swh.loader.dir.tasks import LoadDirRepository
l = LoadDirRepository()
l.run_task(dir_path=dir_path, origin=origin, visit_date=visit_date,
revision=revision, release=None, branch_name='master')
Celery
To use celery, add the following entries in the
{/etc/softwareheritage | ~/.config/swh | ~/.swh}
/worker.yml` file:
task_modules:
- swh.loader.dir.tasks
task_queues:
- swh_loader_dir
cf. swh-core's documentation for more details.
You can then send the following message to the task queue:
from swh.loader.dir.tasks import LoadDirRepository
# Fill in those
origin = {'url': 'some-origin', 'type': 'dir'}
visit_date = 'Tue, 3 May 2017 17:16:32 +0200'
release = None
revision = {}
occurrence = {}
# Send message to the task queue
LoaderDirRepository().run(('/path/to/dir', origin, visit_date, revision, release, [occurrence]))
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file swh.loader.dir-0.0.35.tar.gz
.
File metadata
- Download URL: swh.loader.dir-0.0.35.tar.gz
- Upload date:
- Size: 9.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.5.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
cc36bd04f86b0cb6d1693b8f87c78750c91e6d37dcf6c4084f0530deadc804ce
|
|
MD5 |
a685a5ac2d1f6f5eeddefadeb830bcb0
|
|
BLAKE2b-256 |
b3f3bcaa6af8ecd803e9e58cd7e0022ad280c0fa0dc7115b364321fc7448f974
|
File details
Details for the file swh.loader.dir-0.0.35-py3-none-any.whl
.
File metadata
- Download URL: swh.loader.dir-0.0.35-py3-none-any.whl
- Upload date:
- Size: 24.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.5.3
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
6a22c2cd2413ff2efdaa4bfab13993b669b2abbe44ebf953a74045bdad6a4d84
|
|
MD5 |
3fcf9523111f47ce5f4ffbe360338d2c
|
|
BLAKE2b-256 |
f7e5c9f7d11b03b013c4ca3232d855c87cd62ce8cc49901baafc4ee73e44fc4a
|