Pacifica Archive Interface
Project description
Pacifica Archive Interface
This code is to provide the archive interface for the rest of the Pacifica code base. This code consists of some very specific algorithms and APIs to support data that might exist on tape or spinning disk.
Building and Installing
This code depends on the following libraries and python modules:
HPSS Client 7.4.1p2 Python JSON Python CTypes Python DocTest
This is a standard python distutils build process.
python ./setup.py build
python ./setup.py install
Running It
There are three ways of running the archive interface; POSIX, ORACLE_HSM_SIDEBAND and HPSS.
Posix File System Backend
python ./scripts/archiveinterfaceserver.py -t posix -p 8080 -a 127.0.0.1 --prefix /path
HPSS Archive Backend
python ./archiveinterfaceserver.py -t hpss -p 8080 -a 127.0.0.1 --prefix /path
ORACLE_HSM_SIDEBAND
python ./archiveinterfaceserver.py -t hsmsideband -p 8080 -a 127.0.0.1 --prefix /path
Config file
You can also pass a config file via the --config option. This is the first option and has highest priority. Second highest priority is the environment variable ARCHIVEI_CONFIG. This will be looked at if the --config option was not used. The final option is the application will default to config.cfg if neither of the first two options occured.
Config File Example:
Note here that the different backends use different config options. These are required for their respected archive types.
[posix]
use_id2filename = false
[hpss]
user = hpss.unix
auth = /var/hpss/etc/hpss.unix.keytab
[hsm_sideband]
sam_qfs_prefix = /tmp/path
schema = schema_name
host = host
user = user
password = pass
port = 3306
ID Mapping to File Names
The Pacifica software depends on a flat ID space for indexing files. This needs
to map to filenames on the backend storage in a nice way. To limit the number of
files in a single directory (or number of directories in a directory) we use the
algorithm in archiveinterface.id2filename
. This takes a number and breaks it
into bytes. Each byte is then represented in hex and used to build the directory
tree.
For example id2filename(12345)
becomes /39/3039
on the backend file system.
API Examples
Verify working
To verify the system is working do a GET against the system with no id specified.
curl -X GET http://127.0.0.1:8080
Sample output:
{
"message": "Pacifica Archive Interface Up and Running"
}
Put a File
The path in the URL should be only an integer specifying a unique file in the archive. Sending a different file to the same URL will over-write the contents of the previous file. Setting the Last- Modified header sets the mtime of the file in the archive and is required.
curl -X PUT -H 'Last-Modified: Sun, 06 Nov 1994 08:49:37 GMT' --upload-file /tmp/foo.txt http://127.0.0.1:8080/12345
Sample output:
{
"message": "File added to archive",
"total_bytes": "24"
}
Get a File
The HTTP GET
method is used to get the contents
of the specified file.
curl -o /tmp/foo.txt http://127.0.0.1:8080/12345
Sample output (without -o option): "Document Contents"
Status a File
The HTTP HEAD
method is used to get a JSON document describing the
status of the file. The status includes, but is not limited to, the
size, mtime, ctime, whether its on disk or tape. The values can be found
within the headers.
curl -I -X HEAD http://127.0.0.1:8080/12345
Sample output:
HTTP/1.0 204 No Content
Date: Fri, 07 Oct 2016 19:51:37 GMT
Server: WSGIServer/0.1 Python/2.7.5
X-Pacifica-Messsage: File was found
X-Pacifica-File: /tmp/12345
Content-Length: 18
Last-Modified: 1473806059.29
X-Pacifica-Ctime: 1473806059.29
X-Pacifica-Bytes-Per-Level: (18L,)
X-Pacifica-File-Storage-Media: disk
Content-Type: application/json
Stage a File
The HTTP POST
method is used to stage a file for use. In posix this
equates to a no-op on hpss it stages the file to the disk drive.
curl -X POST http://127.0.0.1:8080/12345
Sample Output:
{
"file": "/12345",
"message": "File was staged"
}
Move a File
The HTTP PATCH
method is used to move a file.
The upload file contains the path to current file on archive
The Id at the end of the url is where the file will be moved to
curl -X PATCH --upload-file /tmp/foo.json http://127.0.0.1:8080/2
Sample Output:
{
"message": "File Moved Successfully"
}
Sample Upload File
{
"path": "/path/to/file/file.1"
}
Extending Supported Backends
Create a backend directory
Under pacifica-archiveinterface->archiveinterface->archivebackends add a directory for the new backend type
Create Classes that Implement the Abstract Backend Class methods
Abstract backend classses can ge found under: pacifica-archiveinterface->archiveinterface->archivebackends->abstract Descriptions of all the methods that need to be abstracted exists in the comments above the class.
Update Backend Factory
Update the archive backend factory found here: pacifica-archiveinterface->archiveinterface->archivebackends->archive_backend_factory.py In this file is a load_backend_archive method. This method needs to have its logic extended to support the new backend type. This also entails loading the appropriate files for this backend using import
Update Interface Server
Update the archiveinterfaceserver.py file to support the new backend choice. File located: pacifica-archiveinterface->scripts->archiveinterfaceserver.py In this file the type argument is defined with its supported types. Need to extend that array to include the new backend type
Post Deployment Testing
Inside the post_deployment_test directory there is a file called deployment_test.py This file will run a series of tests against a deployed archive interface. The test are ordered so that they post, stage, status, and get files properly. There are a few global variables at the top of the file that need to be adjusted to each deployment.
Variables to set in deployment_test.py
export ARCHIVE_URL='http://127.0.0.1:8080/'
- ARCHIVE_URL is the URL to the newly deployed archive_interface
Running deployment_test.py
pytest -v post_deployment_tests/deployment_test.py
Output will be the status of the tests against the archive interface
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pacifica-archiveinterface-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e9e4a4ff8e318885542f8ffede69ac8d7b331d44322fdc34399a3687d466da6 |
|
MD5 | 13fbcde7a0463afa75ecda71c0184326 |
|
BLAKE2b-256 | 96ff1073f395b1ddf776247df0e9d0ddf0dab108835fd602a0453edd97e217c6 |
Hashes for pacifica_archiveinterface-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 378e5ac30ee2e0c7d4206116d4c244cca6d55e5617a45de43bd64b689767d625 |
|
MD5 | 1f156eb68d01b3be5a8209e17e8351ee |
|
BLAKE2b-256 | 5cc1fec3dad4cb85b5c076d57a68891936dc4e8cd85c3d5c0e1d2c86509c4fbb |
Hashes for pacifica_archiveinterface-0.1.2-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9744c532aa047e8649eb718bf8a700a6bc9dfe954a406159dbeecb7595e02ab |
|
MD5 | 3755125b5744a9415a05fcd6eac58140 |
|
BLAKE2b-256 | e2b3dd35dece0eb4785c7ec92be312a3cefff23853fa0dc95f3c145dedb90f8c |