Skip to main content

This tools allows you to get a local copy of all your cold stored archives on Logs Data Platform.

Project description

Latest Version Latest version

This tools allows you to get a local copy of all your cold stored archives on Logs Data Platform:

[2018-12-14 17:20:00,200 INFO LDP Mirror] Create local DB if not exists
[2018-12-14 17:20:00,204 INFO LDP Mirror] Looking for new archives of stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,190 INFO LDP Mirror] Service ldp-jb-52560 found for stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,579 INFO LDP Mirror] Archive 9fb75957-2cde-435e-bdd9-6dfd33663f2c added to cache
[2018-12-14 17:20:05,850 INFO LDP Mirror] Archive 11d8630a-7b38-4fa4-9d7c-dfd17b0b00f6 added to cache
...
[2018-12-14 17:20:08,392 INFO LDP Mirror] Directory mirror/e9397556-31d1-4d4d-b2bd-e5367b522cc8 created
[2018-12-14 17:20:08,535 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst missing, download scheduled
[2018-12-14 17:20:08,635 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst missing, download scheduled
...
[2018-12-14 17:20:09,535 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst after 596 seconds
[2018-12-14 17:20:09,745 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst after 598 seconds
...
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst

Quickstart

First, install ldp-archive-mirror using pip:

pip install -U ldp-archive-mirror

Then you can use the binary ldp-mirror:

usage: ldp-mirror [-h] [--app-key KEY] [--app-secret SECRET]
              [--consumer-key KEY] [--ovh-region REGION] [--db DIR]
              [--mirror DIR] [--ldp-host HOST] [--ldp-token TOKEN]
              STREAM_ID [STREAM_ID ...]

LDP archive Mirror CLI - 0.1.0

positional arguments:
  STREAM_ID            LDP Stream UUIDs

optional arguments:
  -h, --help           show this help message and exit
  --app-key KEY        OVH application key (default: None)
  --app-secret SECRET  OVH application secret (default: None)
  --consumer-key KEY   OVH consumer key (default: None)
  --ovh-region REGION  OVH region (default: ovh-eu)
  --db DIR             Where to place the local sqlite database (default: db)
  --mirror DIR         Where to place your archives (default: mirror)
  --ldp-host HOST      If set, push logs of the current application to given
                       LDP hostname
  --ldp-token TOKEN    If set, push logs of the current application to
                       associated LDP stream token

Setup

1. Create an application

To interact with the APIs, the application needs to identify itself using an application_key and an application_secret. To get them, you need to register your application. Depending the API you plan to use, visit:

On the restriction step, we invite you to set the following access rules:

curl -XPOST -H"X-Ovh-Application: OVH_API_AK" -H "Content-type: application/json" \
https://eu.api.ovh.com/1.0/auth/credential  -d '{
    "accessRules": [
        {
            "method": "GET",
            "path": "/dbaas/logs"
        },
        {
            "method": "GET",
            "path": "/dbaas/logs/*/output/graylog/stream"
        },
        {
            "method": "GET",
            "path": "/dbaas/logs/*/output/graylog/stream/*/archive*"
        },
        {
            "method": "POST",
            "path": "/dbaas/logs/*/output/graylog/stream/*/archive/*/url"
        }
    ],
    "redirection":"https://www.mywebsite.com/"
}'

Once created, you will obtain an application key (OVH_API_AK) and an application secret (OVH_API_AS).

2. Environment variables

Default cli values can be set using environment:

Cli parameter Name About
–app-key OVH_API_AK OVH application key
–app-secret OVH_API_AS OVH application secret
–consumer-key OVH_API_CK OVH customer key
–ovh-region OVH_API_REGION OVH api location. Default: ovh-eu
–db DB_DIRECTORY Where to place the local sqlite database. Default: db
–mirror MIRROR_DIRECTORY Where to place your archives. Default: mirror
–ldp-host LDP_HOST If set, push logs of the current application to given LDP hostname
–ldp-token LDP_TOKEN If set, push logs of the current application to associated LDP stream token

3. Launch

Once all the mandatory environment variables set, launch the process like this:

$ ldp-mirror STREAM_ID [STREAM_ID ...]

This will:

  • looks for the LDP service associated with the given STREAM_ID (s)
  • populate a local cache with all the archives found on the API
  • request for each of them a temporary download url
  • download the files when unseal time is reached
  • ask every hour the api if a new archive is available

Create docker image from sources

As this application is supposed to be kept alive indefinitely, launching it from a Docker daemon looks obvious.

To build the image form the sources, uses the given Makefile:

$ git clone https://github.com/ovh/ldp-archive-mirror
$ cd ldp-archive-mirror
$ make build-docker

And to run it:

$ docker run -v /my_backup/mirror/:/data/mirror -v /my_backup/db:/data/db \
-e OVH_API_AK=MY_OVH_AK -e OVH_API_AS=MY_OVH_AS -e OVH_API_CK=MY_OVH_CK \
-t MY_LDP_STREAM_ID_1 MY_LDP_STREAM_ID_2

Requirements

  • Python >= 3.3

License

OVH SAS

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
ldp-archive-mirror-0.1.7.tar.gz (12.3 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page