This tools allows you to get a local copy of all your cold stored archives on Logs Data Platform.
Project description
***********************************
Logs Data Platform - Archive mirror
***********************************
.. image:: https://img.shields.io/pypi/v/ldp-archive-mirror.svg
:target: https://pypi.python.org/pypi/ldp-archive-mirror/
:alt: Latest Version
.. image:: https://travis-ci.org/ovh/ldp-archive-mirror.svg?branch=master
:target: https://travis-ci.org/ovh/ldp-archive-mirror
:alt: Latest version
This tools allows you to get a local copy of all your cold stored archives on Logs Data Platform::
[2018-12-14 17:20:00,200 INFO LDP Mirror] Create local DB if not exists
[2018-12-14 17:20:00,204 INFO LDP Mirror] Looking for new archives of stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,190 INFO LDP Mirror] Service ldp-jb-52560 found for stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,579 INFO LDP Mirror] Archive 9fb75957-2cde-435e-bdd9-6dfd33663f2c added to cache
[2018-12-14 17:20:05,850 INFO LDP Mirror] Archive 11d8630a-7b38-4fa4-9d7c-dfd17b0b00f6 added to cache
...
[2018-12-14 17:20:08,392 INFO LDP Mirror] Directory mirror/e9397556-31d1-4d4d-b2bd-e5367b522cc8 created
[2018-12-14 17:20:08,535 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst missing, download scheduled
[2018-12-14 17:20:08,635 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst missing, download scheduled
...
[2018-12-14 17:20:09,535 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst after 596 seconds
[2018-12-14 17:20:09,745 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst after 598 seconds
...
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst
Quickstart
==========
First, install **ldp-archive-mirror** using `pip <https://pip.pypa.io/en/stable/>`_::
pip install -U ldp-archive-mirror
Then you can use the binary `ldp-mirror`::
usage: ldp-mirror [-h] [--app-key KEY] [--app-secret SECRET]
[--consumer-key KEY] [--ovh-region REGION] [--db DIR]
[--mirror DIR] [--ldp-host HOST] [--ldp-token TOKEN]
STREAM_ID [STREAM_ID ...]
LDP archive Mirror CLI - 0.1.0
positional arguments:
STREAM_ID LDP Stream UUIDs
optional arguments:
-h, --help show this help message and exit
--app-key KEY OVH application key (default: None)
--app-secret SECRET OVH application secret (default: None)
--consumer-key KEY OVH consumer key (default: None)
--ovh-region REGION OVH region (default: ovh-eu)
--db DIR Where to place the local sqlite database (default: db)
--mirror DIR Where to place your archives (default: mirror)
--ldp-host HOST If set, push logs of the current application to given
LDP hostname
--ldp-token TOKEN If set, push logs of the current application to
associated LDP stream token
Setup
=====
1. Create an application
------------------------
To interact with the APIs, the application needs to identify itself using an
`application_key` and an `application_secret`. To get them, you need
to register your application. Depending the API you plan to use, visit:
- `OVH Europe <https://eu.api.ovh.com/createApp/>`_
- `OVH US <https://api.ovhcloud.com/createApp/>`_
- `OVH North-America <https://ca.api.ovh.com/createApp/>`_
- `So you Start Europe <https://eu.api.soyoustart.com/createApp/>`_
- `So you Start North America <https://ca.api.soyoustart.com/createApp/>`_
- `Kimsufi Europe <https://eu.api.kimsufi.com/createApp/>`_
- `Kimsufi North America <https://ca.api.kimsufi.com/createApp/>`_
On the restriction step, we invite you to set the following access rules::
curl -XPOST -H"X-Ovh-Application: OVH_API_AK" -H "Content-type: application/json" \
https://eu.api.ovh.com/1.0/auth/credential -d '{
"accessRules": [
{
"method": "GET",
"path": "/dbaas/logs"
},
{
"method": "GET",
"path": "/dbaas/logs/*/output/graylog/stream"
},
{
"method": "GET",
"path": "/dbaas/logs/*/output/graylog/stream/*/archive*"
},
{
"method": "POST",
"path": "/dbaas/logs/*/output/graylog/stream/*/archive/*/url"
}
],
"redirection":"https://www.mywebsite.com/"
}'
Once created, you will obtain an **application key (OVH_API_AK)** and an **application
secret (OVH_API_AS)**.
2. Environment variables
------------------------
Default cli values can be set using environment:
============================ ==================== ============================================================================
Cli parameter Name About
============================ ==================== ============================================================================
--app-key `OVH_API_AK` OVH application key
--app-secret `OVH_API_AS` OVH application secret
--consumer-key `OVH_API_CK` OVH customer key
--ovh-region `OVH_API_REGION` OVH api location. Default: *ovh-eu*
--db `DB_DIRECTORY` Where to place the local sqlite database. Default: *db*
--mirror `MIRROR_DIRECTORY` Where to place your archives. Default: *mirror*
--ldp-host `LDP_HOST` If set, push logs of the current application to given LDP hostname
--ldp-token `LDP_TOKEN` If set, push logs of the current application to associated LDP stream token
============================ ==================== ============================================================================
3. Launch
---------
Once all the mandatory environment variables set, launch the process like this::
$ ldp-mirror STREAM_ID [STREAM_ID ...]
This will:
- looks for the LDP service associated with the given `STREAM_ID`(s)
- populate a local cache with all the archives found on the API
- request for each of them a temporary download url
- download the files when unseal time is reached
- ask every hour the api if a new archive is available
Create docker image from sources
================================
As this application is supposed to be kept alive indefinitely, launching it from a Docker daemon looks obvious.
To build the image form the sources, uses the given `Makefile`::
$ git clone https://github.com/ovh/ldp-archive-mirror
$ cd ldp-archive-mirror
$ make build-docker
And to run it::
$ docker run -v /my_backup/mirror/:/data/mirror -v /my_backup/db:/data/db \
-e OVH_API_AK=MY_OVH_AK -e OVH_API_AS=MY_OVH_AS -e OVH_API_CK=MY_OVH_CK \
-t MY_LDP_STREAM_ID_1 MY_LDP_STREAM_ID_2
Requirements
============
- Python >= 3.3
Project Links
=============
- PyPI: https://pypi.python.org/pypi/ldp-archive-mirror
- Issues: https://github.com/ovh/ldp-archive-mirror/issues
License
=======
`OVH SAS <./LICENSE>`_
Logs Data Platform - Archive mirror
***********************************
.. image:: https://img.shields.io/pypi/v/ldp-archive-mirror.svg
:target: https://pypi.python.org/pypi/ldp-archive-mirror/
:alt: Latest Version
.. image:: https://travis-ci.org/ovh/ldp-archive-mirror.svg?branch=master
:target: https://travis-ci.org/ovh/ldp-archive-mirror
:alt: Latest version
This tools allows you to get a local copy of all your cold stored archives on Logs Data Platform::
[2018-12-14 17:20:00,200 INFO LDP Mirror] Create local DB if not exists
[2018-12-14 17:20:00,204 INFO LDP Mirror] Looking for new archives of stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,190 INFO LDP Mirror] Service ldp-jb-52560 found for stream e9397556-31d1-4d4d-b2bd-e5367b522cc8
[2018-12-14 17:20:05,579 INFO LDP Mirror] Archive 9fb75957-2cde-435e-bdd9-6dfd33663f2c added to cache
[2018-12-14 17:20:05,850 INFO LDP Mirror] Archive 11d8630a-7b38-4fa4-9d7c-dfd17b0b00f6 added to cache
...
[2018-12-14 17:20:08,392 INFO LDP Mirror] Directory mirror/e9397556-31d1-4d4d-b2bd-e5367b522cc8 created
[2018-12-14 17:20:08,535 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst missing, download scheduled
[2018-12-14 17:20:08,635 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst missing, download scheduled
...
[2018-12-14 17:20:09,535 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst after 596 seconds
[2018-12-14 17:20:09,745 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst after 598 seconds
...
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst
[2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst downloaded
[2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst
Quickstart
==========
First, install **ldp-archive-mirror** using `pip <https://pip.pypa.io/en/stable/>`_::
pip install -U ldp-archive-mirror
Then you can use the binary `ldp-mirror`::
usage: ldp-mirror [-h] [--app-key KEY] [--app-secret SECRET]
[--consumer-key KEY] [--ovh-region REGION] [--db DIR]
[--mirror DIR] [--ldp-host HOST] [--ldp-token TOKEN]
STREAM_ID [STREAM_ID ...]
LDP archive Mirror CLI - 0.1.0
positional arguments:
STREAM_ID LDP Stream UUIDs
optional arguments:
-h, --help show this help message and exit
--app-key KEY OVH application key (default: None)
--app-secret SECRET OVH application secret (default: None)
--consumer-key KEY OVH consumer key (default: None)
--ovh-region REGION OVH region (default: ovh-eu)
--db DIR Where to place the local sqlite database (default: db)
--mirror DIR Where to place your archives (default: mirror)
--ldp-host HOST If set, push logs of the current application to given
LDP hostname
--ldp-token TOKEN If set, push logs of the current application to
associated LDP stream token
Setup
=====
1. Create an application
------------------------
To interact with the APIs, the application needs to identify itself using an
`application_key` and an `application_secret`. To get them, you need
to register your application. Depending the API you plan to use, visit:
- `OVH Europe <https://eu.api.ovh.com/createApp/>`_
- `OVH US <https://api.ovhcloud.com/createApp/>`_
- `OVH North-America <https://ca.api.ovh.com/createApp/>`_
- `So you Start Europe <https://eu.api.soyoustart.com/createApp/>`_
- `So you Start North America <https://ca.api.soyoustart.com/createApp/>`_
- `Kimsufi Europe <https://eu.api.kimsufi.com/createApp/>`_
- `Kimsufi North America <https://ca.api.kimsufi.com/createApp/>`_
On the restriction step, we invite you to set the following access rules::
curl -XPOST -H"X-Ovh-Application: OVH_API_AK" -H "Content-type: application/json" \
https://eu.api.ovh.com/1.0/auth/credential -d '{
"accessRules": [
{
"method": "GET",
"path": "/dbaas/logs"
},
{
"method": "GET",
"path": "/dbaas/logs/*/output/graylog/stream"
},
{
"method": "GET",
"path": "/dbaas/logs/*/output/graylog/stream/*/archive*"
},
{
"method": "POST",
"path": "/dbaas/logs/*/output/graylog/stream/*/archive/*/url"
}
],
"redirection":"https://www.mywebsite.com/"
}'
Once created, you will obtain an **application key (OVH_API_AK)** and an **application
secret (OVH_API_AS)**.
2. Environment variables
------------------------
Default cli values can be set using environment:
============================ ==================== ============================================================================
Cli parameter Name About
============================ ==================== ============================================================================
--app-key `OVH_API_AK` OVH application key
--app-secret `OVH_API_AS` OVH application secret
--consumer-key `OVH_API_CK` OVH customer key
--ovh-region `OVH_API_REGION` OVH api location. Default: *ovh-eu*
--db `DB_DIRECTORY` Where to place the local sqlite database. Default: *db*
--mirror `MIRROR_DIRECTORY` Where to place your archives. Default: *mirror*
--ldp-host `LDP_HOST` If set, push logs of the current application to given LDP hostname
--ldp-token `LDP_TOKEN` If set, push logs of the current application to associated LDP stream token
============================ ==================== ============================================================================
3. Launch
---------
Once all the mandatory environment variables set, launch the process like this::
$ ldp-mirror STREAM_ID [STREAM_ID ...]
This will:
- looks for the LDP service associated with the given `STREAM_ID`(s)
- populate a local cache with all the archives found on the API
- request for each of them a temporary download url
- download the files when unseal time is reached
- ask every hour the api if a new archive is available
Create docker image from sources
================================
As this application is supposed to be kept alive indefinitely, launching it from a Docker daemon looks obvious.
To build the image form the sources, uses the given `Makefile`::
$ git clone https://github.com/ovh/ldp-archive-mirror
$ cd ldp-archive-mirror
$ make build-docker
And to run it::
$ docker run -v /my_backup/mirror/:/data/mirror -v /my_backup/db:/data/db \
-e OVH_API_AK=MY_OVH_AK -e OVH_API_AS=MY_OVH_AS -e OVH_API_CK=MY_OVH_CK \
-t MY_LDP_STREAM_ID_1 MY_LDP_STREAM_ID_2
Requirements
============
- Python >= 3.3
Project Links
=============
- PyPI: https://pypi.python.org/pypi/ldp-archive-mirror
- Issues: https://github.com/ovh/ldp-archive-mirror/issues
License
=======
`OVH SAS <./LICENSE>`_
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
ldp-archive-mirror-0.1.0.tar.gz
(12.3 kB
view hashes)