download-manager

Download Manager is a single python tool dedicated to help to download products from LTA and AIP.

Download manager features

The tool has been implemented with the objective to implements the following features:

• Manage download in a configurable local folder,

• Manage partial downloads and recovery downloads,

• Manage parallelized downloads

• Manage bulk download

• Management of connections error/retry

• Monitoring of downloads (bandwidth/progress) and errors

• Run in command line (GUI is a nice to have)

• Support of multiple sources

• Manage checksum validation of downloads

• Local storage management (identification of incomplete downloads to be resume, evictions...)

• Manage/anticipate quota limitation:

  • bandwidth limitation
  • parallel transfers number
  • transfer volume per time ..

• Download list issued from an OData filter

• Manage OData endpoint notifications/action when new product matching filter is up to allow performing routine downloads.

Install the download manager

Installing download-manager with execute the following in a terminal:

pip install download-manager

Options

Getting started

Download one product, with one thread:

download_manager --service odata://service.com/ -P 1 -u user -p password -l 1

Download 10 products:

download_manager --service odata://service.com/ -u user -p password -l 10

Use the silent option:

download_manager --service odata://service.com/ -u user -p password --quiet

Use filter to download specific products:

download_manager --service odata://service.com/ -u user -p password -f "Online eq true and startswith(Name,'S1')"

Use the resume option:

download_manager --service odata://service.com/ -u user -p password --db_folder 'database_folder' --resume

Use the storage management option:

download_manager --service odata://service.com/ -u user -p password -f "Online eq true and startswith(Name,'S2')" -l 25 -S 278209392 -o tests -db resources

Use python code:

from download_manager.download_manager import DownloadManager
from requests.auth import HTTPBasicAuth

service = 'https://odata.service.com'
auth=HTTPBasicAuth("user", "pwd")
dm = DownloadManager(service=service, auth=auth)
dm.start()

nodes = dm.find_nodes(filter="startswith(Name, 'S2B')")

for node in nodes:
    if node['Attributes']['cloudCover'].value == 0:
        dm.submit(node)

dm.join()
dm.stop()

Limitation

For now only odata implementation is available, quota still not supported.

The error management is implemented but cannot be parametrized in command line.

Offline product are not yet supported.