gamdam 0.5.0

Installation

gamdam requires Python 3.8 or higher. Just use pip for Python 3 (You have pip, right?) to install gamdam and its dependencies:

python3 -m pip install gamdam

gamdam also requires git-annex v10.20220222 or higher to be installed separately in order to run.

Usage

gamdam [<options>] [<input-file>]

gamdam reads a series of JSON entries from a file (or from standard input if no file is specified) following the input format described below. It feeds the URLs and output paths to git-annex addurl, and once each file has finished downloading, it attaches any listed metadata and extra URLs using git-annex metadata and git-annex registerurl, respectively.

Note that the latter step can only be performed on files tracked by git-annex; if you, say, have configured git-annex to not track text files, then any text files downloaded will not have any metadata or alternative URLs registered.

Library Usage

gamdam can also be used as a Python library. It exports the following:

async def download(
    repo: pathlib.Path,
    objects: AsyncIterator[Downloadable],
    jobs: Optional[int] = None,
    addurl_opts: Optional[List[str]] = None,
    subscriber: Optional[anyio.abc.ObjectSendStream[DownloadResult]] = None,
) -> Report

Download the items yielded by the async iterator objects to the directory repo (which must be part of a git-annex repository) and set their metadata. jobs is the number of parallel jobs for the git-annex addurl process to use; a value of None means to use one job per CPU core. addurl_opts contains any additional arguments to append to the git-annex addurl command.

If subscriber is supplied, it will be sent a DownloadResult (see below) for each completed download, both successful and failed. This can be used to implement custom post-processing of downloads.

class Downloadable(pydantic.BaseModel):
    path: pathlib.Path
    url: pydantic.AnyHttpUrl
    metadata: Optional[Dict[str, List[str]]] = None
    extra_urls: Optional[List[pydantic.AnyHttpUrl]] = None

Downloadable is a pydantic model used to represent files to download; see Input Format above for the meanings of the fields.

class DownloadResult(pydantic.BaseModel):
    downloadable: Downloadable
    success: bool
    key: Optional[str] = None
    error_messages: Optional[List[str]] = None

DownloadResult is a pydantic model used to represent a completed download. It contains the original Downloadable, a flag to indicate download success, the downloaded file’s git-annex key (only set if the download was successful and the file is tracked by git-annex) and any error messages from the addurl process (only set if the download failed).

@dataclass
class Report:
    downloaded: int
    failed: int

Report is used as the return value of download(); it contains the number of files successfully downloaded and the number of failed downloads.