bq-tools-dirupload 0.8.13.4

Upload/Download files to bisque/ViQi in parallel

Tools

bq-dirupload : upload local files to remote system

bq-dirfetch : download remote files to local system

Configuration

Create a file config file containing you login info at ~/.bisque/config

    [default]
    host=https://data.viqi.org/
    user=demo
    password=<demopass>

bq-dirupload

1. Upload directories and files to a ViQi/BisQue server

usage: Upload files to bisque [-h] [-c CONFIG] [--profile PROFILE] [-n] [-d [DEBUG]] [--debug-file DEBUG_FILE] [-q] [-a CREDENTIALS] [--host HOST] [--user USER] [--password PASSWORD] [--alias ALIAS] [--tag TAG]
                              [--path-tags PATH_TAGS] [--re-tags RE_TAGS] [--re-only] [--mustmap] [--include INCLUDE] [--exclude EXCLUDE] [--dataset DATASET] [--threads THREADS] [--empty] [-s] [-r]
                              [--tagmap TAGMAP] [--fixedtags FIXEDTAGS] [--json-args JSON_ARGS] [--protocol {binary,fsxlustre,multipart}] [--destination-dir DESTINATION_DIR] [--transfer-only]
                              [--register-only REGISTER_ONLY] [--archive-type {zip-bisque,zip-multi-file,zip-time-series,zip-z-stack,zip-dicom}] [--watch]
                              [directories ...]

positional arguments:
  directories           director(ies) to upload

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        bisque config
  --profile PROFILE     Profile to use in bisque config
  -n, --dry-run         report actions w/o changes
  -d [DEBUG], --debug [DEBUG]
                        set debug level: debug,info,warn,error
  --debug-file DEBUG_FILE
                        output filename for debug messages
  -q, --quiet           print actions
  -a CREDENTIALS, --credentials CREDENTIALS
                        A bisque login.. admin
  --host HOST           Default bisque server to connect to
  --user USER           User to connect with
  --password PASSWORD   passwor to connect with
  --alias ALIAS         Use admin credentials to login as alias
  --tag TAG             Add name:value pair, can be templated with other values mycode:$site$season
  --path-tags PATH_TAGS
                        tag names for a parsable path i.e. /root/project/date//subject/ or \root\project\data\subject
  --re-tags RE_TAGS     re expressions for tags i.e. (?P<location>\w+)--(?P<date>[\d-]+)
  --re-only             Accept files only if match re-tags
  --mustmap             Contextual tag must have a value in a tagmap
  --include INCLUDE     shell expression for files to include. Can be repeated
  --exclude EXCLUDE     shell expression for files to exclude. Can be repeated
  --dataset DATASET     create dataset and add files to it
  --threads THREADS     set number of uploader threads
  --empty               Allow empty files to be uploaded
  -s, --skip-loaded     Skip upload if there is file with the same name already present on the server
  -r, --replace-uploaded
                        Force upload of metadata even if file exists on server
  --tagmap TAGMAP       Supply a map tag/value -> tag/value found in tag path and re decoder. [context_key:]carp=carpenteria or [context_key:]@tagmap.csv
  --fixedtags FIXEDTAGS
                        key:tag=value or key:@fileoftags fixed tags to add to resource: First column is key: including filename or image_path
  --json-args JSON_ARGS
                        Load default arguments from json file i.e { "--path-tags" : "project/date" }
  --protocol {binary,fsxlustre,multipart}
                        Set the upload protocol or let tyhe system choose
  --destination-dir DESTINATION_DIR
                        Upload to destination-dir on server .. must be valid mount i.e /home/my-uploads
  --transfer-only       Upload files only without registration
  --register-only REGISTER_ONLY
                        register files without actually uploading them use argument as prefix path for
  --archive-type {zip-bisque,zip-multi-file,zip-time-series,zip-z-stack,zip-dicom}
                        zip archive will be given a type: bisque, z-stack, t-stack
  --watch               Watch directories listed for new files and transfer

bq-dirupload -n  --threads 1 --re-tags "(?P<photo_site_code>\w+)_(?P<target_assemblage>\D+)(?P<plot>\d+)_(?P<season>\D+)(?P<year>\d+).+\.JPG" --dataset upload --tagmap target_assemblage:@speciesmap.csv --tagmap photo_site_code:@locationmap.csv --tagmap season:fa=fall --tagmap year:15=2015 --fixedtags photo_site_code:@photo_code_reference_2019_0912.csv TopLevelDir

 Magic decoder ring:
    -n : dry run
    --threads 1: one thread for debugging
    --retags :   use filename to create tags: photo_site_code, target_assemblage, season and year.
    --dataset : create a dataset "upload"
    --tagmap target_assemblage:@speciesmap.csv: use value ins speciesmap.csv to rename tag/values for target_assemblage
    --tagmap photo_site_code:@locationmap: Use location map to rename tag/value from photo_site_code
    --tagmap season:fa=fall : rename season 'fa' to 'fall'
    --tagmap year:15=2015 : remame year from '15' to 2015
    --fixedtags photo_site_code:@photo_code_reference_2019_0912.csv  :  use photo_site_code to read a set of fixed tags to be applied to the resource

   A map is consists of [context_tag:]oldval=newval or [context_tag:]@map.csv where csv is a two column table of old value, new value

Other interesting Arguments

    --debug-file somefile :  write actions to somefile
    --path-tags   map components of the file path  to metadata tags i.e.   on the path ghostcrabs/manua/winter/somefile.jpg
                  --path-tags=project/site/season  ->  { project:ghostcrabs, site:manua, season:winter} as tags on somefile.jpg
                  --path-tags=/site//              ->  {site:manua }   skipping root and season elements

Examples

1. Simple decoding of path/filename into metadata tags

   bq-dirupload --skip --include *.JPG --tag activity:bear --re-tags "(?P<location>\w+)_(?P<camera>\w+)__(?P<date>[\d-]+)__(?P<time>[\d-]+).*" ~/Desktop/pics/bears/

2. Complex mapping of path/filename metadata into tag

   bq-dirupload -n --threads 1 --re-tags "(?P<photo_site_code>\w+)_(?P<target_assemblage>\D+)(?P<plot>\d+)_(?P<season>\D+)(?P<year>\d+).+\.JPG" --dataset upload --tagmap target_assemblage:@speciesmap.csv --tagmap photo_site_code:@locationmap.csv --tagmap season:fa=fall --tagmap year:15=2015 --fixedtags photo_site_code:@photo_code_reference_2019_0912.csv

Magic decoder ring:

    -n : dry run to see action without doing them
    --threads 1: one thread for debugging
    --retags :   use filename to create tags: photo_site_code, target_assemblage, season and year.
    --dataset : create a dataset "upload"
    --tagmap target_assemblage:@speciesmap.csv: use value ins speciesmap.csv to rename tag/values for target_assemblage
    --tagmap photo_site_code:@locationmap: Use location map to rename tag/value from photo_site_code
    --tagmap season:fa=fall : rename season 'fa' to 'fall'
    --tagmap year:15=2015 : remame year from '15' to 2015
    --fixedtags photo_site_code:@photo_code_reference_2019_0912.csv  :  use photo_site_code to read a set of fixed tags to be applied to the resource

   A map is consists of [context_tag:]oldval=newval or [context_tag:]@map.csv where csv is a two column table of old value, new value

Other interesting Arguments

    --debug-file somefile :  write actions to somefile
    --path-tags   map components of the file path  to metadata tags i.e. project/site/season   would map ghostcrabs/manua/winter/somefile.jpg
                  would get { project:ghostcrabs, site:manua, season:winter} as tags on somefile.jpg

bq-dirfetch

Downoad files to the local fileystsem

usage: Fetch files from bisque [-h] [-c CONFIG] [--profile PROFILE] [-n]
                               [-d [DEBUG]] [--debug-file DEBUG_FILE] [-q]
                               [-a CREDENTIALS] [--bisque-host BISQUE_HOST]
                               [--alias ALIAS] [--threads THREADS]
                               [--meta {false,xml,tablecsv,json}]
                               [--view VIEW]
                               [datasets [datasets ...]]

positional arguments:
  datasets              datasets to download

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        bisque config
  --profile PROFILE     Profile to use in bisque config
  -n, --dry-run         report actions w/o changes
  -d [DEBUG], --debug [DEBUG]
                        set debug level: debug,info,warn,error
  --debug-file DEBUG_FILE
                        output filename for debug messages
  -q, --quiet           print actions
  -a CREDENTIALS, --credentials CREDENTIALS
                        A bisque login.. admin
  --bisque-host BISQUE_HOST
                        Default bisque server to connect to
  --alias ALIAS         Use admin credentials to login as alias
  --threads THREADS     set number of uploader threads
  --meta {false,xml,tablecsv,json}
                        fetch metadata of resources: xml, tablecsv or false
  --view VIEW           Metadata view i.e. deep,clean or view=TAG1,TAG2

bq-dirfetch -n  --threads 1 dataset_uri

Examples

  $ bq-dirfetch  https://data.viqi.org/data_service/00-aGLFAaVudK2SesxZbFgtSA
  $ ls My\ data

  30PKCVampNFp1Exte_DSP.tif   30PKCVampNFp6Exte_DSP.tif 30PKCVampNFz5Live5-21-2004_2-21-06_PM.tif  'My Data.tablecsv'
  30PKCVampNFp3Exte_DSP.tif   30PKCVampNFz1Live5-21-2004_1-36-13_PM.tif   D4P1.TIF
  $ cat My\ data/My\ Data.tablecsv
  name,filename,tag1,tag1.subtag1,upload_datetime
  D4P1.TIF,D4P1.TIF,,,2021-04-06 11:45:25.939292
  30PKCVampNFp1Exte_DSP.tif,30PKCVampNFp1Exte_DSP.tif,,,2021-04-06 11:47:12.354264
  30PKCVampNFp3Exte_DSP.tif,30PKCVampNFp3Exte_DSP.tif,,,2021-04-06 11:53:07.626649
  30PKCVampNFp6Exte_DSP.tif,30PKCVampNFp6Exte_DSP.tif,val1,subval1,2021-04-06 11:54:38.046633
  30PKCVampNFz1Live5-21-2004_1-36-13_PM.tif,30PKCVampNFz1Live5-21-2004_1-36-13_PM.tif,,,2021-04-06 12:07:28.216727
  30PKCVampNFz5Live5-21-2004_2-21-06_PM.tif,30PKCVampNFz5Live5-21-2004_2-21-06_PM.tif,,,2021-04-06 12:12:12.523292

Advanced Configuration and Arguments

Pass a file of arguments with --json-args=path-to-file.json where path-to-file.json has:

{
    "directories"   : [ "tmp", "tmp1" ],
    "--path-tags"   : "top/d1/d2",
    "--profile"     : "hq"
}

Furthermore you can use an environment variable to pass this information

export BQ_OEM_TRANSFER=path-to-file.json