Skip to main content

Monitor files by wildcard patterns, upload them to ALIYUN OSS, then move to backup directory.

Project description

Source:https://github.com/tanbro/aliyunoss2-autoupload
Package:https://pypi.org/project/aliyunoss2-autoupload/

Circle CI Codacy Codacy Coverage PYPI Version PYPI License PYPI Python Versions PYPI Status PYPI Format

aliyunoss2-autoupload

Monitor files by wildcard patterns, upload them to ALIYUN OSS, then move to backup directory.

Usage

Command Line

After the package installed, run the command in a terminator, show help messages

$ aliyunoss2-autoupload --help

usage: aliyunoss2-autoupload [-h] [--version] {run,echo_config_sample} ...

Watch files in a directory and upload them to Aliyun OSS on file writing
completed

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

subcommands:

  {run,echo_config_sample}
                        <sub_command --help> Print the help of sub_commands
    run                 Start to run then program. It will monitor and upload
                        files continuously.
    echo_config_sample  Echo configure file sample
$ aliyunoss2-autoupload run --help

usage: aliyunoss2-autoupload run [-h] [--only-once]
                                 [--config-file CONFIG_FILE]
                                 [--logging-config-file LOGGING_CONFIG_FILE]

optional arguments:
  -h, --help            show this help message and exit
  --only-once, -o       Upload only once, then exit. Will NOT monitor files.
                        (default=False)
  --config-file CONFIG_FILE, -c CONFIG_FILE
                        The program configuration file. The program will first
                        try to load configuration file by environment variable
                        ${ALIYUNOSS2_AUTOUPLOAD_CONF}. If the environment
                        variable not assigned, then try to load configuration
                        file "conf/aliyunoss2-autoupload.yml"
  --logging-config-file LOGGING_CONFIG_FILE, -l LOGGING_CONFIG_FILE
                        The logging configuration file. The program will first
                        try to load logging configuration file by environment
                        variable ${ALIYUNOSS2_AUTOUPLOAD_LOG_CONF}. If the
                        environment variable not assigned, then try to load
                        configuration file
                        "conf/aliyunoss2-autoupload.log.yml"
$ aliyunoss2-autoupload echo_config_sample --help

usage: aliyunoss2-autoupload echo_config_sample [-h] {prog,log}

positional arguments:
{prog,log}  Configure file to echo

optional arguments:
-h, --help  show this help message and exit

Configuration File

The program will first try to load configuration file from environment variable ALIYUNOSS2_AUTOUPLOAD_CONF. If the environment variable not assigned, then try to load configuration file “conf/aliyunoss2-autoupload.yml”.

The YAML file is like blow:

---

## Aliyun OSS configs
oss:
    ## Name of your Aliyun OSS bucket
    name: "your_bucket_name"
    ## Endpoint URL of Aliyun OSS bucket
    endpoint: "oss-xx-xxxxxx.aliyuncs.com"
    ## cname of the domain of Aliyun OSS bucket. Empty if no cname.
    cname: ""
    ## Access Key ID of Aliyun OSS bucket
    access_key_id: "your_access_key_id"
    ## Access Key Secret of Aliyun OSS bucket
    access_key_secret: "your_access_key_secret"

## Directory name configs
dir:
    ## Calculate uploading file relative name by this local directory
    rel_dir: ""
    ## Upload files to OSS in this dir
    oss_dir: ""
    ## Move uploaded file the the directory. It MUST be a different dir from where the files are. If not, the file will be uploaded again and again.
    bak_dir: ""

## watcher configs
watcher:
    ## The time interval(seconds) this program scan the directory
    interval: 30
    ## If the interval between the current time and the file\'s modification time is greater than this value, the write is considered complete.
    write_complete_time: 30
    ## Pattern of the files to watch and upload
    patterns: "files/*.*"
    ## If find patterns recursively
    recursive: false
    ## pool of at most max_workers threads to execute upload/backup tasks. If max_workers is None or not given, it will default to the number of processors on the machine, multiplied by 5.
    max_workers: ~

Also, the program will first try to load logging configuration file by environment variable ALIYUNOSS2_AUTOUPLOAD_LOG_CONF. If the environment variable not assigned, then try to load configuration file “conf/aliyunoss2-autoupload.log.yml”.

Logging config file is also YAML. Go to https://docs.python.org/3/library/logging.config.html for more information about Python logging config.



Install

Install by pip

pip install aliyunoss2-autoupload

Install from code

git clone https://github.com/tanbro/aliyunoss2-autoupload.git
cd aliyunoss2-autoupload
path/of/your/python setup.py install

Changelog

0.1

Added
  • Reload configuration every time before scanning files.
  • –only-once command line option.
Optimized
  • test case for main() function

0.1b2

Date:2018-06-20
Adds
  • More detailed documents.
Bug Fixes:
  • Remove files in .gitignore, but tracked.

0.1b1

Date:2018-06-19
Changes
  • Support old Python 2.7, Python 3.4
  • Config file name extension changed from ".yaml" to ".yml"
  • Default config file environment variable name
  • Default config file path if no environment variable
Adds
  • Some simple test cases
  • CircleCI
  • Codacy

0.1a3

The first ever-usable version.

Date:2018-04-18
Changes
  • Now Python 3.5+ ONLY, because glob.iglob has no recursive argument and "**" wildcard in lower Python version
Bug Fixes
  • Backup directory errors
  • OSS exception

CONTRIBUTING

Contributions are welcome!

It’s advised to develop in venv

/path/of/your/python -m venv .venv
source .venv/bin/activate
git clone git@github.com:tanbro/aliyunoss2-autoupload.git
cd aliyunoss2-autoupload
python setup.py develop

Authors

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
aliyunoss2_autoupload-0.1-py2.py3-none-any.whl (15.6 kB) Copy SHA256 hash SHA256 Wheel py2.py3
aliyunoss2-autoupload-0.1.tar.gz (31.9 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page