Skip to main content

File movement utility

Project description

The objective of BuffaloFQ is to provide a very simple and reliable method of moving files around an architecture.

Background, Examples, and FAQ

Please see the Wiki: https://github.com/kenfar/buffalofq/wiki

Installation:

$ pip install buffalofq

Configuration:

Most configuration is handled through a simple yaml file, that is kept in the xdg-compliant config directory. On linux this would be:

  • $HOME/.config/buffalofq_mover/[config-name1].yml
  • $HOME/.config/buffalofq_mover/[config-name2].yml
  • $HOME/.config/buffalofq_mover/[config-name3].yml

Config example:

  • name: ids_to_load #
  • status: enabled # choices are: enabled, disabled
  • polling_seconds: None # delay in seconds between checks for new files, defaults to 300
  • limit_total: -1 # choices: -1 (run continuously), 0 (run until source_dir is empty), [some number] run until this number have been moved.
  • port: None # defaults to 22
  • key_filename: None # defaults to id_buffalofq_rsa
  • log_dir: /data/logs # location buffalofq_mover will write its logs
  • log_level: None # choices are: info, warning, error, critical, defaults to debug
  • sort_key: time # choices are: None, name, or name of a field within filename, defaults to None
  • source_host: localhost # must be localhost at this time
  • source_user: None # not yet used, defaults to current userid
  • source_dir: /data/output #
  • source_fn: ‘*’ # wild-card for selecting source files
  • source_post_dir: /data/archive #
  • source_post_action: move # choices: move, delete, None
  • dest_host: datawarehouse #
  • dest_user: None # used to log into dest_host, defaults to current userid
  • dest_dir: /data/input #
  • dest_fn: None # needed if dest_post_action is symlink or move
  • dest_post_dir: None # not used yet
  • dest_post_action: None # choices: symlink, move, None

Run:

To run once, you can simply run it like:

  • $ ./buffalofq_mover –config-name [config-name1]

A trivial way to keep it running continuously:

  • $ nohup ./buffalofq_mover –config-name [config-name1] &

0.0.3 - add dest_post_action of move - change config values of pass to None - add config defaults & validation - housekeeping

0.0.2 - initial genuinely usable working version - add ability to receive files in order of their name, or any field within the name - add ability to provide name of ssh key to use - rename audit file & buffalofq program to buffalofq_mover - misc general improvements and refactorings

0.0.1 - initial working version

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
buffalofq-0.0.3.tar.gz (11.5 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page