Function Parameters

CLI Options

Description

Copy files retrieved from an API to an S3 bucket.

Utilizes the AWS SDK for Python (Boto3) to add and backup API files as objects in an S3 bucket.

Credentials

Use a shared credentials file to export records to an S3 bucket.

Click here to learn about Boto3 shared credentials file, expected default location of file (~/.aws/credentials), supported configuration variables, and profiles.

Note: Be aware that environment variables will take precedence over a shared credentials file! Click here to learn more about the order that Boto3 searches for credentials.

The CLI will use the [default] settings in the shared credentials file unless the option --profile-name is used.

Example ~/.aws/credentials file:

[default]
aws_access_key_id=food
aws_secret_access_key=bard

If using the CLI option --profile-name option then a specific configuration profile can be used in the ~/.aws/credentials file:

[default]
aws_access_key_id=food
aws_secret_access_key=bard

[dev]
aws_access_key_id=snack
aws_secret_access_key=singer

Function Examples

# The s3_uploader argument must be passed as True to copy files as objects to an S3 bucket
# If s3_uploader is True then the endpoint_url argument must also be passed 

# Copies objects to an S3 bucket 
websnap(s3_uploader=True, endpoint_url="https://examplecloud.com")

# Copies objects to an S3 bucket and at maximum 4 backup objects are allowed for each 
# config section
websnap(s3_uploader=True, endpoint_url="https://examplecloud.com", backup_s3_count=4)

CLI Examples

Required Options

• The following CLI options must be used to enable websnap to upload files as objects in an S3 bucket:
  • --s3-uploader
  • --endpoint-url must be assigned to the complete URL to use for the constructed S3 client

Basic Usage

• Copies objects to an S3 bucket using default argument values

 websnap-cli --s3-uploader --endpoint-url "https://examplecloud.com"

Advanced Usage

• Copies objects to an S3 bucket
• Specify dev as the profile to use in the ~/.aws/credentials file
• At maximum 4 backup objects are allowed for each config section
• Enable logging to a file

 websnap-cli --s3-uploader --profile-name dev --endpoint-url "https://examplecloud.com" --backup-s3-count 4 --file-logs

Configuration

• A valid .ini or .json configuration file is required.
• Websnap expects the config to be config.ini in the same directory as websnap package is being executed from.
  • However, this can be changed using the config function argument (or CLI --config option).

S3 Configuration Example Files

S3 Configuration Sections (one per API URL endpoint)

• Each file retrieved from an API requires its own configuration section!
• The section name be anything, it is suggested to have a name that relates to the copied file.

Example S3 config section configuration with key prefix:

[resource]
url=https://www.example.com/api/resource
bucket=exampledata
key=subdirectory_resource/resource.xml

Example S3 config section configuration without key prefix:

[project]
url=https://www.example.com/api/project
bucket=exampledata
key=project.json

Copy files retrieved from an API to a local machine.

Examples

Function

# Write files retrieved from an API to local machine using default argument values
websnap()

# Write files retrieved from an API locally and file logs are enabled
websnap(file_logs=True)

CLI

• Write copied files to local machine using default argument values:

   websnap-cli 
  

• Write copied files locally and file logs are enabled:

   websnap-cli --file-logs
  

Configuration

• A valid .ini or .json configuration file is required for both function and CLI usage.
• Websnap expects the config to be config.ini in the same directory as websnap package is being executed from.
  • However, this can be changed using the config function argument (or CLI --config option).
• Each file that will be retrieved from an API requires its own section.
• If the optional directory key/value pair is omitted then the file will be written in the directory that the program is executed from.

Configuration Example Files

Sections (one per API URL endpoint)

Example local machine configuration section:

[project]
url=https://www.example.com/api/project
file_name=project.json
directory=projectdata

Websnap supports optional rotating file logs.

• The following CLI option must be used to enable websnap to support rotating file logs: --file-logs
  • In function usage the following argument must be passed to support rotating file logs: file_logs=True
• Log is called by default websnap.log
• If log keys are not specified in the configuration [DEFAULT] section then default values in the table below will be used.
• log_when expects a value used by logging module TimedRotatingFileHandler.
• Click here for more information about how to use TimedRotatingFileHandler.
• The default values result in the file logs being rotated once every day and no removal of backup log files.

Configuration

Example log configuration:

[DEFAULT]
log_when=midnight
log_interval=1
log_backup_count=7

[DEFAULT] Section

Websnap supports optionally specifying the minimum download size (in kilobytes) a file must be to copy it from the configured API URL endpoint.

• By default the minimum default minimum size is 0 kb.
  • Unless specified in the configuration this means that a file of any size can be downloaded by websnap.
• Configured minimum download size must be a non-negative integer.
• If the content from the API URL endpoint is less than the configured size:
  • An error will be logged and the program continues to the next config section.
  • If the CLI option --early-exit (or function argument early_exit=True) is enabled then the program will terminate early.

Configuration

Example minimum download size configuration:

[DEFAULT]
min_size_kb=1

[DEFAULT] Section