multiriver 1.1.4

A tool for bulk river creation

Azure Blob Storage and Rivery CLI Tool

A command-line interface (CLI) tool to interact with Azure Blob Storage and manage rivers in Rivery. This tool allows you to configure Azure and Rivery credentials, generate river templates based on blob names, populate rivers, delete rivers, and list source configurations.

Table of Contents

• Prerequisites
• Installation
• Configuration
  • Configure Credentials
  • Configure Source
• Commands
  • populate_rivers
  • delete_rivers
  • list_source_configs
• Usage Examples
  • Example Workflow
• License
• Contact

Prerequisites

• Python 3.6 or higher
• Azure Blob Storage account
• Rivery account and API token
• Click library (pip install click)
• Other dependencies as required by the code (e.g., rich for table display)
• Virtual Environment: It is recommended to use a virtual environment to manage dependencies.

Installation

1. Create a Virtual Environment

   It is recommended to create a virtual environment to isolate the dependencies.

   python3 -m venv venv
   source venv/bin/activate
   

2. Install the CLI Tool

   Install the multiriver package using pip:

   pip install multiriver
   

   (Ensure that the package multiriver is available in the Python Package Index or replace this command with the appropriate installation method if installing from a local source.)

Configuration

Before using the CLI tool, you need to configure your Azure and Rivery credentials, as well as source settings.

Configure Credentials

Use the configure_creds command to set up your Azure Storage and Rivery API credentials.

multiriver configure_creds \
    --account-name YOUR_AZURE_ACCOUNT_NAME \
    --account-key YOUR_AZURE_ACCOUNT_KEY \
    --rivery-api-token YOUR_RIVERY_API_TOKEN

Options:

• --account-name (optional): Azure Storage account name.
• --account-key (optional): Azure Storage account key.
• --connection-string (optional): Azure Storage connection string.
• --sas-token (optional): Shared Access Signature (SAS) token.
• --rivery-api-token (optional): Rivery API token.
• --rivery-host (optional): Rivery host URL (default: https://console.rivery.io).

Note: You must provide at least one Azure credential and the Rivery API token.

Configure Source

Use the configure_source command to set up source settings for Azure Blob Storage.

multiriver configure_source \
    --container-name YOUR_CONTAINER_NAME \
    --prefix YOUR_BLOB_PREFIX \
    --template-river-id TEMPLATE_RIVER_ID \
    --filename-template FILENAME_TEMPLATE \
    --group-id GROUP_ID \
    --cron-schedule CRON_SCHEDULE

Options:

• --container-name (optional): Name of the Azure Blob Storage container.
• --prefix (optional): Prefix to filter blobs.
• --template-river-id (optional): ID of the base template river in Rivery.
• --filename-template (optional): Template used for generating river names.
• --group-id (required): ID of the group to attach the rivers to in Rivery.
• --cron-schedule (optional): Cron expression to schedule new rivers.

Note: You must provide the --group-id option. If --filename-template is provided, river templates will be generated based on the blob names.

Commands

populate_rivers

Populate rivers in Rivery based on the generated templates.

multiriver populate_rivers --group-id GROUP_ID

Options:

• --group-id (required): ID of the group to attach the rivers to in Rivery.

delete_rivers

Delete all rivers associated with a specific group in Rivery.

multiriver delete_rivers --group-id GROUP_ID

Options:

• --group-id (required): ID of the group whose rivers will be deleted.

list_source_configs

List all source configurations that have been set up.

multiriver list_source_configs

Usage Examples

Example Workflow

1. Create a Virtual Environment

   python3 -m venv venv
   source venv/bin/activate
   

2. Install the CLI Tool

   pip install multiriver
   

3. Configure Credentials

   multiriver configure_creds \
       --account-name myazureaccount \
       --account-key myazurekey \
       --rivery-api-token myriverytoken
   

4. Configure Source

   multiriver configure_source \
       --container-name mycontainer \
       --prefix data/ \
       --template-river-id 123456789 \
       --filename-template "{entity_name}_data_{date}.csv" \
       --group-id 987654321 \
       --cron-schedule "0 0 * * *"
   

5. Populate Rivers

   multiriver populate_rivers --group-id 987654321
   

   This command will:

   • Generate river templates based on the blobs in the specified container and prefix.
   • Create new rivers in Rivery using the templates.
   • Schedule the rivers if a cron schedule was provided.

6. Delete Rivers

   If you need to delete all rivers associated with the group:

   multiriver delete_rivers --group-id 987654321
   

7. List Source Configurations

   To view all source configurations:

   multiriver list_source_configs
   

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

For any questions or issues, please open an issue on the GitHub repository or contact the maintainer.