Virt-S3 🪣

A Virtualitics utility package to handle file I/O with Object Storage Systems like AWS S3 and Minio.

With versatility in mind, virt-s3 was designed to be a relatively lightweight package that can either used independently or in conjunction with the larger Virtualitics AI platform. The virt-s3 module includes two primary submodules s3 and fs that implement each API function of the virt-s3 module specific to the target system: either S3/S3-like systems or local file systems.

We hope that you can use it, break it, and even help us improve it!

Table of Contents

1. Prerequisites
2. Example Usage
3. Architecture
4. Getting Started
5. Documentation

Prerequisites

• Requires python>=3.11
• Local File System features currently only support posix / pathing (Linux, Mac, etc.)
  • Support for Windows \ pathing [Coming Soon]

Example Usage

Creating a Bucket

import virt_s3

# ENV variable `Local_FS` = '1' or '0' (local file system or S3)
params = virt_s3.get_default_params()

# use context manager to manage session scope
with virt_s3.SessionManager(params=params) as session:
    virt_s3.create_bucket('test-bucket', params=params, client=session)

Uploading a File

import virt_s3
import pandas as pd

# ENV variable `Local_FS` = '1' or '0' (local file system or S3)
params = virt_s3.get_default_params()

# path to locally saved csv file
fpath = "/tmp/data.csv"

# use context manager to manage session scope
with virt_s3.SessionManager(params=params) as session:
    s3_key = f"fixture/data/data.csv"
    virt_s3.upload_data(fpath, s3_key, params=params, client=session)

Reading a File

import virt_s3
import pandas as pd

# ENV variable `Local_FS` = '1' or '0' (Local file system or S3)
params = virt_s3.get_default_params()

# use context manager to manage session scope
with virt_s3.SessionManager(params=params) as session:
    data = virt_s3.get_file(saved_key, bytes_io=True, params=params, client=session)
    df = pd.read_csv(data)

Architecture

virt-S3 can be run on a local machine or from within a docker container. Additionally, it includes a variety of ways to interact with Object Storage Systems like AWS S3 and Minio in different hosting environments along with support for local file system access on host machine within docker container.

This versatility along with its lightweight set of dependencies allows virt-s3 to be easily installed and used in various types of environments.

Getting Started

1. Create a fresh virtual environment with python >= 3.11

2. Install the necessary dependencies

Basic Install (No Extras)

$ pip install virt-s3

Install with Single Extra

$ pip install "virt-s3[s3]"

Install with Multiple Extras

$ pip install "virt-s3[s3,dataframe,image]"

• The Following Extras are Available:

  • s3 = installs dependencies required to interact with object stores like Minio/S3 (primarily relying on boto3)
  • dataframe = installs dependencies required for using numpy, pandas, and pyarrow dataframe/parquet operations
  • image = installs dependencies required to utilize image operations (e.g. get file as an image)

• e.g. If you want to use virt_s3, but can't install pandas or pyarrow in your restricted environment, then you can simply install virt_s3 without the dataframe extra dependencies. You won't be able to use virt_s3.extras.CSVFileValidator, virt_s3.extras.ParquetFileValidator, read_parquet_file_df, and write_parquet_file_df but these are also not necessarily core functions of the library (therefore extras).

3. Make sure the following environment variables are set

## Local File System Environment Variables
LOCAL_FS_USER=<your username>
LOCAL_FS=0   # use the local fs mirror or s3/minio: 1 = True, 0 = False
LOCAL_FS_ROOT_DIR=</path/to/your/data/dir/>

## S3 Environment Variables
S3_URL=<your s3/minio url>  # e.g. http://mock-s3:9000 or http://localhost:9000
S3_DEFAULT_BUCKET=test-buck<your bucket name>
AWS_SECRET_ACCESS_KEY=<your aws secret access key>
AWS_ACCESS_KEY_ID=<your aws access key id>
AWS_REGION=<your aws region>  # e.g. us-east-1

• Note: S3_URL can be replaced with a localhost url (e.g. http://localhost:9000) if not being run within a docker container

4. Run the above example usage

Code Documentation

• Full Module Table of Contents
• Full Module API Specs