Skip to main content

Python utilities used for interacting with Minio Object Storage

Project description

Christopher H. Todd's PROJECT_STRING_NAME

The PROJECT_GIT_NAME project is responsible for ...

The library ...

Table of Contents

Dependencies

Python Packages

Libraries

minio_bucket_helpers.py

This library is used to interact with Minio object storage. Will handle functions used to interact with buckets (creating, downloading, finding, etc)

Functions:

def get_buckets(minio_client):
    """
    Purpose:
        Get a list of buckets that exist in the Minio Client
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
    Returns:
        buckets (List of Bucket Objs): List of Bucket OBJs in Minio
    """
def get_bucket_names(minio_client):
    """
    Purpose:
        Get a list of buckets that exist in the Minio Client
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
    Returns:
        bucket_names (List of Strings): List of Buckets in Minio
    """

def create_bucket(minio_client, bucket_name):
    """
    Purpose:
        Create a specified Bucket by name
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of bucket to create
    Returns:
        N/A
    """
def delete_bucket(minio_client, bucket_name):
    """
    Purpose:
        Delete a specified Bucket by name
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of bucket to delete
    Returns:
        N/A
    """

minio_client.py

MinioClient Class for interacting with minio object store. Objects will be created connected to Minio

Classes:

class MinioClient(object):
    """
        MinioClient Class. Class objects will hold connection to the
        Minio service and can be used to interact with buckets and objects
    """

minio_connection_helpers.py

This library is used to interact with Minio object storage. Functions establish a connection to the Minio service that can be used to interact with the service and pass to the other helper functions

Functions:

def connect_to_minio(minio_url, access_key=None, secret_key=None, secure=False):
    """
    Purpose:
        Connect to Minio and return the minio_client of minio lib
    Args:
        minio_url (String): URL of Minio
        access_key (String): Access Key for Minio
        secret_key (String): Secret Key for Minio
    Returns:
        minio_client (minio client Obj): Client obj connection to Minio
    """
def build_minio_url(minio_host, minio_port=9000):
    """
    Purpose:
        Create the Minio URL from host and port
    Args:
        minio_host (String): Host of Minio
        minio_host (Int): Port of Minio (Defaults to 9000)
    Returns:
        minio_url (String): URL of Minio
    """

minio_exceptions.py

File for holding custom exception types that will be generated by the minio_helpers libraries

Exceptions:

class BucketAlreadyExists(Exception):
    """
    Purpose:
        The BucketAlreadyExists will be raised when attempting to create a bucket
        that already exists
    """
class BucketDoesntExist(Exception):
    """
    Purpose:
        The BucketDoesntExist will be raised when attempting to delete a bucket
        that doesn't exist in Minio
    """
class ObjectAlreadyExists(Exception):
    """
    Purpose:
        The ObjectAlreadyExists will be raised when attempting to create an object
        that already exists
    """
class ObjectDoesntExist(Exception):
    """
    Purpose:
        The ObjectDoesntExist will be raised when attempting to pull an object
        that doesn't exist in Minio
    """
class ObjectDecodingNotSupported(Exception):
    """
    Purpose:
        The ObjectDecodingNotSupported will be raised when attempting to decode a
        filetype from Minio that is not yet supported (such as .avro)
    """

minio_general_helpers.py

This library is used to interact with Minio object storage.

Functions:

N/A

minio_object_helpers.py

Example executable Python scripts/modules for testing and interacting with the library. These show example use-cases for the libraries and can be used as templates for developing with the libraries or to use as one-off development efforts.

Functions:

def get_objects(minio_client, bucket_name):
    """
    Purpose:
        Get a list of objects that exist in the Minio Client
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of the bucket to get objects for
    Returns:
        objects (List of Object Objs): List of Object OBJs in Minio
    """
def get_object_names(minio_client, bucket_name):
    """
    Purpose:
        Get a list of objects that exist in the Minio Client
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of the bucket to get objects for
    Returns:
        object_names (List of Strings): List of Objects in Minio
    """
def is_object_in_bucket(minio_client, bucket_name, object_name):
    """
    Purpose:
        Check if Object exists in Bucket
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of the bucket to check for object
        object_name (String): Name of object to check for in Minio
    Returns:
        object_exists (Boolean): Boolean if the object exists or not
    """
def get_object_stats(minio_client, bucket_name, object_name):
    """
    Purpose:
        Get Stats of the Object
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of the bucket to check for object
        object_name (String): Name of object to get stats for in Minio
    Returns:
        object_stats (Dict):Dict of stats about the object
    """
def download_object_to_memory(minio_client, bucket_name, object_name, encoding="utf-8"):
    """
    Purpose:
        Download an Object from Mino into memory (if supported)
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of the bucket to get object from
        filename (String): Location (And Path) of file to upload
        object_name (String): Name of object to upload in Minio
    Returns:
        parsed_object (Obj, depending on extension): Object parsed from Minio from the
            extension of the file. Current supported = .txt -> str, .json -> Dict/JSON
    """
def download_object_to_file(minio_client, bucket_name, object_name, filename=None):
    """
    Purpose:
        Download a file from Minio to local storage
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of the bucket to get object from
        filename (String): Location (And Path) of file to upload
        object_name (String): Name of object to upload in Minio
    Returns:
        N/A
    """
def upload_object(minio_client, bucket_name, filename, object_name=None):
    """
    Purpose:
        Uploading a local file to Minio
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        bucket_name (String): Name of the bucket to get to upload object to
        filename (String): Location (And Path) of file to upload
        object_name (String): Name of object to upload in Minio
    Returns:
        N/A
    """
def delete_object(minio_client, bucket_name, object_name):
    """
    Purpose:
        Delete a specified Object by name
    Args:
        minio_client (minio client Obj): Client obj connection to Minio
        object_name (String): Name of object in Minio to delete
    Returns:
        N/A
    """

Example Scripts

Example executable Python scripts/modules for testing and interacting with the library. These show example use-cases for the libraries and can be used as templates for developing with the libraries or to use as one-off development efforts.

connect_to_minio.py

Purpose:
    Connecting to Minio

Steps:
    - Connect to Minio

function call:python3 connect_to_minio {--access-key=access_key} \
    {--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port}

create_bucket_in_minio.py

Purpose:
    Create Bucket in Minio

Steps:
    - Connect to Minio
    - Create a Bucket

function call:python3 create_bucket_in_minio.py {--access-key=access_key} \
    {--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port} \
    {--bucket-name=bucket_name}

delete_bucket_from_minio.py

Purpose:
    Delete Bucket from Minio

Steps:
    - Connect to Minio
    - Delete a Bucket

function call:python3 delete_bucket_from_minio.py {--access-key=access_key} \
    {--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port} \
    {--bucket-name=bucket_name}

get_objects_from_bucket.py

Purpose:
    Get all Objects (Or a Specific Object) From a Bucket in Minio

Steps:
    - Connect to Minio
    - Get the Bucket Obj
    - Get objects from Minio and store locally

function call:python3 get_objects_from_Bucket.py {--access-key=access_key} \
    {--secret-key=secret_key} {--minio-host=minio_host} {--minio-port=minio_port} \
    {--bucket-name=bucket_name} {--object-name=object_name}

Notes

  • Relies on f-string notation, which is limited to Python3.6. A refactor to remove these could allow for development with Python3.0.x through 3.5.x

TODO

  • Unittest framework in place, but lacking tests

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ctodd-python-lib-minio, version 1.0.2
Filename, size File type Python version Upload date Hashes
Filename, size ctodd_python_lib_minio-1.0.2-py3-none-any.whl (12.8 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size ctodd-python-lib-minio-1.0.2.tar.gz (10.4 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page