manonaid-helpers 0.1.4

A small example package

Installation

pip install manonaid_helpers

See PyPi for package index.

Requirements

Azure Storage connection string has to be set as environment variable AZURE_STORAGE_CONNECTION_STRING or the seperate environment variables AZURE_STORAGE_KEY and AZURE_STORAGE_NAME which will be used to create the connection string.

Set variables like below: For Windows PowerShell $env:AZURE_STORAGE_CONNECTION_STRING = 'blahblahblah....'

For Command Prompt set AZURE_STORAGE_CONNECTION_STRING=blahblahblah....

Usage

Download

1. Fetch as pd.DataFrame

We can use the returnAsDataFrameDict() method to fetch one or multiple blobs as DataFrames

While fetching multiple blobs the return object is a dictonary of DataFrames with the blob name(with the extension) as the key.

If type_correct_date_time_cols is set as True: For all blobs that contain columns with names containing "date" or "time" are converted to datetime with format="%Y-%m-%dT%H:%M:%S"

delimitter is set to',' by default

from manonaid_helpers.download import Download

df = Download(
    source="cyi5", 
    extension=".csv", 
    folder="pack_test/"
).returnAsDataFrameDict()

2. Download a specific folder from a container

We can download a folder by setting the folder argument.

from manonaid_helpers.download import Download

Download(
   source='cyi5',
   folder="pack_test/",
   extension='.csv',
).returnAsDataFrameDict()

3. Download a given list of files

We can give a list of files to download with the list_files argument.

from manonaid_helpers.download import Download

Download(
   source='cyi5',
   folder="pack_test/",
   extension='.csv',
   list_files=["20230301_ais_accuracy.csv"],
).returnAsDataFrameDict()

List blobs

With the Utils.list_blobs method we can do advanced listing of blobs in a container or specific folder in a container. We have several argument we can use to define our scope of information:

• name_starts_with: This can be used to filter files with certain prefix, or to select certain folders: name_starts_with=folder1/subfolder/lastfolder/