datup 0.0.2

The version of this library and document is V 0.0.2

datup

The version of this library and document is V 0.0.2 This library has 3 methods and 1 Class

How is it work?

import datup as dt

To Instance the Class

job = dt.DataIO("aws_acces_key_id","aws_secret_access_key","datalake")

Can I test my updates?

Yes, there is a file called _testing.ipynb where you can test your changes. The variables, must be initialized always for modularity.

Class DataIO:

A group of methods for I/O data from a AWS S3 Datalake

Parameters
----------
aws_acces_key_id : str
    The class must be intialized with aws_acces_key_id
aws_secret_access_key : str
    The class must be intialized with aws_secret_access_key
datalake : str
    The class must be intialized with the name of the datalake
prefix_s3 : str, default "s3://"
    The s3 prefix used to denote a s3 address
local_path : str, default "/tmp/"
    The local path used by boto3 for upload from Temp folder to S3 bucket.


Methdos
-------
download_csv(self,stage=None,filename=None,datecols=False,sep=",",encoding="ISO-8859-1",infer_datetime_format=True,low_memory=False,indexcol=None,ts_csv=False,freq=None)

        Return a dataframe downloaded from a specified datalake

        This function takes the aws credentials from DataIO class and use it for download 
        the required data.

        Parameters
        ----------
        stage : str, default None
            It is the set of folders after the datalake to the file that is required to download 
        filename : str, default None
            Is the name of the filename to download without csv suffix
        datecols: bool or list of int or names or list of lists or dict, default False
            Took it from Pandas read_csv parse_dates description
            The behavior is as follows:
                * boolean. If True -> try parsing the index.
                * list of int or names. e.g. If [1, 2, 3] -> try parsing columns 1, 2, 3 each as a separate date column.
                * list of lists. e.g. If [[1, 3]] -> combine columns 1 and 3 and parse as a single date column.
                * dict, e.g. {‘foo’ : [1, 3]} -> parse columns 1, 3 as date and call result ‘foo’
                If a column or index cannot be represented as an array of datetimes, say because of an unparseable value
                or a mixture of timezones, the column or index will be returned unaltered as an object data type. For
                non-standard datetime parsing, use pd.to_datetime after pd.read_csv. To parse an index or column with
                a mixture of timezones, specify date_parser to be a partially-applied pandas.to_datetime() with
                utc=True. See Parsing a CSV with mixed timezones for more.
                Note: A fast-path exists for iso8601-formatted dates.
        sep: str, default ‘,’
            Took it from Pandas read_csv sep description
            Delimiter to use. If sep is None, the C engine cannot automatically detect the separator, but the Python
            parsing engine can, meaning the latter will be used and automatically detect the separator by Python’s
            builtin sniffer tool, csv.Sniffer. In addition, separators longer than 1 character and different from '\s+'
            will be interpreted as regular expressions and will also force the use of the Python parsing engine. Note
            that regex delimiters are prone to ignoring quoted data. Regex example: '\r\t'.
        encoding: str, default ‘ISO-8859-1’
            Took it from Pandas read_csv encoding description
            Encoding to use for UTF when reading/writing (ex. ‘utf-8’).
        infer_datetime_format: bool, default True
            Took it from Pandas read_csv infer_datetime_format description
            If True and parse_dates is enabled, pandas will attempt to infer the format of the datetime strings in the
            columns, and if it can be inferred, switch to a faster method of parsing them. In some cases this can
            increase the parsing speed by 5-10x.
        low_memory: bool, default True
            Took it from Pandas read_csv low_memory description
            Internally process the file in chunks, resulting in lower memory use while parsing, but possibly mixed
            type inference. To ensure no mixed types either set False, or specify the type with the dtype parameter.
            Note that the entire file is read into a single DataFrame regardless, use the chunksize or iterator
            parameter to return the data in chunks. (Only valid with C parser).
        indexcol : int, str, sequence of int / str, or False, default None
            Took it from Pandas read_csv index_col description
            Column(s) to use as the row labels of the DataFrame, either given as string name or column index. If a
            sequence of int / str is given, a MultiIndex is used. 
            Note: index_col=False can be used to force pandas to not use the first column as the index, e.g. when
            you have a malformed file with delimiters at the end of each line.
        ts_csv : bool, default False
            If ts_csv is True then activates the ts_csv downloaded. If True, indexcol and freq, both must
            be different to None
        freq : str, default W-MON
            Is the frequency of getting data

        Returns
        -------
        DataFrame
            A DataFrame is return as two-dimensional data structure

        Examples
        --------
        >>> object.download_csv(stage='stage',filename='filename')  # doctest: +SKIP

download_csvm(self,uris)

    Return a set of dataframes downloaded from a specified datalake in a list

    This function takes the aws credentials from DataIO class and use it for download 
    the required data through download_csv method.

    Parameters
    ----------
    uris : dict
        Is the dictionary with all parameters necessary for the download_csv method

    Returns
    -------
    List of DataFrames
        A list of DataFrames are returned
    List of DataFrames Names
        A list of DataFrames Names are return in string type

    Examples
    --------
    >>> uris = {
            "uri_1":{
                "stage":"stage",
                "filename":"filename",
                "datecols":False,
                "sep":";",
                "encoding":"ISO-8859-1"
            },
            "uri_2":{
                "stage":"stage",
                "filename":"filename",
                "datecols":False,
                "sep":";",
                "encoding":"ISO-8859-1"
            }
        }
    >>> df, df_names =object.download_csvm(uris=uris)  # doctest: +SKIP

upload_csv(self,df,stage=None,filename=None,index=False,header=True,date_format="%Y-%m-%d",ts_csv=False)

    Return a uri where the dataframes was uploaded in csv format 

    This function takes the aws credentials from DataIO class and use it for upload
    the required data.

    Parameters
    ----------
    df : DataFrame
        Is the DataFrame for uploading to S3 datalake
    stage : str, default None
        It is the set of folders after the datalake to the file that is required to upload 
    filename : str, default None
        Is the name of the filename to upload without csv suffix
    index : bool, default False
        Took it from Pandas to_csv index description
        Write row names (index).
    header : bool or list of str, default True
        Took it from Pandas to_csv index description
        Write out the column names. If a list of strings is given it is assumed to be aliases for
        the column names.
    date_format : str, default %Y-%m-%d
        Took it from Pandas to_csv index description
        Format string for datetime objects.
    ts_csv : bool, default False
        If ts_csv is True then activates the ts_csv upload. If True, index must be different to False

    Returns
    -------
    Str
        The uri where DataFrame was uploaded into S3

    Examples
    --------
    >>> object.upload_csv(df,stage="stage",filename="filename")  # doctest: +SKIP