Skip to main content

tools collection for data engineer

Project description

This pacakge provides various tools to perform task on data, in easy and efficient manner; more modules could be added into the tools collection with development.

  1. universal way to connect most database softwares via JDBC (include kerberos auth for Hive), using Fast/Batch load technology to speed up the temporary table creation and query; as well as functions to convert clob into string or save the blob into specified file.

  2. add multiprocessing capablity to pandas dataframe when dealing with cpu intensive operation on large volume data.

  3. form based authentication module for requests package.

  4. restapi client using aiohttp package with retry function.

sample usage:

## connect to mysql
    import pydtc

    conn = pydtc.connect('mysql', '', 'user', 'pass')
    pydtc.read_sql('select * from demo.sample', conn)

### or use with clause for auto close
    with pydtc.connect('mysql', '', 'user', 'pass') as conn:
        conn.read_sql('select * from demo.sample')
        # pydtc.read_sql('select * from demo.sample', conn)

    ## DBAPI 2.0    
    with pydtc.connect_dbapi('mysql', '', 'user', 'pass') as conn:
        pd.read_sql('select * from demo.sample', conn)

## pandas multiprocessing groupby then apply
    def func(df, key, value):
        dd = {key : value}
        dd['some_key'] = [len(df.other_key)]

        return pd.DataFrame(dd)

    new_df = pydtc.p_groupby_apply(func, df, 'group_key')

## access web page in website with form based authenticaion
    from pydtc import HttpFormAuth
    import requests

    r = requests.get('', auth=HttpFormAuth('user', 'password'))

## restapi get and update
# Fake Online REST API for Testing and Prototyping
    from pydtc import api_get, api_update

    # or
    api_update('', data={'title': 'foo'}, method='patch')

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pydtc-0.6.1.tar.gz (10.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page