This pacakge provides various tools to perform task on data, in easy and efficient manner; more modules could be added into the tools collection with development.
-
universal way to connect most database softwares via JDBC (include kerberos auth for Hive), using Fast/Batch load technology to speed up the temporary table creation and query; as well as functions to convert clob into string or save the blob into specified file.
-
add multiprocessing capablity to pandas dataframe when dealing with cpu intensive operation on large volume data.
-
form based authentication module for requests package.
-
restapi client using aiohttp package with retry function.
sample usage:
## connect to mysql
import pydtc
conn = pydtc.connect('mysql', '127.0.0.1', 'user', 'pass')
pydtc.read_sql('select * from demo.sample', conn)
conn.close()
### or use with clause for auto close
with pydtc.connect('mysql', '127.0.0.1', 'user', 'pass') as conn:
conn.read_sql('select * from demo.sample')
# pydtc.read_sql('select * from demo.sample', conn)
## DBAPI 2.0
with pydtc.connect_dbapi('mysql', '127.0.0.1', 'user', 'pass') as conn:
pd.read_sql('select * from demo.sample', conn)
## pandas multiprocessing groupby then apply
def func(df, key, value):
dd = {key : value}
dd['some_key'] = [len(df.other_key)]
return pd.DataFrame(dd)
new_df = pydtc.p_groupby_apply(func, df, 'group_key')
## access web page in website with form based authenticaion
from pydtc import HttpFormAuth
import requests
r = requests.get('http://www.example.com/private_webpage.html', auth=HttpFormAuth('user', 'password'))
## restapi get and update
# Fake Online REST API for Testing and Prototyping
# https://jsonplaceholder.typicode.com/
from pydtc import api_get, api_update
api_get('https://jsonplaceholder.typicode.com/todos/1')
# or
api_update('https://jsonplaceholder.typicode.com/todos/1', data={'title': 'foo'}, method='patch')
1 maintainer
Metadata
1 maintainer
Release files for pydtc 0.8.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pydtc-0.8.4.tar.gz | 10.3 kB | Details |
Release files / pydtc-0.8.4.tar.gz
| Download URL | pydtc-0.8.4.tar.gz |
|---|---|
| Size | 10.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ce9cb0a3a948747ce91d0dfee80484ccbd6cd3f0dad6134e32784841172eaa90
|
|
BLAKE2b-256 checksum How to use checksums |
530aa02726305a0efef91769f480c0d32e33eef33dc40e4c2245d9e0783e6cc7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/45.2.0 requests-toolbelt/0.8.0 tqdm/4.30.0 CPython/3.8.10
|
