Codenize your data sources
Project description
akagi
Free software: MIT license
Features
akagi supports iter and save interface for various data sources such as Amazon Redshift, Amazon S3 (more in future).
Installation
Install via pip:
pip install akagi
or from source:
$ git clone https://github.com/ayemos/akagi akagi $ cd akagi $ python setup.py install
Example
MySQLDataSource
with MySQLDataSource.for_query(
'select * from (select user_id, path from logs.imp limit 10000)', # Your Query here
) as ds:
ds.save('./akagi_test') # save results to local
for d in ds:
print(d) # iterate on result
RedshiftDataSource
with RedshiftDataSource.for_query(
'select * from (select user_id, path from logs.imp limit 10000)', # Your Query here
'logs', # schema
'imp', # table (Those two are used to generate unique prefix for S3 object (e.g. logs/imp/20170312_081527)
'log-redshift-unload.ap-northeast-1', # S3 Bucket for intermediate storage
) as ds:
ds.save('./akagi_test') # save results to local
for d in ds:
print(d) # iterate on result
S3DataSource
with S3DataSource.for_prefix(
'image-data.ap-northeast-1',
'data/image_net/zebra',
FileFormat.BINARY) as ds:
...
LocalDataSource
with LocalDataSource.for_path(
'./PATH/TO/YOUR/DATA/DIR',
'csv') as ds:
ds.save('./akagi_test') # save results to local
for d in ds:
print(d) # iterate on result
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
akagi-0.1.14.tar.gz
(7.7 kB
view hashes)
Built Distribution
akagi-0.1.14-py2.py3-none-any.whl
(15.6 kB
view hashes)
Close
Hashes for akagi-0.1.14-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b35114278ceac90b4afbc20bc6a063e6071fd304e5379f029cdace599a9dacb1 |
|
MD5 | 77228584f5cdc2dc8bf29ac3fc8d5c30 |
|
BLAKE2b-256 | a4e36cf0e1f60ebe27107968e6f2e8a5548b7e6f5c5a5108d22e276af5cd5f24 |