ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.10.0-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5eaf0bde46aa72dc5e4a6cfe5c6f4da8529b260951d6f741885c71c0e6ebfbc0 |
|
MD5 | f70e6d3253ebf60f5fbf5c6d8024779b |
|
BLAKE2b-256 | e95961be4221033a810b3084a0cb846fdb05c23bb2ad9b5e47943737bd604026 |
Hashes for pyodps-0.10.0-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12a6c07597e606716369b2c9fec00e7eb9825c111cb7adaf9f74a7acbe4eba42 |
|
MD5 | a136cd57511e1159faf16f66a71f0f92 |
|
BLAKE2b-256 | 97c72f77c1d35f17cb2623299a1673f09366a7488ac3ca2f9bcccc3a45e15124 |
Hashes for pyodps-0.10.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89266de40658da19c57ee5675544713d59fb42c8a55bd047034c737d55948d37 |
|
MD5 | dcfd8d7b7c5719562f8cc21a02301495 |
|
BLAKE2b-256 | 90f81c8101e1d3fe86cad61dbe806e0cbee8be81197a5fcfb70ae303997316fe |
Hashes for pyodps-0.10.0-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7eaa4f353ca50b07a18d9f6998f811449eee5eea3f6e14fe3a7eb65ea5a97ca |
|
MD5 | 757f782c850554e4e631c6caee539bd6 |
|
BLAKE2b-256 | 37a26a85255debb5e32ccdfb607ef97fa405103356563ac2be7a5bbb151a6f05 |
Hashes for pyodps-0.10.0-cp37-cp37m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81c6b7207b06ce72ed3e2809af6ee322dcffdbf7a7c2c5bce1beb1b93de23f0a |
|
MD5 | d692311884dc8078eb0c57616719d5d5 |
|
BLAKE2b-256 | 7068a3099b0215dbd9913f890a8d159996ebf0fea99167d8bfb7e8a93b896a49 |
Hashes for pyodps-0.10.0-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6fb9e106eaa8a2e4f1244b18a911d0de01c2dd2a7c2a8831280ec79b3fec170b |
|
MD5 | 9d4c6aa49c8ba8a2179c59f92170be59 |
|
BLAKE2b-256 | 7587b72726e41b0d72211a3757a1d999529958a45a761d6e807f62a3de46bed4 |
Hashes for pyodps-0.10.0-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1f9780e37f333d7ea06512ebaf70e2b306e37c6500c7dc49dd927d116e393c0 |
|
MD5 | 3d076ba151104344aafe97baa2fc0cf7 |
|
BLAKE2b-256 | 0e8c5bd2804462e8fc70021a1bc3527b0475dd0310fd9bf2242f52c37382d2e9 |
Hashes for pyodps-0.10.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd9e40cdd1676580a0f0f0015e2f151053193a9e34e42330392dfd73e9b06b5b |
|
MD5 | 31d1b077024449857e150b7e1616339e |
|
BLAKE2b-256 | 05a4a11e02026071a688796010c4977751b4225519c1db705d97039fcc4efa23 |
Hashes for pyodps-0.10.0-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9307082c5203c812dffaf8e8f4d843e165d7f14f380a340b76b90867be7e8b3 |
|
MD5 | da4eebd79c9ddff88fbaf514a5fa5ae0 |
|
BLAKE2b-256 | 745c0563e02776c311ab193880335c8d6fd0b36e470bb533eb1ae141382f50bf |
Hashes for pyodps-0.10.0-cp36-cp36m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbb669b162b56cf89481ed8f5ed0b90422202ac7cd3882bd65b398e8e091095c |
|
MD5 | 870c0d6f227f5465f5054f3638fe0e1e |
|
BLAKE2b-256 | 94c6780ed9bc90ed5e237bd9dcaafb9766464d6835c5ee97a6756672a482ce5f |
Hashes for pyodps-0.10.0-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | db02f24541b651563b8d4bba935ae9df50f92c3e6c405359e41d0da72e05d633 |
|
MD5 | 029cffede45dce82474f12c2a676370f |
|
BLAKE2b-256 | a464fc0d5c6c4c05d11a733d3e5f6dfbba15a4303a62d195e1ea891e94633079 |
Hashes for pyodps-0.10.0-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d92520dda4c2ad6344ab66a085473ae1b8347bee8f6f47c1c0820f3633dd45c |
|
MD5 | 7ac79c3cf98b564091fe36e6b2ee7d21 |
|
BLAKE2b-256 | 9687911b09ffc7b1a88cd81c347ae2ad3119230deeb6653c76ca7fb6be24d298 |
Hashes for pyodps-0.10.0-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35bde4d302324c59ccd408432fc4234d413fd458c92c8cd5ec68acdcb80b1df3 |
|
MD5 | 78a16d99b76b6efabcf994b8df1668e9 |
|
BLAKE2b-256 | ae69dd11d0e2e3b437be599d075370db8503314de77c5fa6d4178efcbdc6acc5 |
Hashes for pyodps-0.10.0-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 743edef9a8e2d20271df5f6bf622c2f539388274e3b080d5679d0de7143add66 |
|
MD5 | 06c73213d48808beadf6cd7cb0d46526 |
|
BLAKE2b-256 | 041ea97709c636e41f297f6559acab934e30567b00b4046af909a5219220a97b |
Hashes for pyodps-0.10.0-cp35-cp35m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67d5151f9082543b6693d71d6342ac7a359dd99e9e417a1ba2d1ca742f0aa9b0 |
|
MD5 | a3c09b020e258d1264f4196ec078ae57 |
|
BLAKE2b-256 | e14774b9e1839208bc07308758a5224b7a2ef6376b11c33533756dbb99eca509 |
Hashes for pyodps-0.10.0-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e8d7b41d1044e5206da860a707cf0372f9caef180af7612441e0d9551dcac5d |
|
MD5 | e671592674548edc10eabbec15a65ed0 |
|
BLAKE2b-256 | 30c65f533df6db7325567a67409fe63086c7b1ebac841e6207f6fa5ea74c9e06 |
Hashes for pyodps-0.10.0-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b401f33d80cb69082df5810ef2d3c07390698ff8bd87a6805fad19d83600dc66 |
|
MD5 | 80ff376a615b355a828e1cc0bbd31dc8 |
|
BLAKE2b-256 | 5e04f9852ff337a5cc35e60255db64961080403b91eab73635fd0fdcd29e6ea6 |
Hashes for pyodps-0.10.0-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3d246c3e0f57a6227cfee470a8cad7e251195d49df9d08530d23570b368345f8 |
|
MD5 | 53e49f6250e1aa2c0467dfdc634c70e9 |
|
BLAKE2b-256 | c31a052e3ff270cf2a20e7380d2e7495866ae2d16d625aa8c68fce35ceb3cbc3 |
Hashes for pyodps-0.10.0-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea93179369ea56e76b93dd1a5e8f810ce222673fdca23cd97cb332d99f110a52 |
|
MD5 | 6c455c2e130004003faf2b0a2fe4f5e9 |
|
BLAKE2b-256 | 3291e6fb0e982997d081a725b32f26bef2a4dda8a94ca5b519595dfd83c6d1ce |
Hashes for pyodps-0.10.0-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 599d5d192124b9bb055a367fb53b79e1aaac0d08b588828b8f006b3226340b7e |
|
MD5 | 264b018ec4981918369553fd2275d494 |
|
BLAKE2b-256 | a6097b27f5695893227e11afd2c264dca452c192dcaf90e2617ad50083ab44f0 |
Hashes for pyodps-0.10.0-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69f60cb8deb16a1005003a89ea8a65e14d6624192519a03c941680410f76440d |
|
MD5 | 416cea9b618aab5b40782384c5282d1f |
|
BLAKE2b-256 | 84748dcfb80d6137a5eb5985df2c4a16d26b74c1cf54e566c3432a55418216e9 |