ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.7.9-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 64477463da711862db81e6c67f162e7ab4d2bc6472a0cf462d6c71634261d1b6 |
|
MD5 | 9255cb2382bc4e90ab208777781180c2 |
|
BLAKE2b-256 | 94afb2957adf0d08f97c7c7f420514cb12e71d51059d9a81c0266a1d6862cffd |
Hashes for pyodps-0.7.9-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 255e2c1a57bef2fb7a1bd8df612cb7fba8243f1e2b1e623037a22ac520683df3 |
|
MD5 | 33524f144da9ed78403e0eb93dc99360 |
|
BLAKE2b-256 | 0a20113cd8bbaa1a06555993e960b016cbaf07eb9733b5ffc15b209d8efa69e5 |
Hashes for pyodps-0.7.9-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8caeb467944177318a19036b17918964efbfba47dee15a8cbbd8b3e865afe7ce |
|
MD5 | 2c12eaa6997b8bc78e5acde93d8bc856 |
|
BLAKE2b-256 | 07e6a6ef51d2d98f99b17ea7c452847923c3b0cd2a34cd5a4086a746a06f0869 |
Hashes for pyodps-0.7.9-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a32772cc64d5aea443c79a117fe52ec20886511affa1f1cd12d6013ef7f820f |
|
MD5 | 81824de0ff063a3f4b21ed02541c037f |
|
BLAKE2b-256 | ae65b65759fab969440615b70df5d36bf6f1c6c430b714665bd06ab2210d7cb5 |
Hashes for pyodps-0.7.9-cp36-cp36m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c4d8ac91cec43da8e3a28180cf26879358cee8062ae151358e850eeabb1dbb8 |
|
MD5 | d165de95f0890f804d005ffda2aeb8ef |
|
BLAKE2b-256 | c8c89e6ad8995deb61ba43775e014618d1809126554a554069cb6587722f405f |
Hashes for pyodps-0.7.9-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a50c7c82b811897421278d3317b5ef637b6ad27f847a73dab8638a7bbb45cf6f |
|
MD5 | eda81579daa7fffdfbf7ad5b6f5c39e5 |
|
BLAKE2b-256 | 0edecd749383a68608f8ac7b04420fd6e56ced82a47c7af4efda3816ebd994f0 |
Hashes for pyodps-0.7.9-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ac691ccd61d2c378dae6ad95babb4090faf15a659d28128ed3949d8bbdbd2f1 |
|
MD5 | 17c76671eb051ea74cfc6f54e10bc177 |
|
BLAKE2b-256 | ad613b1ccfbaf817fb08978318a05b251c6440365bfcb9281a54e1e0e3101aba |
Hashes for pyodps-0.7.9-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e862c31e6d554f2e952a9fe56b42bd50bd77fa342205f78a83d0ebb90e6e7f16 |
|
MD5 | e590b5a8e10e014e74028cc671f19be2 |
|
BLAKE2b-256 | 7c2b87a9bbe71d7578a03ebcc39685a4de211b5a26a6c85f1145e75aa15be9ef |
Hashes for pyodps-0.7.9-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8204ae1eceeb2bde18e2d1213e6bfd7ec0b82109687b66441cf7a3de69cfdf91 |
|
MD5 | da29c113e47dd2723bde4f482ea29aff |
|
BLAKE2b-256 | 582d701c07d520599d9b6e537ef958a51401b8bab1ef8f96b8041c9ae39dea0b |
Hashes for pyodps-0.7.9-cp35-cp35m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eab369d2931ead8335ebe36e6b6b4483792fd7ed31714696cf1a6d8d63a35dc0 |
|
MD5 | 0832ee237a363a994f204231e633b8b1 |
|
BLAKE2b-256 | 92b6ba8596cbac0c9e2b6f9e6f138c90286f91deb20319c2ae2ee5a501a1e27a |
Hashes for pyodps-0.7.9-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d9a9fc18cd8611b9257a4ff99deac35ca38d55b63c598f9dbde59511a2c0deb |
|
MD5 | 610db4731b67159c99f867707eab8650 |
|
BLAKE2b-256 | 4b1364346c4eee16568d745fa601295a54cb4c7bd67fbba0cbe2c3dc1f94300f |
Hashes for pyodps-0.7.9-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8ca610d748ec98bee4ca47df7ebcd3ca0a6b7ab0ab8e8ffa9357b37495e8fc76 |
|
MD5 | 203aaaaebae8d13ef88fb3afd262ee01 |
|
BLAKE2b-256 | a13811e1ec7605647fddab41cc30cf2559dec040825564f85bc9d93709c6c8ee |
Hashes for pyodps-0.7.9-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7997a6d69eb91a4f640ced889e40c596bcbfc16874d65c5113ddfdb52b19cbeb |
|
MD5 | afd8e0a58121f0131ff2fd9425045b0a |
|
BLAKE2b-256 | 218cba3355f26398e14ed7659be64fa0a58f4b69104f8f1c257c53597f9e904e |
Hashes for pyodps-0.7.9-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec0d99b021c51d263a889c5069aeaf92abed79ab3c55dc503a18fb9518271454 |
|
MD5 | 4dfc540ba6366453b0882a41c3aec61d |
|
BLAKE2b-256 | d7c26f3aeb61317878cefd45dc1c4d0ad1f778d59e0889e8e9c764310db15c4a |
Hashes for pyodps-0.7.9-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a572d2ee36dc1138d3d3e7abfa7a6d879ee469d414c1b2c5deef129ab2dc798b |
|
MD5 | e9336829dc96f09788c3ec4c116fbb6a |
|
BLAKE2b-256 | 561f77aa329afa8f1003e68c84feb91cd33093396d35714a722f81c8275ed04a |
Hashes for pyodps-0.7.9-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc828b6c189d268c46fc2c6e0f43e0237fe6aeafebf537e3f6974c99ba45d232 |
|
MD5 | 947066781083e0d08b36214969d92466 |
|
BLAKE2b-256 | 5a8a2d949d869691b10391db6456d3a44b4a6bc94393df075c3a5484c09ddd3b |
Hashes for pyodps-0.7.9-cp27-cp27m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89cd7066c820784fd5f359dac45d49eea7c91c578f6bc36e21b9336b037382ad |
|
MD5 | 1100dce748da57e08bd3bd3fb9c1c0c0 |
|
BLAKE2b-256 | 9ad009c183fedf0e05264c3ac0b4acddffc1c4941d29e448c5a133aaf4bf0a4b |