ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.7.20.1-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 410678d8a89ec8a8860e1f533304e3ca9a78a8116cee11d15305dcaf7be05539 |
|
MD5 | 2d710e8ccd155a12a388cb8364ee29b7 |
|
BLAKE2b-256 | 8174fab4d201aa5591f3ef54b6ed862a7d459ac2d79200b098b906073de2b174 |
Hashes for pyodps-0.7.20.1-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a992199792748e2be913e59e300e4900cf9e66081b7305ba3dbe6b25d0514a4a |
|
MD5 | b9a2a3d5826aa6842ff23c4b9195aba8 |
|
BLAKE2b-256 | 391397f7c30e5f91dc153a05f4f756fe4c1c775eeef85011dc2e0633a6d29cb5 |
Hashes for pyodps-0.7.20.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c074b9e4a8a0f0231be3d7d60f2a251ac2683778d414a54ec56ce0421b79ff3 |
|
MD5 | c3d81979cd9e28436b1e4d56c4994e23 |
|
BLAKE2b-256 | 4ea2318623b09db245d193717cd28822e19c4fc246c9968fece159bd2abedddc |
Hashes for pyodps-0.7.20.1-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 416ae29c7deffa440eb46145a982dc88a0c6f82163ff7eca7d2b2f06ad8ec393 |
|
MD5 | 25166f4133a3289b76f6f1278433eac7 |
|
BLAKE2b-256 | a6dcf4dbbee87fcd3571f39e6021c51b5352350cd2688f9b1fa3605985e12ce3 |
Hashes for pyodps-0.7.20.1-cp37-cp37m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f0fc2ff2c84c8cbc1efb966e90737a6603fb3a095aa82b4bdd1a2963dffe0c9 |
|
MD5 | cd56499209bbc7312bd0c45b1892fd6d |
|
BLAKE2b-256 | a8e5a164dd7a8bd5b642695afa2cb6711b30b34bff3466ebab2c4aa1fd967c62 |
Hashes for pyodps-0.7.20.1-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c41255ab6dd025c1704a2c20efb0a5ca54675258012c6589a46d2120308e408 |
|
MD5 | 4b9180990c08652ec10b8a0c551cb726 |
|
BLAKE2b-256 | 48de20d80cd9dfa42c97e20c821ef8bed013c0b76c8873dd7976905fe3fbb540 |
Hashes for pyodps-0.7.20.1-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3787644648f4b927203e1833247f442e5d0db531a680962e9e240b932ab2331b |
|
MD5 | e845e81db959f924b78948c4aa634cf4 |
|
BLAKE2b-256 | dc7453b74ef3db4327bd6d44771b12baa362b4f9576bddb5d6e41956a9a76855 |
Hashes for pyodps-0.7.20.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 496729d62939a397b6479c3fd1559e20b7986ba12bc1c8ecfe47211d0bc53b10 |
|
MD5 | 4451f9071dd411ddd55c48af4832ac44 |
|
BLAKE2b-256 | e7061d394335ddfac9d41edc444916bf3e325929660c03b22f5d138235478468 |
Hashes for pyodps-0.7.20.1-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e6128be2a233a31fb083cab9c74c79b25765a039c135b6d2e0e9e85a8e20d4e |
|
MD5 | 1a8c331b904a8ed17ffbb6b1fe5ec7c4 |
|
BLAKE2b-256 | 0f84206968f35c0b45513c299f72714b81b39efb0c7150346fa4b28becba2237 |
Hashes for pyodps-0.7.20.1-cp36-cp36m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9dfbc0ab136dd9f4ed3f5ba76e4a346b706cd675276110b61d7c06cac47587a9 |
|
MD5 | c8956858d19c90c95afc1da78982e31b |
|
BLAKE2b-256 | 0725259b87827af4b905b0e49d6856d3ff39dbbd6716bd9b4735f3fdeae50cb1 |
Hashes for pyodps-0.7.20.1-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d6f4e2276fc76cb2c803f2a0423852a41f7f5c836e3f15729ead6c4ed790f78d |
|
MD5 | 98dde00ff85d41fa43be80629c9fdfaa |
|
BLAKE2b-256 | 5998c73a8a16ddf3892082bfb1fa488617d68092ea743dd6fbe0b7f975c51ce6 |
Hashes for pyodps-0.7.20.1-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1752297c6f7b07fa8304cb3d99925afc962c909214cf3ace511b459389794dad |
|
MD5 | 63240d6f80552b1fd9e438a87bca0747 |
|
BLAKE2b-256 | e2b7c205a183e50381a0885cb6b22a04777dfd470183dd5d082d06c21598387d |
Hashes for pyodps-0.7.20.1-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e4060e1514e260ac5ea35462ea84cbc770d0e7fc6fa74e7498f222babcb42dc |
|
MD5 | 73e7565112e90ac14d3985fb87d98ada |
|
BLAKE2b-256 | 072f702e1d663ac052cfb37513062c6e516b09b40bd27b76d093965fce7a3486 |
Hashes for pyodps-0.7.20.1-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ae527a3c26c8583ed26e41ccf94e6e88ec04172a4ce8f8d4e741afaf14a2c0a |
|
MD5 | 432166303dd27a0aa76880ad56a3473d |
|
BLAKE2b-256 | 2821c9ce78dfd039e5de44d89a51a8bcfd65007f8cdac99235f44f4e057c0c96 |
Hashes for pyodps-0.7.20.1-cp35-cp35m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4dbf76b65cdddafc3f91940af5048535435a480b50ade43ad8d49883b06a6488 |
|
MD5 | 158a874a3ac98f9a2740593b02ec42c5 |
|
BLAKE2b-256 | c73cd41f18a116d8368d6309405d5ebd9c0fae2a8126e20c48401e1fb8903540 |
Hashes for pyodps-0.7.20.1-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c9e9c8a26809f7e6978187934626ad131d4d74bcc866581315e66faa23fb7e0 |
|
MD5 | 3da305c26ac4d4f94aa20a0db59e83ad |
|
BLAKE2b-256 | fcfad9c734bf62fd4e6fc48977b9a8271b4fb497f587ebdd462b1037c51e5701 |
Hashes for pyodps-0.7.20.1-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94499b6e6a57265c9e8c1ec61d0937296fa77b3fb81fa5c9187a5e10b6b3e2a7 |
|
MD5 | 2eef3f8a18f95123d165dfa51bf78b8a |
|
BLAKE2b-256 | 12d15c6a3762795423ad1bc3338e4813b1d584ec67e5b7b3a280dd480a58def5 |
Hashes for pyodps-0.7.20.1-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe15b0b65de15298a2371cd8d06b676fb3f9a8f55bc860be0de04261ba6eb3d2 |
|
MD5 | db0ba3097ff711c1bcbd9a011ae1ea23 |
|
BLAKE2b-256 | 1ba45f99870980ac03faaa80767722657c8cdb5ac8de7e082b2682bdc2d8565f |
Hashes for pyodps-0.7.20.1-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9998f983c9e323f43e40d3635f57e3b7f81b7368e14d28e2e9d97417a57a61ce |
|
MD5 | b52c3dd5a16b563b82505ba115af9708 |
|
BLAKE2b-256 | 2a452cdcfd7738fcae9bf57d362814d64246ced834c8504530458cbf97f75037 |
Hashes for pyodps-0.7.20.1-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 725fb9e686daccffb17f04e1a3c0548500114b37d22e40c7d4417dbcb55c39f7 |
|
MD5 | 59adae671548517e70c84261f1abdf93 |
|
BLAKE2b-256 | 54fbf1b65ad029e6213fc0c1c17ee15ae5e497d460dca8ad544b363ae8ca49db |
Hashes for pyodps-0.7.20.1-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97f1b390d4d4a72117cf4654e577b679bf9806a80c7d031a7b488420ece684a0 |
|
MD5 | b26f75c8f785c2c6193cb0d135661257 |
|
BLAKE2b-256 | 2935dedcdbd64c4edd35efb5fdfdd6a016fa3a6be326c4f6d3a3504f3e974018 |
Hashes for pyodps-0.7.20.1-cp27-cp27m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 988062f98ae7e406cbfae0a7be2b843f5250465e12bf03c0c47b6f1f69e6b287 |
|
MD5 | 05f8225cbfc5d1fa44381e7ca60bc283 |
|
BLAKE2b-256 | ccc1202423872bd2f4b2abfecc64e71e640a82e368ffa181a6662b42b6156ea5 |