ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
To integrate with Mars:
pip install pyodps#egg=pyodps[mars]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.11.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13358bb4d83bd308166c457b21e42509e8aeb33a4fad1e47a29b012e04e6854a |
|
MD5 | 1039bac08711c033d1592f992888d520 |
|
BLAKE2b-256 | db5b15ef8315232fc0e40778481a8d39b5a1b3da32acbd12d76e49d8ee7778d5 |
Hashes for pyodps-0.11.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74739541ee0b4c41b7c91c961045c5fbaf0c6a8e91562abeca4e7f547506f4de |
|
MD5 | 5e030313adb0a17cb1a2c01082d46923 |
|
BLAKE2b-256 | db9894e27db1ec88c3491ee2d84f55ab9d8642c4da8f9cd5aff87d42ca28badf |
Hashes for pyodps-0.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 145a7f8cd2db661d132dcbadc4d5e558b53a9974d9ee1a214e71cb908a5f3d73 |
|
MD5 | cb1b47291466b58caa150470177930cc |
|
BLAKE2b-256 | 41cac4166985fdc049a403ea88065ec2d75d800bf688ea6514dc67386a205c56 |
Hashes for pyodps-0.11.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfcc511025e3e5f094bf9d9bb17b876cb697b3e53afeeab8a06b90257fd0ee39 |
|
MD5 | aab3a643b7ee37961c26e52e1cd28deb |
|
BLAKE2b-256 | 5f8cf9847efffb04eb49fe6717698b68b63e5de64cbca97110bbbac48ff78d15 |
Hashes for pyodps-0.11.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b0c41317e7be2f42c48042a0c1526be481ef155122d73154ec5fed7f17983fd |
|
MD5 | c4342fd32b87f126d80d35977f5b47eb |
|
BLAKE2b-256 | 016d5f0f38eb628a849764ee884c9ebb4b916940fa58a3a6446c80c0799dd615 |
Hashes for pyodps-0.11.0-cp310-cp310-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c513d97a62c0eff3245226ae1a50943fc44d21321404da0012617809e58744c7 |
|
MD5 | 6181c4abb9109aa47d34417fc914ac9d |
|
BLAKE2b-256 | 19a939c7ec3290300c6019aa1958a21aa834574b13630c1e22a866b595d57157 |
Hashes for pyodps-0.11.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c020d3864855709a84c285b0aad5da38049416708a905a0c858a056daa8b8e9 |
|
MD5 | 26e39be29f6ed9d516b9bb493a47f8ca |
|
BLAKE2b-256 | 1cd0cd1d954caaeaabc7f0ece17371e96e909c8b7210174f1d97daf407dd57bb |
Hashes for pyodps-0.11.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7aa643774f949728a2ebe68a40e8eb00aa5d534200cbe1add59774995f9600de |
|
MD5 | 2e1d8dea5d6c244fdee9928fbac35dc2 |
|
BLAKE2b-256 | ea33267246d24b143142610f3a94c43b62cfa0d9d447059a8ccaa5b7e99d1fa1 |
Hashes for pyodps-0.11.0-cp39-cp39-manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c317d3164bba16e74220d17c960578dc2ff9d7c26377b3b004228c86857ef1db |
|
MD5 | 97302a17115d3709000a707640dcf150 |
|
BLAKE2b-256 | b13e44240e8f110cbfbabc61f76412ffdd3143dc0e52fbbd1f7927c78de85cac |
Hashes for pyodps-0.11.0-cp39-cp39-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ad95d7623eba6018c85911c2e6609acfcb824030e68385d65e30f6b87d9593ee |
|
MD5 | 5a539d50b940ac030903786c178ca403 |
|
BLAKE2b-256 | 90186ee098ff4b471c49277b68d8bb9857bba109755b5d11b5de73557aaa30db |
Hashes for pyodps-0.11.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4cf47f2eea42c0121746da27ca0d9423d5661fba82048a69d171ed0a902fdb90 |
|
MD5 | f7488a599b1d5cebebe3d16a40b4ca4a |
|
BLAKE2b-256 | afeb13123a22c708ea8bc759c969ae4874aff7d2325e8698e4bcc776bf08de46 |
Hashes for pyodps-0.11.0-cp39-cp39-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cce4bdbf14e01d77105332be40c59a40d66c908aef0127f550ad9846e072567b |
|
MD5 | 78ba9152018f1a01fe19cb790b8b0266 |
|
BLAKE2b-256 | aa877c415211109a6c139bb3713317a5e52b1c62932ddb1c35a809dded9af3a7 |
Hashes for pyodps-0.11.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 769dbbbf94cb98c8d9dca0bad820314e213d34391826e1b9af0226c22eb12dbb |
|
MD5 | e1b1f0f54e47613c1bff345f1b1911de |
|
BLAKE2b-256 | aadc7b822b24ca576a159f08a63df923a289171510deb6557ad6f742b7b0ca29 |
Hashes for pyodps-0.11.0-cp38-cp38-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58902a2488d7d73103d84078952602a19bdf63503e740dd4acc912f99a4c5d06 |
|
MD5 | 9cdfd5c4cb436b597d4d6bc27ee4872e |
|
BLAKE2b-256 | 7ab6ab57d29617da22c146a7b3280525bdd1a715580a92872265c619c5f207cb |
Hashes for pyodps-0.11.0-cp38-cp38-manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c65f53ed5978124721c1bb007f45c209a16913eaee281926e483d6b33058519 |
|
MD5 | 519d578658990240b1a2ef2be6624552 |
|
BLAKE2b-256 | a5e477e8d7ccdbf76a1b0cc8149fc9ed13f5c6cf69ff298380a73340033bf0d3 |
Hashes for pyodps-0.11.0-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9134bacded3005e6050bfd976a9b7594fb2d40033a76d7161055b14432d49f5e |
|
MD5 | 0300f0f05c159346a631b1cb7eae26df |
|
BLAKE2b-256 | 4370c3bce104fbe4d97622347ef1f374365778c5e29f9de3311032998afbd7df |
Hashes for pyodps-0.11.0-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 430f8e96578e2ed8123aeb60b43dc55442f9108a148e94c629886ada31261c7c |
|
MD5 | 503ebca27121160bbbe0dea898d689cf |
|
BLAKE2b-256 | acc675a8124f120dc0f9598f6da073371171ec24f30b52510915259af6216e08 |
Hashes for pyodps-0.11.0-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fdb1e448a8702b3cc145cca50ef328efda7d4eec8e2059f8727a38788bcc3e04 |
|
MD5 | 38223fdd4de6d4adf06618dd8e676d51 |
|
BLAKE2b-256 | 0287c71097b761de660082bd9842208de280b08a634b8db7a3211c8887257447 |
Hashes for pyodps-0.11.0-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da0aafb8faea311b5ff50725551e3eeb8b6d48499431b8f137fa8647fc424e42 |
|
MD5 | 87b8785162563f95332ae1fc599e91c6 |
|
BLAKE2b-256 | 88fb7f80ee77ac47da78f56d16f201f601e1da66d8ee68fe6d05fa0b0fa7f1e2 |
Hashes for pyodps-0.11.0-cp37-cp37m-manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2e536bded3823a1923e19e8ebb6cd73739eb3bcfd1b538aef937475f6440424 |
|
MD5 | 7e45155a597ef6e6d16eabd8fb5d4be9 |
|
BLAKE2b-256 | 59b02ce88e36888309ca59da51bd1a7a43c4c7eeb3484106d2756b3ab0ec9648 |
Hashes for pyodps-0.11.0-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 26f888a99e171aa1dc5858f86497edb5e2467432ddf1b51e85049cde0c2bca99 |
|
MD5 | 5c7ff059835f7bda39cadbdac2d5c125 |
|
BLAKE2b-256 | e33451d72870d09cf36c279f5c1056bd200787e2ba7e9da7611e276a754cbcc5 |
Hashes for pyodps-0.11.0-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75701a6efdc12da333a8129b55081e32ccfa6ce80dbde1e9c253f5443a7b9732 |
|
MD5 | cf8bdc8e8fee08567a0513b6305734e3 |
|
BLAKE2b-256 | b6469de250bff413eb24305fc6d6d00c4d52cb7d98f2edec322e0a376d431791 |
Hashes for pyodps-0.11.0-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe695424ab592b1d2f59a501253efbaff766f0effda22fa4cd7bb5563f8d3218 |
|
MD5 | a97984d0876c81d8d89eb96b2e2eb4d7 |
|
BLAKE2b-256 | 0c868ee8b48050e0ecae0c591bcf8ba18b722e3a468f949c87dafd8de00f5475 |
Hashes for pyodps-0.11.0-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 479b0be17c278223ffe6028d951289046c2e1086089b369ba6c9b2eb1d51acae |
|
MD5 | 7c922cddf073d0a092c2029ff7f2a941 |
|
BLAKE2b-256 | 36917a890d3f934d89d61427f66bf39d164d292c8a51ff7abcf2535891a60b96 |
Hashes for pyodps-0.11.0-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f8ff33a5d9152f633a9756a20a2826977369d8504e508ccbd01b21e80ee0a32 |
|
MD5 | a8c6e629b3e461649fcff854b45ff64b |
|
BLAKE2b-256 | 60ff94b6f290dffe92816a286a621e2db892a0b6ed927d70decfac614d1b795f |
Hashes for pyodps-0.11.0-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dee7126df581c8b055b65e5336d7644fe6a0a8a6884ac6280ca84ef778c36590 |
|
MD5 | 5ddee95432d72ea96efbe120eedc01a6 |
|
BLAKE2b-256 | b7b98cd44eeec773728b80eeb37ba52cfe58d7b75ad225beb4f4332045995cc8 |
Hashes for pyodps-0.11.0-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 54ae00831836a42769e8d962f844e33821bcb2e295bb1a0d35ddfe808ac653f3 |
|
MD5 | 03bcfc35ef5c0a46ff39c7bde5583fce |
|
BLAKE2b-256 | 00f7a9ba5849a2c5a0bb727333ab6559d84a1220b079493c3cfb5e8fd347b210 |
Hashes for pyodps-0.11.0-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 228a33850bbe2f955a5e3e02f3d93397c26c0bc76014c62efbcab3f972adbf1f |
|
MD5 | 334a502bbfe62157f348fbb054ab6a5d |
|
BLAKE2b-256 | 3c9da0f17c77d016361bb0df13116801d855843e13823e2e40914293dd48ce11 |
Hashes for pyodps-0.11.0-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0facda2ce1342be2597f5f4f365a114175b4c26fb612b95f3efa364269842ffa |
|
MD5 | 5893c096799dfdeeb8cb01f094db0894 |
|
BLAKE2b-256 | 59f0880a9104f068464b5fc7c143d95334b0106a053da6394f9499aa065213d4 |
Hashes for pyodps-0.11.0-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f35e1e2d854c317fcf6232947943f671649a7acd1db2a434a0f81fada87f2515 |
|
MD5 | 1a49632d76e2a2fe614c54ea57c8bf5c |
|
BLAKE2b-256 | 7f5b248ee731e11245874164bd95d0cc5967c9b296207deeb9b4c8cc40246924 |
Hashes for pyodps-0.11.0-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 014ffaab3df46c4151e383a53429ab319082a36559d7eb8b64bc956e8a7362bf |
|
MD5 | c21562ad03ca3b8de1ebfe967d2c6fb5 |
|
BLAKE2b-256 | 4dd61e3dbb6f5c000f5d5da47d38b07799324bf5c26e816214be953d5aaa0595 |
Hashes for pyodps-0.11.0-cp27-cp27m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23afc9b1fa28957b1af067aad339589727a0ed939248a31c375d4514d980b65f |
|
MD5 | 753cea1bd75a6fa2140381a2ef5a8262 |
|
BLAKE2b-256 | 17de8913804e862f7bd48fb4958d70a3ef6df42018c0e105bccab4b746440198 |