ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.9.3.2-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | efadbb649691697fa0aa693b90efde694165fe4d2371100ee291a25ba9a97842 |
|
MD5 | 404932adf485f50705105bda0486fb6c |
|
BLAKE2b-256 | bc5b1518667b18af672747201b6773e65004e6f5012958788b76b8a5b7952f4f |
Hashes for pyodps-0.9.3.2-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6d4b54bccd3d9d43c74f0374d8d655873b0a9c16e30dc31095d2f537cf5e321 |
|
MD5 | 763dfcd5b86e30fd63738079faddbf4d |
|
BLAKE2b-256 | 90dee9ed414e2f2cf8a3efe01637393cc2c9bd5acfbbea2973995793d827323d |
Hashes for pyodps-0.9.3.2-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1c8b51fe43c2152ddbbf4d697f1d9570607049bb45edb62e8a3e03e283076aa |
|
MD5 | 7e1b49dd32c1fe16e0b2edd6af48d539 |
|
BLAKE2b-256 | 9f1706909a0792b650210517f1fce15cd8e0c1b32e52b98151e47eb0d08ea873 |
Hashes for pyodps-0.9.3.2-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82e4f1ec5bdb47b3ab253fffa09b56b0b82ae767211e758d2672a4bf96e0a3f2 |
|
MD5 | 213403d88e744e82788e0e293a434966 |
|
BLAKE2b-256 | 339f1e7309d59ae0cdf8ba29a57c650173db8b2db7a43460394a05f3c1071bc8 |
Hashes for pyodps-0.9.3.2-cp37-cp37m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec6dcf5da4e4b35712a236ae38ebd87e75b477773ba527c0988c450501aa8885 |
|
MD5 | 20c982347ca98ece542048dd9a3e3649 |
|
BLAKE2b-256 | aa5ad25cb41e177ef81c92e9392836f5655252ef914909b6fbdb4ffa10de84e5 |
Hashes for pyodps-0.9.3.2-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8689edf22def7f5a6fa0d0baaa953dfd1e85f3e6a5cd1696854076548da3720 |
|
MD5 | 4f1442ef0939a4f18b28c6069ede456b |
|
BLAKE2b-256 | 9ce797a332b17c1e1a61f1e76b909612712b4fa23c1be81600ea1f9f7d441c5e |
Hashes for pyodps-0.9.3.2-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 64c7a8f34f6ef85cf54354d54eb43974e1401049e3d37e5e55ff13c5440ae5c0 |
|
MD5 | d17880438bb424d318f50ea5a597a415 |
|
BLAKE2b-256 | 940c57a353f36e667cd009414e7eab5e9783552639b0506cb4f83432d6c2cb46 |
Hashes for pyodps-0.9.3.2-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 227f37c0242292a1dab085ff45da3d6d33fe422ddedf1f128d98cfa483f2a274 |
|
MD5 | ba7e605bdcb9bfbac58c28127b1ece18 |
|
BLAKE2b-256 | d14074e29352a2d3d1d70a313ebc1147e93b4e47873ef1fa0d5c4d98436d9815 |
Hashes for pyodps-0.9.3.2-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a7a5ad7937676c2a4caad4f4dc0b2178c12bc18353e5591a0c7efc0a6e40d44 |
|
MD5 | ad6ca2b534a855e4293f5e6bc4c9cb6f |
|
BLAKE2b-256 | fda9d4da14fe523e1471add2006831f8e2cb108fda55e7b1384f985cf8d0f437 |
Hashes for pyodps-0.9.3.2-cp36-cp36m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e620462fd16b9f0a92432b291d43f3f0be21c0f0f652fc0e282c4a3d768addad |
|
MD5 | cd4aaa12bee45bc53611134f569c598a |
|
BLAKE2b-256 | 46b65080c5741b3471a8aaa0659d8df2b8bc27bc65689f99e9057bc889f057c7 |
Hashes for pyodps-0.9.3.2-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a2de1de0064fb0e06c58dfee8926dbd6fca1f978c48a7fda19afd61556f014ce |
|
MD5 | 162374118be5488de0d4002e1495e40e |
|
BLAKE2b-256 | ca3e4d701e69ab792393b3d2816f2e94fc8d2d90fe7a91009e72f2ed5d598f2f |
Hashes for pyodps-0.9.3.2-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 63083c7674a5441694f7d073f410250d8f6ad9937fed653554134ffd08686d0a |
|
MD5 | b632226db70ec95ac84b164014015308 |
|
BLAKE2b-256 | 13b5c916bb3f331479f684feae6cb889904a1a52f179e81284fe0c8c7460b84d |
Hashes for pyodps-0.9.3.2-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 79a08cf28cdd1a67f081111b0fc51aa1b84983254386a316adeb1dcbaa74f6e0 |
|
MD5 | bfc940d82fcd5cbce37b7e4fcf9e5cb3 |
|
BLAKE2b-256 | 04fc5157a108134d5d7ef386e16b5c65450dcfd2d2d19925cf645771a05916ea |
Hashes for pyodps-0.9.3.2-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4579dd5bbb0d33fece36be69a1271be32643c640560bef08404fe57b68941b3c |
|
MD5 | 01145e3a07ae82203de030cc32e72ae9 |
|
BLAKE2b-256 | 908094bc6d16292f7aed463ffddf92b5c467881d166352052f816b5c454cc7dd |
Hashes for pyodps-0.9.3.2-cp35-cp35m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cbf4f51db10687d069ad58301d0e707a5838e7138cc62eb5d934ffbb57ad3f0f |
|
MD5 | 833d9ac84ec53ef1fab17d9226dee292 |
|
BLAKE2b-256 | 3d5ab06022d7b0f65ba8f37ebde10eebff3abe205b27e3914a070ad8b34c0f5d |
Hashes for pyodps-0.9.3.2-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 464261067e1b19cab861d5ba5753987c9c664f6e5c0aae8f52884a712282c85f |
|
MD5 | 8b4bbaeb9ff58fd6523490b4fea4291e |
|
BLAKE2b-256 | 1eda6a5785dc56a646086a8263ba4513c18256e365f3cc149d667ea2873e7f3f |
Hashes for pyodps-0.9.3.2-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 214cee3728a37b5a1a611bf2a0c8a0885eedb9334e8f75a59b146ccc5d7c200d |
|
MD5 | 386f65225a06a63ff37b5d1c0e319ea1 |
|
BLAKE2b-256 | 907b7b39c20e5344fa08f917f76d636f2a5102a4a150ed6f5431389b3d7e2657 |
Hashes for pyodps-0.9.3.2-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a478a7828de96c9af00e0cc18878df5025c6b8a1d0a5b1f475162bb47096ed9 |
|
MD5 | 484ac969c3fa10fde43b75f7306ddfd4 |
|
BLAKE2b-256 | 2ab9d21d1c33d313e6bcde48abff4248293eb525149f13aad22cc62ae7dbe8e3 |
Hashes for pyodps-0.9.3.2-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96996a721db72997255a4ad81796be4b2a51771226b5551f676b56d75c3ca557 |
|
MD5 | af6518194d33b473f699410a8ad656e1 |
|
BLAKE2b-256 | 3c75bba7197169b1f55ead084dba01ccc732d2bbd190efff42ec22005441349c |
Hashes for pyodps-0.9.3.2-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ccbabda906df881d06a75dbdf7ddff999d9d9ca7958bfb36c5c1f58f69b5cc5b |
|
MD5 | 81e299776882aeb2a5704a5e97634789 |
|
BLAKE2b-256 | 31f48b8005be7f2d7f92a6b26da1aa7c3e3b5bea41527b9c77a9775645c2f324 |
Hashes for pyodps-0.9.3.2-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 54b59ef1cfefee104ee3e4db551bdc80291cfc8e1f3769b21ecb5633a9e53013 |
|
MD5 | 24fd110a75d9ea2f85c890f49a662a07 |
|
BLAKE2b-256 | 7fad9a67fea96a3cb31811945d5aac8585fd6c89934cc9c2f030301b16c2bfc7 |