ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.9.3-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb67db2c5cf61601f5de7f79170d569833f2dba97037561593990881027ac1d2 |
|
MD5 | e9b17df6e22a2a72f97c953c866c1c4e |
|
BLAKE2b-256 | 98dd20fad229d3e9f5800301ce1a77a8f675c38f6fa254ede648a2da90b01225 |
Hashes for pyodps-0.9.3-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c1262aadf908fed91204c797f5ec4bf74c5f67bd004a8abb37ff915218f8fff |
|
MD5 | dfa5e7ae0266b8e3e08a59929b0c1453 |
|
BLAKE2b-256 | 9185b789c0df560f02dc1d5209483fb29ba295c9c181dd5185ef2508e33e3c8e |
Hashes for pyodps-0.9.3-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb6b2745f34df556bd2ceaaf7d6b9d7214b717126668d5e8f486c4999dee8618 |
|
MD5 | 844084bb79ce29ceee78d098a51162a1 |
|
BLAKE2b-256 | a53c818100d459c352d883b99b0d78e06ff8fe0a0bf072f1a10f93174899f027 |
Hashes for pyodps-0.9.3-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 857be0e0ddf7162c4c565accf50ced68a0ca97dfac0f7e114501ffb33f778631 |
|
MD5 | 9ec7ee22b347ed1206f5fc5081374644 |
|
BLAKE2b-256 | df85d8846e782ebe229063b8e1ad4e22ee87b0ad9b35fbd7fc5b1fc8a39963b8 |
Hashes for pyodps-0.9.3-cp37-cp37m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c5cfefd7e188aabbb93f3ca58f6184e9bac3efd9e944e1823bb823c6b354fe2 |
|
MD5 | e1be8499b378e01fb938301982325d0c |
|
BLAKE2b-256 | 79d74f8466b1a53e84fc84b4cdd74e213b45c39b8003cd08b0a7eebe02fa47fe |
Hashes for pyodps-0.9.3-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3d4e95c3d4f0a0a741e1b0a60e90b222957198702c83ba6ffb16b768e5c6f4d |
|
MD5 | 87ec5e68bb201ce648e64ed81cf65dd8 |
|
BLAKE2b-256 | 0ba6ec458ac3af951ffa3d361d275c3f56266cfa0f5cb785f3c66a4047150493 |
Hashes for pyodps-0.9.3-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a6277b62aca54dc25c6af392b641918a88c0fdf99529ab4a4e9a6bb086839d92 |
|
MD5 | 9f83f11d77671312a6186c71bb261cc8 |
|
BLAKE2b-256 | 439bd8509f3dd1532587068bad0c64c21c699ec23125e0cd5c5ecb8751b34e95 |
Hashes for pyodps-0.9.3-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eef3b0b9b6202f2bf1ff718a8bf3fdfe9092c371aef16d5977a4829f8f37530e |
|
MD5 | ea0a99729571b70c0ce1800099f16400 |
|
BLAKE2b-256 | 56ef434549a50400712ee5b59f7da56cfaf7ffcda036ba566a54d809af51f647 |
Hashes for pyodps-0.9.3-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a39c545ff5ed1b5598534c3a5fb0966ce0129edc6d73e63bce95bd6cc3580af7 |
|
MD5 | 0595b508f948ba157aaf1ee823a35fec |
|
BLAKE2b-256 | f21520163356ee490c38f92446289a83f9fa261ef9adcadc5f6f8332ebdcde4e |
Hashes for pyodps-0.9.3-cp36-cp36m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01dc0eb4781d706ef26d1870b0b06c129aed55ac98b6719dc652384272867849 |
|
MD5 | d64be6e9c49b75b735cfd56a28dbba62 |
|
BLAKE2b-256 | d3413cd132b53c4ac8e9d0124fd791f28300596a970fada7fdba8ff3428a1257 |
Hashes for pyodps-0.9.3-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e035e6ee418179fd360b272296c493590dd04e441f230a91bb9e570202fc6bb |
|
MD5 | 100ce139ddbdcce21629388f4326927c |
|
BLAKE2b-256 | fc26540f2990880af7d86b57b5ffac4ff97fc7483ae654a0d376c875cf1db577 |
Hashes for pyodps-0.9.3-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eba98692b1627b86fa33199592c0a7b555d82731db71e1a4e219d0def37282a3 |
|
MD5 | 02f16f9aa29cdf5c4d4f319fd6090aaf |
|
BLAKE2b-256 | febe23dc1231e44ef14ffc15b530079087a2922ef6d6e4441893d86a63dbeb4e |
Hashes for pyodps-0.9.3-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 79e80e9be226d727e1bebe231b6fd42786e9a6213a81476ce3ca51a755bfe21a |
|
MD5 | 7f91b65269f4f013fac1558baa6dbe97 |
|
BLAKE2b-256 | 1a21ce65f923ee1a8a50d2faf30d86bf8347a0f35ec9c89f2859a513ed190bdf |
Hashes for pyodps-0.9.3-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 899466a40a1ad7b0ec8e656e9092397ac0f3bc79a21c22521e99ffd4c817c6bc |
|
MD5 | 8d8eceedf06db9b068b092cc3912ad90 |
|
BLAKE2b-256 | 1d9aece839f10cbf3fe3b2d0d0c7ac00dd8712362a5a0b6a8cb25f11aa46a19d |
Hashes for pyodps-0.9.3-cp35-cp35m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 98162b071d060340bd487d04322bfda162219bffff0ddaba3c08ce80a67753ad |
|
MD5 | b99df03d5e9ea6c10bf417a30882fc1c |
|
BLAKE2b-256 | 7240c5587f3cb228d962ece2dbf6c0a12b12ace3e0648452a912006ae7ba6770 |
Hashes for pyodps-0.9.3-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15bac08e69646113f94b5793c2f99d8cfda54edc48e3b2d243fcb06dbcc7b62f |
|
MD5 | 8223c089ea88cf1cdb3d262459536e37 |
|
BLAKE2b-256 | 6ebf41b271b6d9051392faaa386a69a9c436587a8177f1a749394a013abfc447 |
Hashes for pyodps-0.9.3-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d003b69145ddbbe7dfb06a6722f50fd31b225717b148df8aad1f5a0eae073edc |
|
MD5 | 44ff717638a2141acfda16cf19a9a3bc |
|
BLAKE2b-256 | 57bd83849a3e4f8fa0b6927df11be108f65e46fa272481757e1577974a203a17 |
Hashes for pyodps-0.9.3-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb982453731b06799f84d127d74cfc5c60535bc6a569d3c229335ea27d9d424f |
|
MD5 | f697abbfb17462baf235475dd435d354 |
|
BLAKE2b-256 | d1e2734b10b7e1292971887be6a1b29b735525df0301d5d494c9d69d252d42d2 |
Hashes for pyodps-0.9.3-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f886a3892ec0b28915475895c9a71113527090b12cad9172b9fc1165907a368 |
|
MD5 | 9190f04391a61a4e2ddce97805187f66 |
|
BLAKE2b-256 | 7fabc1e72c9d941ec92ce6dc5a142cae9c797e489bac287196997520db57c92b |
Hashes for pyodps-0.9.3-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0838e0dc712bfcb0858f59accf1820c0a779885b188a04a539c13b4aecb5f636 |
|
MD5 | ef1f5a4acfd3df247fc3587e3e9ac0b1 |
|
BLAKE2b-256 | c52f846d9a6fb75a18d424ffc15dea608508bb5042eddb0494d3372da0527d92 |
Hashes for pyodps-0.9.3-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13f0776c7f4ff66133e5ad7315cb9ffd59bfc83450af43b7808b4dc484cd54d6 |
|
MD5 | 5cf4e0ff3cf9513300cc9789e9b899af |
|
BLAKE2b-256 | 2b0105fca90b234d7262f89865bd1d197b6b847252980082cb633eeba4b53d16 |