ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.7.8-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 149a19cf4a0a2381c25134f3a499c85d66c6aeb7f5611d39a05b0e153c10241d |
|
MD5 | d14409422d68046c5d6757013f50d334 |
|
BLAKE2b-256 | 5ee5bc9d915587c555265ebd3b4a34091ae4b7708513467ffccbc6d69772b4bc |
Hashes for pyodps-0.7.8-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff8d6c0666981bdf6750c4699a62b35284868426fcc5d268beac2a2433a0b038 |
|
MD5 | c02955c7c7722735326377d268001097 |
|
BLAKE2b-256 | 88f46a6a1079e9c24f052a0dec434429ef5e12ea3207551202527071261c164e |
Hashes for pyodps-0.7.8-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2a76763bfdcf2edd54930ea77885b25c15ab026085e046b2b376a7751bc77c1 |
|
MD5 | b37286bced65a3468495993e0aa0c449 |
|
BLAKE2b-256 | 98b6fbad35109a94e95b2b8d85aeb72264684cf5a2df9cbdebd40361e7dc1ecc |
Hashes for pyodps-0.7.8-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0f3cb25bafbfa25790f194b4a215beaeb2d7c8d4e43be3a073070d409aba5ec |
|
MD5 | 3d78e2252f4830513cb867835c95daab |
|
BLAKE2b-256 | fa171aff633c1ec8d79cc5c99b033ef441ef9d93d1f6a290a543a4a524c08a67 |
Hashes for pyodps-0.7.8-cp36-cp36m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01cf98caf288393ed4f8a4c27a45f029a472c6692d234d6a5054ce1efc93dcd0 |
|
MD5 | dd3df79595339f28eed0bfb551eb6353 |
|
BLAKE2b-256 | 935b154b1f26ced55684b99ed44e1e0d41cb3a2d944cef8df099ec6561f6f51d |
Hashes for pyodps-0.7.8-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90c23024f8a841fdd84338fc581404989df588813cd3076d1ba7ad03ac2c1264 |
|
MD5 | 43d7d6938d7e47394fac4b6fe218fd8e |
|
BLAKE2b-256 | d0d3425557b189e4e85149ce103bd44540b43004228af72ec262c262d18984ad |
Hashes for pyodps-0.7.8-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c9d43043a2599f277d8dac6a21120511cc5e8f6f175aeedf46e449a30d6fc73 |
|
MD5 | 2c1ee9b011a809fd4d6479a06267a2cb |
|
BLAKE2b-256 | b84704178bae95bc8b4af891fcea2119d609de27109099d52c92abf804893d52 |
Hashes for pyodps-0.7.8-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 828a045c9881d51cac7067b750f8779a23990e3f40e66f9614b7aec8bfd2d604 |
|
MD5 | 33ab9dafb8f11ac9af025a4b9c5508cc |
|
BLAKE2b-256 | 39d189f8fb5881627e4abdeb652acd423be11fcd21b3046aa4ffe33077b51245 |
Hashes for pyodps-0.7.8-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eeed9101f7b096ee4212dfc5569815cec6a6e396714f9775bd82bfb4a676f680 |
|
MD5 | 666cad93cf6c11a50e28eeb22f14520d |
|
BLAKE2b-256 | 76652a82c9e5e96f6d0b4b2ef4ea06ca466a524b1dcfd27c1df512eacbbdb89a |
Hashes for pyodps-0.7.8-cp35-cp35m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb4e20a3385fe88a42d7340a219b5c000fd4e67795ef786c98e841d6c4c29897 |
|
MD5 | 7a30ade7006dc49775a8751af5bee32a |
|
BLAKE2b-256 | 60a08109f015d857556abb98ff93048b4cdc117a92548e5142d1683e2c39a7af |
Hashes for pyodps-0.7.8-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e7537321c795fe2a8cb2c8e06207b93de0d6b2e06426dd3275bc3c317379021 |
|
MD5 | db8f9e74dfe50a5c43433fb863422e73 |
|
BLAKE2b-256 | 13cdb190ad25f3b8cc4fb4584c96630ac62926529e6b83e876f6ff7ef223cc9d |
Hashes for pyodps-0.7.8-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6495f0da078ab033131d6aa541dd634ded580bd99da95399fe8ab0183112d0c8 |
|
MD5 | 41ec6c18057bbfe23df1721ba9411219 |
|
BLAKE2b-256 | 7fe7aea27c2101cbdd19d3723f3c13f5e441de1971e3caf64191c3d55be87352 |
Hashes for pyodps-0.7.8-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38def4c867467148c80dcb2ee5a94a37196a78e409376239cd33449e19efe68a |
|
MD5 | cb9cef8501a892be5fbf214fd1d5f496 |
|
BLAKE2b-256 | 8781dce25e12c894c77f346f6fccde7aa92056baa50a0b6e9df14d928327f906 |
Hashes for pyodps-0.7.8-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e660ee70022cbaaecda9a8fc2c8c1d9d14bf50a027dea59f881df857b4d24cbd |
|
MD5 | 2f69deb7d43a3461883b2c42a596d526 |
|
BLAKE2b-256 | f63d8efee902975c96b4d780a78f31b8bc5d642cd778d8621d75ba715565b83d |
Hashes for pyodps-0.7.8-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4b9d2760690aa2f762cbdf91ec6ffac7b1ed34bf94d7579387ecdd8c921e78a |
|
MD5 | 4afd848e679502bd3d591e6d4f518fc6 |
|
BLAKE2b-256 | 26eab631e3ca1143155c36c7fb423ed68b1b88bc88eb2e2ceeb671b90d7d5a07 |
Hashes for pyodps-0.7.8-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1edc3f066fd7d9b62d59b76ff2f58d283afdb4a20a86253d4fe08ba4cbd85d2 |
|
MD5 | 146e35896c638760f68931ccc874780c |
|
BLAKE2b-256 | e8e9adbccb23581fce29a5b76b5838efb014130292de68487151fbb8fb8466d7 |
Hashes for pyodps-0.7.8-cp27-cp27m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9cc016da8b181dfb517965bca588afc0079f05882af061099905898130b2beb |
|
MD5 | 8d9737aacc1753d99c5c5817e6750d25 |
|
BLAKE2b-256 | 2f4abb28be4b0ebf8de8281e6b294df82184ea44ffed621603b2757c4023b9c9 |