ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.7.10-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 862790731d15d349ddce0e2d5d7e7b4ea607e6b99ea5df2d5cf5c18948940193 |
|
MD5 | 05763feb9bd05fe6d3754badd140c6dc |
|
BLAKE2b-256 | b56b08d61fc83167bd6d92043b1ef985793cd1cef6c37c420a10c259b9456bfa |
Hashes for pyodps-0.7.10-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ed072e1a49755068650f97f3edb37ce980883d7478df9db6a5bb7f3d47ee5ed |
|
MD5 | a7566d097c3cd9ddc458c7efd246c30c |
|
BLAKE2b-256 | 79d234c7deddab978db8e6ab75d838f51303f5150083bb6b31242d26261d778f |
Hashes for pyodps-0.7.10-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8cada1fec2700b460675f38708f12f6f52c804fcd880e1eef7271ae88c88d8ba |
|
MD5 | 3c5594ef2de8759ea8ae8d8eed187c19 |
|
BLAKE2b-256 | 385cafe4398d6c6783f3bd5b5e2ba049f8030bd6173b84a67e3dcecf12e2dd4b |
Hashes for pyodps-0.7.10-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6f1e50d83f8b255b77a1c0c0e44a14a12a8dffd9956ad7061756e361db64a0f |
|
MD5 | 320a37515188550cfe8b0dea6aa44c84 |
|
BLAKE2b-256 | 4a86b2dd9dd0ca501156dc404bc467991a4be403b457ae8644025fd820365c7d |
Hashes for pyodps-0.7.10-cp36-cp36m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b81895caec5bdf5fb3f9b125db07028705d3d85faa7f61e0ed2ca30c300a0c55 |
|
MD5 | 5975a4bd80eb3a671439ac1a2b768542 |
|
BLAKE2b-256 | 06bd08ba9d3974ff48fbbc26eaaa221658e533111e18c7b88d78107183a4bd59 |
Hashes for pyodps-0.7.10-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9990f9fa56dac9f0f1e24d54c8857772298ccf7046bf921e651b1d2a07f3be43 |
|
MD5 | 5939d59f0108d26b3957b1260212b846 |
|
BLAKE2b-256 | 6cd81f876202597ea8e0be8524cf21203ff7149339410ba213af9b63dfc1d330 |
Hashes for pyodps-0.7.10-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2278a84e8941d7126ae1a62bf4749f61efcb030516f5e4007e198c1af3bbb445 |
|
MD5 | 584568ec35f6ba9af00d2cb417f91dfb |
|
BLAKE2b-256 | 23e6a4665e0c769df0917793b1dedd22f33cd69ed0ba4e205294374a733e956a |
Hashes for pyodps-0.7.10-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 83f841f0e56a74e21966d9a38fb3df7cbb0068d0ab9631161d4939c4e8bb1955 |
|
MD5 | a710ca7db26209a3187949388c5dbf83 |
|
BLAKE2b-256 | af2c29409db9d7c661ea1130c0a5adee2dd5e840b2b0376014d310e5c24b4f52 |
Hashes for pyodps-0.7.10-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfcb1c117cc54a4b7a88d2b4df27c935bbf15d07fcf993d65b859d6d59248856 |
|
MD5 | ace0bf47e0965efc484af109f9319a77 |
|
BLAKE2b-256 | aa4524373681d3cc060814066b52c672b5ade23681df690009494ef172283fc8 |
Hashes for pyodps-0.7.10-cp35-cp35m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b63416d291c7be38f3cc78203ce645e4db2ddb495d126ce601ed17d788dd04b2 |
|
MD5 | 48ac90a41d9deecb97057eddff9869fd |
|
BLAKE2b-256 | e36fbf979beaf888f06f074e44da869af22dd7fb99503b85b57315525d981c50 |
Hashes for pyodps-0.7.10-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 31fac6f09d6fd78c8466815fb1fde9e8508864c695dfddb1c7c53279f2564827 |
|
MD5 | c179e736fbba87ed41d7a89f6ecd876a |
|
BLAKE2b-256 | e9ece6882c5f921a0cdc0a3485efd69f8e117512f57a9282e1db42237e7ce984 |
Hashes for pyodps-0.7.10-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3452b189a9a11004c4c46bf006669e28dc4085521cb58cb0cd8c26a9b19fc91 |
|
MD5 | 10f149a93b0f22287d31f02a5fa163c4 |
|
BLAKE2b-256 | 77db8b3a038f2b09897b88472c173a5423cb805c3db5e5e2b516d5093fb5ef05 |
Hashes for pyodps-0.7.10-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 47b0a89241107d40ff1331b0bf88fba3184fca4e9e75d17212231fdd5fcee882 |
|
MD5 | 1722d83f4bb73e56ecca0df430b13dc4 |
|
BLAKE2b-256 | 0259c789e61c5fafaf9504c8d17730f8ab17b60c3f89b9d314eb483075c1b8a2 |
Hashes for pyodps-0.7.10-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1f97e601e15299447bc7d7938b621c1633647e70a785c4ecb64c87d48dd292a6 |
|
MD5 | 6c639f1ac13894e3ff190b2352f81bf5 |
|
BLAKE2b-256 | 00ef31a35a8ec8452ce89c6ce11289f6ad3e374dd3f6b9d2e601fab83eb463b5 |
Hashes for pyodps-0.7.10-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a407e459679dcf3391ee84e4082bb56a5ebe45d6ccf2102e1c61658f37fc5232 |
|
MD5 | b51f19200f81cbf3861b603f6589eb31 |
|
BLAKE2b-256 | 3dd906b99adaffd577f1979caacc35cb7581794e22ad5c5afc565e5c57bd0d27 |
Hashes for pyodps-0.7.10-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c23d54992ff49824072a2bc0c3084a703d265d577ce2c8e6a04b86331404fa1d |
|
MD5 | 948798e261551b1ae99a624e7f0d24b4 |
|
BLAKE2b-256 | dbf55f5c32b892a29329c1c5a180e23e22c19f3ab228c9c642e64cc5f3611059 |
Hashes for pyodps-0.7.10-cp27-cp27m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5cee6910531e5b8aa2d00ef7ac95ea3f40c2486644856c664187a14cb642ea91 |
|
MD5 | 20eca6b5cd9563fe5c23280cc2f2292b |
|
BLAKE2b-256 | 4e2e8bd160e3a991d52c2c591bdf1c51eb2fddfbdfb487dd2c8b3f4e738ad19a |