ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.7.7-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ed3095374c8985a6fbbc17d002754c11bdc676db4f4e46670ae4a01c9f287d6 |
|
MD5 | c8a5913a3e65c409f9d8496dfb4856ea |
|
BLAKE2b-256 | d9462bfb0f8b941fd197bc9fb9b25298a937774f2c1e3152bce3057dd5cd45b0 |
Hashes for pyodps-0.7.7-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c86e819723d596573c5fc249b71ea2dae28e1830cb9c8713bed1b17cb8f77be2 |
|
MD5 | 489b83600267515a93eab6c87985179e |
|
BLAKE2b-256 | f83d7338bbbc6c119ea06367b1dac8c5e8c7d1984ea5e2ffb19bfb0e69232939 |
Hashes for pyodps-0.7.7-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03e540e25e36c06ab3e0724e073b63a0050bce6ebd01fc57c97ffd755cf52750 |
|
MD5 | 5c3a928fb935fe121508a316af09f0bd |
|
BLAKE2b-256 | 1fd107c2b2c53e187a97dd3ee2379e23025cc466fd461a376c1729700c78500c |
Hashes for pyodps-0.7.7-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78ace0d00d59304a198371b61b4f80363695cbf6fa4f8a7822b2457677fca50d |
|
MD5 | 1c98d928f1d996482ffbcba1359a7493 |
|
BLAKE2b-256 | 4c5f2efef86e32a4a50c48f17c4afd8c414576c18f4ebf45c59919fb4e82e287 |
Hashes for pyodps-0.7.7-cp36-cp36m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84b394b7a77ff70e4ee3c837e0e3cd44ba0510d85064f95f0dd75beeab4c6504 |
|
MD5 | 0296324c22ba0529f826c1ebee231887 |
|
BLAKE2b-256 | 620608441ba0fec7827f44d7bd2fffbf6ccf472bc78e22cbf78bab429073059b |
Hashes for pyodps-0.7.7-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e807a244b7d8c86168d9d82d2bcccedd1adcd6cf40ae6e618a6eccfc9c6c1cd4 |
|
MD5 | e7a33b386cb9ce18090266bcba470ed0 |
|
BLAKE2b-256 | fea2c9ac401e6d3abc04e7a5b470da75176e81c7f924e4d0917dd872fd3e64a6 |
Hashes for pyodps-0.7.7-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cdf01f364f6648a3ee97c5217863d53bbd8a9e2d6376b43a40a98bc4df3e262e |
|
MD5 | 506a0e6f2ba7a36aa8695aaa4c0af18b |
|
BLAKE2b-256 | 1041d26e5f18cd5b22d38937a100901d04c2e8ae1d7f34abf1275c94534f6c37 |
Hashes for pyodps-0.7.7-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb56298840f85312791f145a6791b8a7b55541995de645586466393d76e9e6c0 |
|
MD5 | 6e708e5e5b4ce052f2ccc77a93d5bd7e |
|
BLAKE2b-256 | d6f72cf13bb4f9920a8a3c8f9b3938b496ee78297439ce12653866f651d4e1c4 |
Hashes for pyodps-0.7.7-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9595a1245f4db558f294738ba367e0d30936774aa6333a18dcaefcb08505abe |
|
MD5 | f0144bc475d6aa9bd955e4fb0cba4a25 |
|
BLAKE2b-256 | 8eb0bcbb326fbe922810a927592f054729d151ca8144f597c0de5aa34e3bf7a3 |
Hashes for pyodps-0.7.7-cp35-cp35m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85d691a29ad7559c169d7c8e34ffbcf70cbfa4865191a29d2d1dca237cec0b26 |
|
MD5 | 7c52396ab0eaffbdc1910eb76990c41c |
|
BLAKE2b-256 | 2242cb64bb25edc8431718d08c8d12eeb55d53d77a513bfd8e26f3be7dd7e84e |
Hashes for pyodps-0.7.7-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | beb49e69f15a04d9000480b9f4ced07be7526f04d8ec5fa0beaf78b97d6b4808 |
|
MD5 | 48f59fccae2531c9926d60b295fd0c24 |
|
BLAKE2b-256 | a36a5497f6d28cdbd96f7919a21d666c98fb7def15946f311cd2db9e0cad6ece |
Hashes for pyodps-0.7.7-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9f8ed66958be2e04a96e3f062339f28c1a8dc21790ca90f1627aa332b7cadde |
|
MD5 | 2bcd712b5a1e2dde910c4a682f63d3a5 |
|
BLAKE2b-256 | a6bb7cd856f5d73e5dc70beda18e0893d3ec1b9ffecb881e293a2bbc81fe30e9 |
Hashes for pyodps-0.7.7-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e1bf7309d317ca941cc0abc85d7a40be5e9b512fbecafb174fee154c026d14b |
|
MD5 | ebca6141c42cc96d91b23a4d279d7d0e |
|
BLAKE2b-256 | 5ce1e3043f6fb4d2b0dc4d5fab2a7aa41f3e00407b0a0e2c409cb78448659422 |
Hashes for pyodps-0.7.7-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c425d837b54b93683681f11b9b9b0150cfdfa2e4d55b9e1441d4e0077d523ef0 |
|
MD5 | a18d217fb01626b260034dc56ca96b7f |
|
BLAKE2b-256 | 0337217c63b7095bd70b6e5529018d798c4ff95cbd87bbd796c08192f2b907a7 |
Hashes for pyodps-0.7.7-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f63f757ca6412d5c82b9a109d984f311e0730e8649f55006e9727fa0302610fd |
|
MD5 | 59cec5fd792f4691a8f4b9f70233604d |
|
BLAKE2b-256 | b80cab5127a42a0733ba37fe3211baa48e5cc6a9c622e6b8f66bee417a2a1b48 |
Hashes for pyodps-0.7.7-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 99e10b37cbc791a7ab90849ace41ee0f6097bf60e0fee2b4fcf36600e31d652d |
|
MD5 | da70cea5a2a8fa043d55b4420761cb28 |
|
BLAKE2b-256 | bac84dfad97dfe8559643823f871fc73847001bd4b675d72b905c17ecc2107e2 |
Hashes for pyodps-0.7.7-cp27-cp27m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85ecf14a827ca0e8d7b116f355da1ebdd5d724289b04c7ca1fe0f4ae519299c3 |
|
MD5 | aa25d91f2fb9d219f4521ba089592e95 |
|
BLAKE2b-256 | 62b86f09acef2cc5e7f0fcb264cc7816241c344a7c1a4bc9f8f20038b471b9b6 |