ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.7.12.1-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43144e6fc592575d284e4f14d0fb698fd3063d089e0437c1aae6ac09fe7895f6 |
|
MD5 | add086731002a94ca54ea20ec00dced5 |
|
BLAKE2b-256 | e99ac0c22147480f05349f19812aaa2fbc8dd5c0748022f5c1b9b17ac93092fb |
Hashes for pyodps-0.7.12.1-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a33f4fed979246c9c387acb9be520c849078bec362058188ff15f2ffdbf6b961 |
|
MD5 | 28c1f52efb7c8f801c293c36c673b354 |
|
BLAKE2b-256 | d8fa7b48a255567ea975fbbc327510d988ca3fbcdb3503939cf317eead6fdcce |
Hashes for pyodps-0.7.12.1-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f5e328aea9ef6399a8f9f20c23e67ecb488c7070b34d539abe13aa974cf76ee3 |
|
MD5 | eddd5f03c26d8debab08405e7f518484 |
|
BLAKE2b-256 | bfff205a8efd60e912cb3e34c6bc218a0819d14a04198498196550ce764025d7 |
Hashes for pyodps-0.7.12.1-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bbb84217b1c5cdd66e29718ea2d68ac813a5fc4f92d331e572c1820750c7ce50 |
|
MD5 | 6362014f0352e8bd87a5f3f44a4c5cb7 |
|
BLAKE2b-256 | ccc1f359d4d05092543a886d3138bc9f3626b22daee3011706c5a725542961a3 |
Hashes for pyodps-0.7.12.1-cp36-cp36m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ef5616b88caaff041e6935352c4e41b8afbff078fe32521b05fb8553646e016 |
|
MD5 | 1f4fdf9523f15754dce6817e0897246a |
|
BLAKE2b-256 | e459b2f2a998d4d0faa85d67b168b356e2e70e579ea621f7f4d5df00644a0cec |
Hashes for pyodps-0.7.12.1-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 45a3b9cc311040430697eff0e250d28ad3a71b84256a699b409fcdcc5e13620d |
|
MD5 | 89cd1325d73f8f05038609ee34e56094 |
|
BLAKE2b-256 | 4b755d32fad082628d9cc000f5ff6e26d88f4a06a2bef1f7b0825006b6de3c59 |
Hashes for pyodps-0.7.12.1-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 991c7168ab6f2d2829e915f1619a16e1aea59b80a87cc87741ab0203f0c6594c |
|
MD5 | fbdff12e912e6c564935846809022933 |
|
BLAKE2b-256 | 580cb820eb389aa7ca263863e4fde5a59d76834cbe5f9a5294aeee01b8b73be5 |
Hashes for pyodps-0.7.12.1-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fad24d7d42a5d46e105439de1ce2909a609d022139964ba1b5b3475a7a2d3d1e |
|
MD5 | ad07aa2edd8471172be129ab6bc521e0 |
|
BLAKE2b-256 | e81a8ec83e9819ed8cc3050842aa42f9a401f7c4e5d05a7535a59b2b81d4f9e5 |
Hashes for pyodps-0.7.12.1-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a5a20b6782abc2a01f36c1c1654ebaee873031806a516536032f0fd2210aacc |
|
MD5 | d24b063cd324ae7dcae390987327e6cf |
|
BLAKE2b-256 | 0d0915f9404263b27c5d1f2ec607bf9b0a0dcef0318d227029c87c3b21b5f40f |
Hashes for pyodps-0.7.12.1-cp35-cp35m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41fc082aca34d2a02131397dfc581d83f29a741d8a025ffcdee98a2373c52949 |
|
MD5 | ad8ada6041998f68dbfa662ae8405b24 |
|
BLAKE2b-256 | 23af6a0af602262c7451060749fe7a10762482c2e141f9c5cdb427ed5cb7e1f5 |
Hashes for pyodps-0.7.12.1-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f365e57b8d20545d5ef9e83e70a4dbe8a207c84153067082eb5c91c8d31f034e |
|
MD5 | 27ceb695a2295c5148e4ebfcec405256 |
|
BLAKE2b-256 | 4b6497205c6b4a60a495a4147da7979aa0504292df94ffe13c2057c5a66d8189 |
Hashes for pyodps-0.7.12.1-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 80f0b0c058746099c82c9c8096948c299ba275866c5d369912595c49ea43cf60 |
|
MD5 | 9005b7bc4fa0607b246843bc62971247 |
|
BLAKE2b-256 | c6db45d37a978245257022292f729a14ac1b2ff66cd0a0eaf63ddf9d84ddb7fe |
Hashes for pyodps-0.7.12.1-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5cd9657d368f0d07c854b72676a829ffe1a97b229516f3c0ea017b4f9f721db0 |
|
MD5 | 6f6ac9b18c3d6a0d64ce2e98ccda7ca0 |
|
BLAKE2b-256 | 7bbb5d2825ba6addcb843b68030f55d1d8752c572669023bef9f6ca5be950559 |
Hashes for pyodps-0.7.12.1-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8edb27f166627aa27869d3fef0dd2e0320becd082fd3a44bf694e850dd47bfbc |
|
MD5 | 3c27ba731002b704f592b420afe67753 |
|
BLAKE2b-256 | 4a47f4260fefa29496d1b6c76e935062f96fbec68a31cb247ec1c8378cff4e47 |
Hashes for pyodps-0.7.12.1-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bd0ac3d7c635c29188a8a547ff19d1f90be144001fea58ec64a845a29131d7c |
|
MD5 | c38ad54d9a76667a5e43c9e100f23d84 |
|
BLAKE2b-256 | 39a16230fd4d852a16c84f6fc461a018834a194469453ec6022dd347758a437f |
Hashes for pyodps-0.7.12.1-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52e1c1246cb94b8e82d17357d70a7012a1cbdeca01966490d5f8d1ac997c39f2 |
|
MD5 | 0f71341f4489ad25e400496d5a0f69dd |
|
BLAKE2b-256 | ea52a31723c05d02d9b72f2e3a385e0252814b0fd0f5feba0d23a25947585d65 |
Hashes for pyodps-0.7.12.1-cp27-cp27m-macosx_10_11_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a6f679d8bd4bcaaaaf7dc35e726cbc1d09da1f2ca3203e795483ccdf0e1c02e |
|
MD5 | 7db0ab8f36164b37fd8e68f2ca9b88cf |
|
BLAKE2b-256 | 74590d53aa16fcd52cc078cdde9707c9b659808512138b12797e69fe3569024a |