ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
To integrate with Mars:
pip install pyodps#egg=pyodps[mars]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.10.3-cp38-cp38-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2ea6472d716199d3924148e5479e9dd3c9a945353d83d6aedbcd0b23a55f774 |
|
MD5 | 8aa8ec66223e54db29b631e6b01841ed |
|
BLAKE2b-256 | c26a8a331057cf2206279f5509becf3e27a5f62a6f2b95e56986c27acb9133fa |
Hashes for pyodps-0.10.3-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36ac8beb6449356d655ca7b6f6db8b4eeb69370856b7492cad7777110d691056 |
|
MD5 | aea7747cb713bdb50cb209d0b01b9769 |
|
BLAKE2b-256 | 4d8cbca885c728bd128c20d7f2e88d398e465177724ce02e167c2ea301f63924 |
Hashes for pyodps-0.10.3-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38c74aba4908d8ae7fc72100a6bbfdde62b9e62cc313810367fd91a10c60676b |
|
MD5 | 071af107127ad4d579df5b546d7d5c5b |
|
BLAKE2b-256 | 041eb790892c253bc9f4cf4fc5a8391baab4cd79a1c6054684d0d35559a30ba6 |
Hashes for pyodps-0.10.3-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c0f9b56031a0860b372ba37264ffa96c458869c522b772d6c3358cf0da606198 |
|
MD5 | 045577479f2b94abdd5e298ef5c5f775 |
|
BLAKE2b-256 | de3ccd046c0e95a553f3263c220e25ad295a8ec096b8a9a4a3f57120a3079fc9 |
Hashes for pyodps-0.10.3-cp37-cp37m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e91ccede370095e921739c438ae571c395c64a49e7aaddefdb57f9734ef7820d |
|
MD5 | 37636b88523a91aabac7588264e44659 |
|
BLAKE2b-256 | baa9bcd495279706f61d589832c5fc56343647af680edc5df10b9635e8da5fcd |
Hashes for pyodps-0.10.3-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 479959c73c1d565603199a16383f2f7f31ae299b213e28cc9633ac2224711b54 |
|
MD5 | e9a677901582f40d835f2719bbc3afad |
|
BLAKE2b-256 | 7767685a29732870b098e55b972f87dc50c4d088a176be2bbdf7b1df847580e6 |
Hashes for pyodps-0.10.3-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b29e394af683e29c8381bb2e2ab18ceaaaf1255bb94f29759cce60414851e6d1 |
|
MD5 | 1fa593e5d46916c384d8d6a8f40dac80 |
|
BLAKE2b-256 | 0cc71f270db98c0e96f3b2a193fb534024e86e598f6149b01ec12915a0062d8e |
Hashes for pyodps-0.10.3-cp36-cp36m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73c0d9b985982e19771c7928dda827408a1d703c923cc7955e5abcf208236fcc |
|
MD5 | 0343fb5100796348448313e25f1293a2 |
|
BLAKE2b-256 | ddacb90aa4e3888746d297b4a421d0e6781ba8e24877d1ef734e9db302a57acd |
Hashes for pyodps-0.10.3-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9322d608ea65b0464a56b019edf10d8ffcec7316b2dbb8897168e8499a393699 |
|
MD5 | caf18d993373af1f1209444722c70eca |
|
BLAKE2b-256 | f5083bb1a41591f0c9f415425f0989cab656f1f9fe0b5558ef7c4c6138f8ba13 |
Hashes for pyodps-0.10.3-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a93cf4fa0c63d401bb8830111f02b680bcc9a900849bca7ed8165e8699bfc96 |
|
MD5 | cbd3dfb5fea1e7fbd754f5cf282cc4b7 |
|
BLAKE2b-256 | 04c4e7fe2374515a72d9a927df7be5bd9be45159fbca90f6c3ff9a9e170a7647 |
Hashes for pyodps-0.10.3-cp35-cp35m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3687db2655cf6597e4395f486038f3e6a297c9b7d1fb046bfd965c3d8a2f9455 |
|
MD5 | 22e37b847191353312b58dc022a10bdc |
|
BLAKE2b-256 | 5980ff375875aa06261c7298c3a92484fb070e2bfdbbf26fc3d5dfd7c8897e47 |
Hashes for pyodps-0.10.3-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be4810820d4d4b6fad28aa11907fd62b78811537a3bfe58ad3df0ae90ccc245b |
|
MD5 | 7117a5be1bfa0a6c0bd4632ef7dcfc5b |
|
BLAKE2b-256 | d6517ff33e26a344a445680bc70f6fb5f93cf3b621f770bc24cacd3d29b462c5 |
Hashes for pyodps-0.10.3-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eeeed4a251f68a792dc8caf21e945cd4ee264104f6268e8fe6d93701a2c1e13e |
|
MD5 | 84e976f7f096e6b02fb1698977e02ee3 |
|
BLAKE2b-256 | 95507b7454406ef806787f58986cd9b10ceae65de5019a7e11f3adf83af55106 |
Hashes for pyodps-0.10.3-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a268b3342a1d0afaefc292a76b23bb68052636cfdcbceb4da47359424863c98 |
|
MD5 | 504f5fee890184021c826594edb1e1b8 |
|
BLAKE2b-256 | 60bcab8e5137bb612cacfd6bad6d7f092b0a1abdb5c296956597c9ccdea8ec56 |
Hashes for pyodps-0.10.3-cp27-cp27m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1755559e82ad8cb573ab8c036d5604d314ef09a9da94e86ae0c1d88e0acd955 |
|
MD5 | 8c47bea6f63d7c76996293e376920c67 |
|
BLAKE2b-256 | 34c67aefa3fef953c0743487dc33b7a7c1d26ace5983982301e6201aa34001c9 |