ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
To integrate with Mars:
pip install pyodps#egg=pyodps[mars]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.10.7.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d3056fc626df076050b6e6115e4f9bcc4efca42ad7f4d86958b510a6a5f3a15 |
|
MD5 | dc56f3ddc928ba2113de668fc5a48008 |
|
BLAKE2b-256 | 03330d3eb9de6422b1088aa7da13278d3cf5777e78e4634421899e84ad75def3 |
Hashes for pyodps-0.10.7.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7de2c866f2a58f12273aef229db9ecb138b7ba9c900b6685f7ee2fa475aa714 |
|
MD5 | 0fb038fec6833072693413dd17077c7a |
|
BLAKE2b-256 | 5ea6610d33d048c4445751e6bf8f3c09ee4d8375c2ad17b05eb7125f62c85dfa |
Hashes for pyodps-0.10.7.1-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9394f614b74b36d318c563acb97a1a2c719c359a9268bf775a6a948bcc084a83 |
|
MD5 | 74e9fc742aea23b11516f671e773c7f4 |
|
BLAKE2b-256 | aa1842208a740fff03a6735c34d59feb77cc37a12c8e83cca898f64320218401 |
Hashes for pyodps-0.10.7.1-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | acc85ed5175ae48d9b67bbe72f90a09920364310766e7efc6e10a51b6ab161f4 |
|
MD5 | 62d3f12a21c19fd00493d3afee3e8ec2 |
|
BLAKE2b-256 | fefe281dfb13370be6f5a5fcbba479782840604735b8680022d73a9b9aaace7a |
Hashes for pyodps-0.10.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 937086db86f6216ee5925922216b6939f297c017679d20a65f12e543a2890f6a |
|
MD5 | 85d65b70133ab84f89458e8ffc1f0a03 |
|
BLAKE2b-256 | ec28a7c20ed30a8c9e4083ce90b9eac8494842cfe29b8407b5906d2de74fbb6f |
Hashes for pyodps-0.10.7.1-cp37-cp37m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1664bc4b01b130c4d6ec8585e1fe1f9b3fcbbe71ce4cc02de5cfd6edacadeff |
|
MD5 | 95edfe648df1fbc21db5a698a922f37c |
|
BLAKE2b-256 | c756b74c0fdad65a8531fe01ae8e6d14e9b5bf13415058fe056f66009d905adc |
Hashes for pyodps-0.10.7.1-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1cdcc4c59e3be967ef51335206e6c040078926e2e7c0284b9092df3500cc331b |
|
MD5 | 9c6d9563147528fcd892d69627db1c69 |
|
BLAKE2b-256 | 5140814396fd79002c300c3942502224976294f56b561711b057c6953e34314f |
Hashes for pyodps-0.10.7.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 093ddf5652f68d859418487b2d21ebae13bfbc937ee38220bd813a2c24d5099c |
|
MD5 | 7f98b9efb26ce9fee4bb0465df0c4e5e |
|
BLAKE2b-256 | 3afe620cfe6809493ac78490be7c09790bfe3fd1f79415cd577bf8dd33c3e1b9 |
Hashes for pyodps-0.10.7.1-cp36-cp36m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ac6180278509e16ffbd3570bd7ae85b96e25c6a1c51e6538e623f5db1b0cfd9 |
|
MD5 | ffde0c694269331ce6f8335e5db4cc5d |
|
BLAKE2b-256 | 99060cd9262c5fce6421d1a551bd986f126bf6bb50a06b4c49a1debf3fba1370 |
Hashes for pyodps-0.10.7.1-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 59cd8dae254f833f536770114e27b11c4910c889c80c1a3742ff6a4de01779a7 |
|
MD5 | 6741df09016d02396b000b5a7e8663c8 |
|
BLAKE2b-256 | 2d50e6c00db5548e8139f5e5345a4ce0b9b3571a6f38d0e4c44fb2f25d49303a |
Hashes for pyodps-0.10.7.1-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7f9db2c2d056f5196a02beec234f27cf9a4727b2837a68c58cfc45856a043d5 |
|
MD5 | 1dbef98192a54e8a507dd33e80bf729a |
|
BLAKE2b-256 | f7c4cf3efa6a0e3d1ed27926bd8065e40c862993cb9bc11ff6d7615df9226d35 |
Hashes for pyodps-0.10.7.1-cp35-cp35m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a47573325fc16445314acc2c487388625589321ff93f99204d33abbdc0ed1fcf |
|
MD5 | b3c495609f970b7d83a813b6c5d4e732 |
|
BLAKE2b-256 | 38f5ae853f69393728d840584f52509755185e80c51a07833a45273a467ee185 |
Hashes for pyodps-0.10.7.1-cp27-cp27mu-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af5c0a5cd902a005c47507a30a381093d1d3d1ce1503d94abc290cc2268eae68 |
|
MD5 | e4e94a70fdc256aa6cd728cbfa5151c1 |
|
BLAKE2b-256 | 4bd8232497bc5a37befe26129de77b50b826ab08b993ba52decd6a0fe95dad79 |
Hashes for pyodps-0.10.7.1-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 483d9b3eb94e370affe029a4425cb9ce9b578e8f61518238df137ee90e6bb4a8 |
|
MD5 | dcf5aefc8135d5f1629362c8ab27f422 |
|
BLAKE2b-256 | 74b9c23e8f086d5e7051eba9a12f406592db20960916d7dc1c3f0fa63a35895b |
Hashes for pyodps-0.10.7.1-cp27-cp27m-manylinux_2_5_x86_64.manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97597c0e271c373c65cdfe792bb859218ca548030f1373bea3499bc2436802d9 |
|
MD5 | c98bc0172efffe4a825f213e330925ee |
|
BLAKE2b-256 | 647e7b77aca0932572fce33a8817350ca7164e59c59546afd0cdc8d57271e9c7 |
Hashes for pyodps-0.10.7.1-cp27-cp27m-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9e61b6efd6c5818b5735299ee33359c5e578048e6d33a3e2ccef896261f7bcf |
|
MD5 | fa78ca5337a0d900b6f18f1ea529ec80 |
|
BLAKE2b-256 | 892da69f500340a4b6dd2282afbd2de47b9f4bfc264aa844afc42be1ab0884d5 |