ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.8.2-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06544e3c7655686942d5b948edeba186a06eca4201165f295b03da9a6d7931bf |
|
MD5 | d63e5c58814d4344000cbf032df4ea66 |
|
BLAKE2b-256 | 321c0ecbe5a5bbbcf4a99a019d13a01435177f67fffed0817ad4f517713ef29c |
Hashes for pyodps-0.8.2-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b5c13e9a275833bd5de90d019abd2865b0a56852d74d76b8f570eb848823688a |
|
MD5 | b8e06afc648a4f1be37394b5046ce57e |
|
BLAKE2b-256 | e51a2d2162ac21abafe8df295386b58b0820d4945e671f1cd19275ec27773777 |
Hashes for pyodps-0.8.2-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5944eeea7ecb80d150502006d808fa65bd064307b8f15eb39d00777b9e6f9f5 |
|
MD5 | e751c054c98a788e089a295e8b724317 |
|
BLAKE2b-256 | da9822afac93ffe3ff42605a405ea2a300a09f204a99998fc68cf0372e99d56d |
Hashes for pyodps-0.8.2-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d82fdc6b820668fa7e850815938982d23a87c67ecd34b3efaa6d7365e7979bbd |
|
MD5 | cf41a3585dbab7debcd551693e5329f2 |
|
BLAKE2b-256 | 7cf689517e57521a657f9350043fc5a5fc2751f167e0a0ef152d4178d6388f8c |
Hashes for pyodps-0.8.2-cp37-cp37m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 53c88bc28986088eb37ae24aefabcff7eee26e72ba0c03b4abff4bc117096324 |
|
MD5 | 0b71c2f7470b11bcb9a00b5c2035994a |
|
BLAKE2b-256 | 5b44b870a9500f5171b4b2fe9458a0760bc05ece5985ffaf5be866f453c8a5b5 |
Hashes for pyodps-0.8.2-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6cd89b0ed931bfa77e7fff18da4f4becd73e6135f29aecd438690156c3d07c36 |
|
MD5 | ff9c16d1e5c056cfb070300502d88032 |
|
BLAKE2b-256 | 231e3eb9dca3ca701ef9611b149c587f07caa7a13a4412434928877d242d38db |
Hashes for pyodps-0.8.2-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a86128518f7012ed1c7b7dd8786814833ad9d62512fcba6c37e3f1c097acdd09 |
|
MD5 | 22a6584ea9bcd2b3312ae6ff801045df |
|
BLAKE2b-256 | 66da0faffca6606dde1101a4b45fb1aba3b470d5e26b85440ea418feebd19332 |
Hashes for pyodps-0.8.2-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b60cf192fa4c3cadf9575d84b0d33f731260eb2208165bfa61c09d5b03d80f6 |
|
MD5 | e8e9234114baae23844f8e030b52f48f |
|
BLAKE2b-256 | b6f9d58c4cb391cf2a8ba677f52bb007b6469fc700c5aeb41d1e0cbcbaae572e |
Hashes for pyodps-0.8.2-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dfe3db5d8f59b4e235a929904cb5396ee22fa37ec23cb6f493ce9b9c959a5c6b |
|
MD5 | c79e813d49cf19e7f5e12a771e7a9699 |
|
BLAKE2b-256 | 802893d629df9abd6a8b4b72a89c60b8b92b5bca3a4c83a23acc75447fa174f6 |
Hashes for pyodps-0.8.2-cp36-cp36m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e8032a3f471f498bc3d1eade8e3f970de9d1af502dcf5d1191b34151ae10bec |
|
MD5 | e9f87d2713d5a74d2e2a7a5b77067cd3 |
|
BLAKE2b-256 | 2a28b176d791a89c143fc1d5b9d23f9decefcbaff18ac130993b5940da085de0 |
Hashes for pyodps-0.8.2-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec6b8738f092e553444fb19426355b4ea6fd7f1a5aca5e1097f8a727017a0512 |
|
MD5 | 7bf79d6b6c89d03662554a9de5c4c0df |
|
BLAKE2b-256 | 47a2dbf5c36ad21dfa638cdf4891768e1837e91a6f5b6b956df521d17d18b0cc |
Hashes for pyodps-0.8.2-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1811e2908022deecbd4ffbd39716d8a07ae741c43172ac379109580ff73fe9b0 |
|
MD5 | b370b77c18935c41027f85aa8e56e454 |
|
BLAKE2b-256 | 6a3da7fe3047b9ab8ae7a61fdb6a39b08abad920503797e5f500fbe5c2abbff7 |
Hashes for pyodps-0.8.2-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52e3489bb060ebba0bd8eef6f4dbb999c0885d8dba622a5d78a4b86011752016 |
|
MD5 | 44a71f30963eca369c984a034e04ab50 |
|
BLAKE2b-256 | f559163ebb7413833a9c1d9150756c3da2d9a88d78325115a1a3eeedc43b283a |
Hashes for pyodps-0.8.2-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe1970ea631c48f7a3963fdd355e0fedb1113fe719567fa4452722f4535c97b6 |
|
MD5 | 14deb419f2dffff81b9f7aa0fc42e7f9 |
|
BLAKE2b-256 | 07ace6ad1a85d87bdbeab193e7317732c7adc97f02f1c53a0d27ab7108565633 |
Hashes for pyodps-0.8.2-cp35-cp35m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5b9ef798977ea35b871a7724ccbcba3def48c8f08dd7b4b0499d1ed0440c382 |
|
MD5 | e7710218775aeb52991e821ebe1f5b0f |
|
BLAKE2b-256 | 68620dca1f262ae0fbb36178b7492fda95a8605bf660a9bc4d6bab48c5539f85 |
Hashes for pyodps-0.8.2-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9150b7f1537f59ae8d7b274e01993134136361a60779924cbcfd6f179211a72f |
|
MD5 | a5d6383e852ff1bcdbf66aa3acb381b1 |
|
BLAKE2b-256 | 057da1dc194ea237159fd76a708dc628626676115a1cb2200abd36108115082a |
Hashes for pyodps-0.8.2-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d6848e869187b01d7381bb9e5a1d8c054e391a2f5444ecd685c6d1d31049733f |
|
MD5 | f769a85560e0c43f12533810a950f1ce |
|
BLAKE2b-256 | 5947eacbf900e96f8b7eae39b40371564a5f796ebabdc7e29fa4c0a51eea83e5 |
Hashes for pyodps-0.8.2-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a27a4e8a5fb34b9d5f97c697892ea27099a1b361a003ae1c6a9f851793741874 |
|
MD5 | 8bb39d0319e2651a807b6a033053e59a |
|
BLAKE2b-256 | 2e9478744fff605a0ef5bc6b152cffc169b14a8cf7e0929dacef5d71f0cc3dcf |
Hashes for pyodps-0.8.2-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7de7b745642ae53fe4a3c0cfb331e93ac718ea41bcfd771ec22d5e89769d8602 |
|
MD5 | 455c715bb4552b33e95ec69a4e3d3af2 |
|
BLAKE2b-256 | 8a562d8ec8341cc5552748540adfdcef268c4fc02cfa016aad550e97e6637c82 |
Hashes for pyodps-0.8.2-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a0a4833992d8b60c7ff44c847573c32e073f53a8e0271c91c8d3ea803abe415 |
|
MD5 | ac473f0c14f4e6a09ca4b7030b477d14 |
|
BLAKE2b-256 | 764e81f015914381743ce3d6eea4255528c66126b31529806a8d1e3b958f528e |
Hashes for pyodps-0.8.2-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33ce0e80911f9e464414c7e22ce4b9eb7b2eb268c138f2ecbf0dbc925079cd54 |
|
MD5 | 5959d9130e0c49869bd848e68a4df74a |
|
BLAKE2b-256 | 4c723b6f465618580885ca96d4ded0e4b6ffd020608dfac8e47934499b9e3930 |
Hashes for pyodps-0.8.2-cp27-cp27m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 700dcdb0fee11cdc3528f43459d8f9a0ee4fef256dcf8f41ca6cfd836c9f3a22 |
|
MD5 | 8b2965b87c2e2eb7d73c11cc2dce5779 |
|
BLAKE2b-256 | e1f709c1e47d0ef786dcc550a59666bb333c4315228f7269f274eb6ae2cc915c |