ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.8.6-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 301eff00775b83670ed701b80f65458078d7d03fa0c6ab55e2f29f6ccefff364 |
|
MD5 | 2bf2cec6be20973bb7b840620d128e94 |
|
BLAKE2b-256 | 23620b24118ed93e8850ef8f5d5dd273272e60fe9a8375a836e39bfe12b42d80 |
Hashes for pyodps-0.8.6-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 146974b32ef25ea706453c111df0344d4f5ed9af4c9b324c0ecac985bf918f77 |
|
MD5 | 08231fd1cfdc75e1cb0f4865e3741a38 |
|
BLAKE2b-256 | 38618e55f6156ae724b649ec645c8e04c5e6aba4f92f43cdef722fcaaf2cdcad |
Hashes for pyodps-0.8.6-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cea9b5608aedfbeec84e9754be80597aa1bf9fc47bff1d60b6fa9385104c6c37 |
|
MD5 | 668739d930bd322b7d791b95662ecbc4 |
|
BLAKE2b-256 | 22e9fec7d74d439d2630aa034e4581c0eeb358a075e5af4dd46739d1cd6bfc10 |
Hashes for pyodps-0.8.6-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42291f16c8527b5081850fd52a11f9cfcdea366898fe0d3c71c5fe08a0b9a202 |
|
MD5 | c1e5cfd74697f5de91d83b1c54df3759 |
|
BLAKE2b-256 | 79db2e00e4b451a0e4d8897d09fdd64bbbe8b6bb6e59a11b9b3f7a5ce65d070a |
Hashes for pyodps-0.8.6-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b08538d858c8561561def5fbb626969a31913eaf50318b1d04c80887f293837d |
|
MD5 | 076fb33d30fc2ee397370255480300cd |
|
BLAKE2b-256 | 994a141386a9a388b0c037cbd5825f879710364db34fb62d7be415f36faaa532 |
Hashes for pyodps-0.8.6-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 276bf5289921c9754ced28f715d310ab45ed18e407c1f680a9817915c1f3c917 |
|
MD5 | 73a81a809a8f10548c173c374971f611 |
|
BLAKE2b-256 | 130e0c21f34d3bcff685dcacc89c0ba340cdc0e715992b2bf7258c702f1e5ec8 |
Hashes for pyodps-0.8.6-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a13ecad567c6cfeb24666d0c6a57e8219c7f8f32e5b7ad3c13ab15a46feb543 |
|
MD5 | fb97bc1b902d9fbc1f2ea9289cd86184 |
|
BLAKE2b-256 | 6a928ef273551b787396b65b79302a2139957a2db3ccad0b66bf5d006e2acb88 |
Hashes for pyodps-0.8.6-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c045c2ef4c2859c50fdc5d6773a2a59abf0d69b01289667c45d646ec73de181 |
|
MD5 | c558fbc924b8d6175031cf4eb3e6a062 |
|
BLAKE2b-256 | dfdb9fe5bb668a9c486e52ba739b8ad7c98439bdc86747e1b45f4c898c4e6a0a |
Hashes for pyodps-0.8.6-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8758c2fab2efd93830dbbdcbe55b8ef0d4bb0cb4db83d6c1a5fca951bfa05d5 |
|
MD5 | 7980f8d20572b7581b3f5acbbe05bcd1 |
|
BLAKE2b-256 | 2bcc459c950be101f6412fad79065023dbed191ea7c26f6d4f14df2c4005d1d6 |
Hashes for pyodps-0.8.6-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 049b177d6f055cb8ac8efa8440f68fffc8f8f3596eee17daad9537b1f906c47c |
|
MD5 | 7ff7b3c660b67ce711ccc8f752f79d5f |
|
BLAKE2b-256 | c511310ce3925e83835745ac030777decc262592ad01cc84847ddc12714b8897 |
Hashes for pyodps-0.8.6-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29061bedd48d02c31682555604c80ddff46631915feb4bcede71950177697b08 |
|
MD5 | be5ce4606c27748dd829f509b613c660 |
|
BLAKE2b-256 | 6edcc59bc95e0825603803d988ce4a507b4938c44e8a31ff935ac273c1888221 |
Hashes for pyodps-0.8.6-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9c40bb7b7f19c522fa5630d60beabe93041dc09b4fb45d2e2d0775a64c7805e |
|
MD5 | 627dde6b8dc40f390c27f70849fa86d3 |
|
BLAKE2b-256 | 328f261696834ebc91e3da9d3cfe5790defd170c96ab1e5ccbffbf3ec5299a70 |
Hashes for pyodps-0.8.6-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef8971b149a0097d7823ac08118042d5065a2398efdb797ca0df0f92cb84807c |
|
MD5 | dc5d7d419ac4c0a9df19f0ed969e05cc |
|
BLAKE2b-256 | dd30dd44c1cfdfead5d58a79105ae1d28ad9c7156f6004948e9f16b54d9f43e3 |
Hashes for pyodps-0.8.6-cp27-cp27m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d1112e06db2f4d8c47fa8ebd6f323a71bfb9d1da0ee1235a39e665398ac89bb5 |
|
MD5 | 279cc512d7f06eea64968a472d2195d9 |
|
BLAKE2b-256 | 0953d5de499a3aae864e280ad6390cd070c23e088e849bf78123a0f4c238c61b |