ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install pyodps[full]
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.8.4-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 45dda506f2c24ae10bc2be61cd577b4e73f980900f6442396dd3af88f8a84996 |
|
MD5 | 102bd5d36fb4cd57373e1d5cf9851c51 |
|
BLAKE2b-256 | e7fe2a6ff59bca6c6ae832b21f006aca55729360faa7513e4adcc63acc980e89 |
Hashes for pyodps-0.8.4-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3432f2165cce5e7b2102700afdd8a92fec546b35d191f47df9770b3e5054ccfa |
|
MD5 | b2d5bd95bcb2e4d0967fcba2dbb05cf7 |
|
BLAKE2b-256 | 871a2701401ccb33734a87de9773608afc41a77d792107258f1f92f96469e716 |
Hashes for pyodps-0.8.4-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c8e9d623226d917241a59febe4e06151c1d5cc483095bc81b8c6798bbc587ba3 |
|
MD5 | 482b9503ff6f3755d9d4fdbe2df4c52c |
|
BLAKE2b-256 | 09b3ea14af954680fc2cf6b9ab64efb2b6c7db406f3b76d10eac19463b4136f0 |
Hashes for pyodps-0.8.4-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e89b9c37b2a054a8958f834c59a030a12fcd2907e20005c7d7e9a0c3e5068a31 |
|
MD5 | 6dccbc78334729210175a079907804b0 |
|
BLAKE2b-256 | 498ff00eaebd375f9ff653ef8964cb7e3ce7f074a0c74c4f5c872e9e37b823be |
Hashes for pyodps-0.8.4-cp37-cp37m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c68f28f301209812e59d12248fb828a60e8a5c515429a5157c129f683b723000 |
|
MD5 | 01439c3eb52fb4fb565acf89bafd270f |
|
BLAKE2b-256 | 8f0f43a27c8923fce035e560a56b4175a3fd7f5e762c47e418d7266098cc5e0e |
Hashes for pyodps-0.8.4-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df1efb7c391303ce32a7792c7360564d378fb694834f33ec4d8557b70f03a35a |
|
MD5 | f84bcf429527295ba861005dadbcf265 |
|
BLAKE2b-256 | 4d8f90627faae688a021f81c514ec55022bbafc1116cf28d7d0585e7d0967bf6 |
Hashes for pyodps-0.8.4-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 668c424119e15b80baf7cdf93c69dd4d08b97bc1aaa5228d9e5ebc2e29166972 |
|
MD5 | 742d4fcec2fc61818a00704dc952f172 |
|
BLAKE2b-256 | 199b1749eb1bfb562411d41698e266fadf2a6209c636b06d6ca4c942e54d7ed2 |
Hashes for pyodps-0.8.4-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ea7e7caeb5c582a8915803d16c2211a18b68b42f603e368efee49863cfbb228 |
|
MD5 | 50cabcbd5a93c4120d399dddbcdcbfd2 |
|
BLAKE2b-256 | d0b6c92efa4c2f058d55aa2f8cbc506924f741ae5e7256a92bc21a1d67688400 |
Hashes for pyodps-0.8.4-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ed7ab8d58e22ba1a1aa7e7cfb243dd870f059d5a50b146d1db440ccedbb2f32 |
|
MD5 | 3e4da8c0aa01ee6945a81e7d18f56b38 |
|
BLAKE2b-256 | 27494fc768e9e87047faf0a4519334b95e38d11c2eb5df8cd3a4d89079fe280b |
Hashes for pyodps-0.8.4-cp36-cp36m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 550b8689d67669635bbb060e1b05d653681692ea9d665e3ed180e368ec5a57b7 |
|
MD5 | e6084ae3a277bab46858a4a61a576bd7 |
|
BLAKE2b-256 | 95c461f0d94001f9541ac4357be68df08bb613abc27d96d0b54da4282d9e8cb8 |
Hashes for pyodps-0.8.4-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 878a3d6d331753cd073fae3e9fd04774f93fb41d3c18f5256da8117c5e96f416 |
|
MD5 | b945fc3f5189a5e712ffce0bff74daff |
|
BLAKE2b-256 | d4e284a2e50676c4a6d81fff51b4e1558351e0733f725f2379c220c1dc75d906 |
Hashes for pyodps-0.8.4-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f68d7d0fe333cad799b99cb39d50a5f309e1594e4bfd9585d618dda105932f5 |
|
MD5 | c1c67473758cd02de444c1345ac54513 |
|
BLAKE2b-256 | 8fbc5c74e549649d179c3cac5ed477e04497d42d19a74fa6324514f1f955a41a |
Hashes for pyodps-0.8.4-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3887dac8d89b688d3829b3741d1c1116619a775d4aa0a9bf8f6f7d3b7ffe22a5 |
|
MD5 | 158185ba77f50df10400238e17601f32 |
|
BLAKE2b-256 | 27394ce3bcfe15f0efcc24c30756ff096006d979396b8b699aea92c607ba3023 |
Hashes for pyodps-0.8.4-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c8a397cd629f43f0cbff7010bf35880c24cbf25c3d9e04095cb5b4c0b4f4c27f |
|
MD5 | eea839f9afbe23d6fca7b69093f49e94 |
|
BLAKE2b-256 | 59f9b0069c8895487fe320e8577f1c204495f95083d3dc392d12f4bb536f9bdb |
Hashes for pyodps-0.8.4-cp35-cp35m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0708dd9aeabe004634f5f1a946951c062375c8ff84a75709d7159c931dbc0717 |
|
MD5 | 7da3084ca1088912742671f709651ca5 |
|
BLAKE2b-256 | b8243a6176fdb682844d6344960a1cefe468de38299cd18e27879b02a8f5c757 |
Hashes for pyodps-0.8.4-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 975e893765f342eaa47e4832c2de07a11602e2705cc0076bb1bbf7161136db6b |
|
MD5 | f9d0b5af844a359f5cea07d52fb22308 |
|
BLAKE2b-256 | f1b50724aa0f55d50dbb4fd2dea034d5b0ec04e4b86f993ea2a8d8314cea5137 |
Hashes for pyodps-0.8.4-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 27f3f6729018d2be9b526b1742894bc94303a3071456d091b3b4b1904a2a6552 |
|
MD5 | 6bea6882d31636aef872f9a70796e182 |
|
BLAKE2b-256 | c0d0727df661d3e0beae1e1428423bf7bf0dcf0b59a568bf78303272ddaf2c9b |
Hashes for pyodps-0.8.4-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74e78936ea9bed55eb745c072cf9160e2e80c1ef12758e6521c2f1635501e73f |
|
MD5 | 735efba6c7ab1ac9b1ac15ecfe532070 |
|
BLAKE2b-256 | acfd6e7e38ae10949c40209f2914b7b8366d2c5b0eb4b87fc184a5674333ad02 |
Hashes for pyodps-0.8.4-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0efdecd578a0ccccad563bf31d8bbfda6cbd51a5efa90ad19799307275fbf30d |
|
MD5 | 2c1ce580174d6f713cb5aa6e644b1348 |
|
BLAKE2b-256 | 28e49b507d018381c2cbb4179d5f9cda7eec877c2cabac9309352e51c1347ba7 |
Hashes for pyodps-0.8.4-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96b33d429f1c1ab426f935636e56600ce952a650181a41285b7b89158deed2f7 |
|
MD5 | e6d66c9dbf31cc2b161131524eaa76b6 |
|
BLAKE2b-256 | 1bd26fd2e7b8a08a7739b5f9e0e5fa864c500e07124657effb88c74171839878 |
Hashes for pyodps-0.8.4-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee2cd5453ae2761813b25402c77028655b60bfd7e9995ffe96ab2fd8dcf59c46 |
|
MD5 | 6af8ad1f097deb84ec3701f836f38419 |
|
BLAKE2b-256 | 733d01094e721f711096ccca499abc5f885e022cad9308de555e41116455895b |