ODPS Python SDK and data analysis framework
Project description
Elegent way to access ODPS API. Documentation
Installation
The quick way:
pip install 'pyodps[full]'
If you don’t need to use Jupyter, just type
pip install pyodps
The dependencies will be installed automatically.
Or from source code:
$ virtualenv pyodps_env
$ source pyodps_env/bin/activate
$ git clone <git clone URL> pyodps
$ cd pyodps
$ python setup.py install
Dependencies
Python (>=2.6), including Python 3+, pypy, Python 2.7 recommended
setuptools (>=3.0)
requests (>=2.4.0)
Run Unittest
copy conf/test.conf.template to odps/tests/test.conf, and fill it with your account
run python -m unittest discover
Usage
>>> from odps import ODPS
>>> o = ODPS('**your-access-id**', '**your-secret-access-key**',
... project='**your-project**', endpoint='**your-end-point**')
>>> dual = o.get_table('dual')
>>> dual.name
'dual'
>>> dual.schema
odps.Schema {
c_int_a bigint
c_int_b bigint
c_double_a double
c_double_b double
c_string_a string
c_string_b string
c_bool_a boolean
c_bool_b boolean
c_datetime_a datetime
c_datetime_b datetime
}
>>> dual.creation_time
datetime.datetime(2014, 6, 6, 13, 28, 24)
>>> dual.is_virtual_view
False
>>> dual.size
448
>>> dual.schema.columns
[<column c_int_a, type bigint>,
<column c_int_b, type bigint>,
<column c_double_a, type double>,
<column c_double_b, type double>,
<column c_string_a, type string>,
<column c_string_b, type string>,
<column c_bool_a, type boolean>,
<column c_bool_b, type boolean>,
<column c_datetime_a, type datetime>,
<column c_datetime_b, type datetime>]
DataFrame API
>>> from odps.df import DataFrame
>>> df = DataFrame(o.get_table('pyodps_iris'))
>>> df.dtypes
odps.Schema {
sepallength float64
sepalwidth float64
petallength float64
petalwidth float64
name string
}
>>> df.head(5)
|==========================================| 1 / 1 (100.00%) 0s
sepallength sepalwidth petallength petalwidth name
0 5.1 3.5 1.4 0.2 Iris-setosa
1 4.9 3.0 1.4 0.2 Iris-setosa
2 4.7 3.2 1.3 0.2 Iris-setosa
3 4.6 3.1 1.5 0.2 Iris-setosa
4 5.0 3.6 1.4 0.2 Iris-setosa
>>> df[df.sepalwidth > 3]['name', 'sepalwidth'].head(5)
|==========================================| 1 / 1 (100.00%) 12s
name sepalwidth
0 Iris-setosa 3.5
1 Iris-setosa 3.2
2 Iris-setosa 3.1
3 Iris-setosa 3.6
4 Iris-setosa 3.9
Command-line and IPython enhancement
In [1]: %load_ext odps In [2]: %enter Out[2]: <odps.inter.Room at 0x10fe0e450> In [3]: %sql select * from pyodps_iris limit 5 |==========================================| 1 / 1 (100.00%) 2s Out[3]: sepallength sepalwidth petallength petalwidth name 0 5.1 3.5 1.4 0.2 Iris-setosa 1 4.9 3.0 1.4 0.2 Iris-setosa 2 4.7 3.2 1.3 0.2 Iris-setosa 3 4.6 3.1 1.5 0.2 Iris-setosa 4 5.0 3.6 1.4 0.2 Iris-setosa
Python UDF Debugging Tool
#file: plus.py
from odps.udf import annotate
@annotate('bigint,bigint->bigint')
class Plus(object):
def evaluate(self, a, b):
return a + b
$ cat plus.input 1,1 3,2 $ pyou plus.Plus < plus.input 2 5
Contributing
For a development install, clone the repository and then install from source:
git clone https://github.com/aliyun/aliyun-odps-python-sdk cd pyodps pip install -r requirements.txt -e .
If you need to modify the frontend code, you need to install nodejs/npm. To build and install your frontend code, use
python setup.py build_js python setup.py install_js
License
Licensed under the Apache License 2.0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyodps-0.8.3-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 71d00a8228ec4d77a8596171e9b577d32e377b6f7c9763d824b98bd6dacb9ae3 |
|
MD5 | 7be439f9d2da55ca89e7c1aa7e5cb4a5 |
|
BLAKE2b-256 | 1558192978e7f96717b99aa2774dde18288245a671d161d442f9c877de863196 |
Hashes for pyodps-0.8.3-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 730d287723b7da976f7497528e09768434eb4f1b57754e54ab8a1af90d2adc21 |
|
MD5 | 89e42aeb485d413aeffd1862e4dd0ad1 |
|
BLAKE2b-256 | 48e29a0edbdd050355e0e5a0dfacfdf6b094c3ae5f8fa3f7079f3d966dc1cfe9 |
Hashes for pyodps-0.8.3-cp37-cp37m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fe6869b8a415fab77f767e8530a92c7715364627a8dfd1bb90017bf79720dcb |
|
MD5 | b6bfc563904183bcf2ee9a2480105ba2 |
|
BLAKE2b-256 | a99c6aa8f770abe044642a6bc4151b2d24755fdb2335d61ddf11186f27aecae6 |
Hashes for pyodps-0.8.3-cp37-cp37m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aaa0453e3882c7200d330df3fdeaacf8715f638b1f7a499587057d2d39ae6d11 |
|
MD5 | db691071d504bf476c6b483a98057d13 |
|
BLAKE2b-256 | b2769ca437ad0b301a7ef1b2dc61459bb457cd2bf257e97c4075d56d54e7fb63 |
Hashes for pyodps-0.8.3-cp37-cp37m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14c56ab878a063e96219d2229b5fa28d512ae8def3aceaca02ad55b1bfab6e68 |
|
MD5 | 40654faa1fcd4a1ad376302c7809ec56 |
|
BLAKE2b-256 | 7e25056a6526643e05b09702c45bf6accd59e32cfd5cb8a45286da041e3ef301 |
Hashes for pyodps-0.8.3-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c961fb5eb97ee814a7714ff1f8467cb584fbefbf02121b9d21e249ab1954650e |
|
MD5 | d1ed5a7ac8b8192d86e42a27c2374db1 |
|
BLAKE2b-256 | 2086a78ea0027653c12b75b9bb2750fc28b63dc86f8b0c2222363b237bbf9937 |
Hashes for pyodps-0.8.3-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 942bf882bce640e6a9b90443b7fa4e18087bb0e7bfc4a186ccbd6342200d1063 |
|
MD5 | 9b44e381317d1a6c11f85ca7d400773a |
|
BLAKE2b-256 | 92b4b3ba3aebdddb6e28d1ffed068a65050777bd28428c4b6144ba3b261a3361 |
Hashes for pyodps-0.8.3-cp36-cp36m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ffeb55e7c0b679460d487f8a3ff1d1c1e32af722715348da76d46fbbc7b21917 |
|
MD5 | d987ebb2c9e9b1c192d441af2a6a0f82 |
|
BLAKE2b-256 | 85050b7d349fe030dc4f41d319c826b256d5cead8154e010ef35527097db5960 |
Hashes for pyodps-0.8.3-cp36-cp36m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a647551ed2fa02b7ce5ad0674842605a8b3784c97917b8fe3ed818195384428 |
|
MD5 | 39c9a0d7f2d49747c6bddebeb67ef65a |
|
BLAKE2b-256 | b2996632d79609acfccec0393cd54d4e8ece880a18f31ab03b89c7ec005a041c |
Hashes for pyodps-0.8.3-cp36-cp36m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e2ba80cc437da434e955d82a65e8cfd25ab1cd100f75e4c4205410f9cde802c |
|
MD5 | e931f7dbb2e16d91f1f458ddc665f926 |
|
BLAKE2b-256 | fcc313fc5e3a7494133d2d088c7f82db4233e9f49cdbd9cfc27cf5654e701ad1 |
Hashes for pyodps-0.8.3-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67c01aea98a7eec3f461b74e42d17a6327f3112e0947811464cabdb41f45e692 |
|
MD5 | 532470a9a6d5e6ee5eb0b3e1dd1b7590 |
|
BLAKE2b-256 | 6064a17807c0973a6fe29f6a487a8b124cf388b108e7f1c60ccedb9a4347ed06 |
Hashes for pyodps-0.8.3-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c332400d25ae3fdd974f6912cc2ff4a74eff22a72b4aaf5e30d3dd1b6145f60 |
|
MD5 | 07d5e6863992029cbd97301b158dad7a |
|
BLAKE2b-256 | 090e11be856f0afb0901e2f55b156ea885ba44013a6527d990cfb7e8caf65f87 |
Hashes for pyodps-0.8.3-cp35-cp35m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f8ea231e2ac864b141fd2287148f28d0517c967f52f6ab29b25e01b6d20ad664 |
|
MD5 | a4fab637409f2d986c794c4fb4b422fa |
|
BLAKE2b-256 | a0cf8d81168c844a510e606d84296994ae73246b5863032d485a870ce9b0b57a |
Hashes for pyodps-0.8.3-cp35-cp35m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60fd3aff5ab2891faa11efbe5fcbe92da081bee26bc27fc6e22ad1a13f97ec0a |
|
MD5 | 7fe3ec347a7edfe02fd0d863beeacaf8 |
|
BLAKE2b-256 | 4a02710455b67d13634f4f188b35266176680b85c1c154fca81ebad093a29be6 |
Hashes for pyodps-0.8.3-cp35-cp35m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 051fd09185969ea4ed98f04867e9e0d48882e20f3d402674330cf187d32bf53c |
|
MD5 | 97b8f4edac4ca01ae381551fdc728187 |
|
BLAKE2b-256 | 5198cf228039466d21846fc3cfe8fe9efcffcfd2fb02220158c61982d3dbb668 |
Hashes for pyodps-0.8.3-cp27-cp27mu-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e2954637b1aa25368394015912903378968559b837cc870a1c4c3649c1ef6a1 |
|
MD5 | dd14ef6ac5e2156c85c28d22f306aac8 |
|
BLAKE2b-256 | 15e0ba690b9c319ace7507fbbd4d08e631c30a814942ff5ffdf606210c00ee77 |
Hashes for pyodps-0.8.3-cp27-cp27mu-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc9e6c05911c30b2cc072ce8f780aeddf3b14f49495c48add8233cd498b7b1b6 |
|
MD5 | a90f3e757e8fc48650f4ab25fb645f2b |
|
BLAKE2b-256 | d7202b4d54c71ade1d7af8df88d6ad94ebaf7cafcb65c76725d7eb14d2d952a4 |
Hashes for pyodps-0.8.3-cp27-cp27m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 02cb6fc1d08d059650db2e6bce26942dd04b285ec3831107e4a7244ad9c638de |
|
MD5 | f2c0d3cc75c9a1d29a1f3014a7b97d79 |
|
BLAKE2b-256 | 66f1dabe6af44d38869915ad75f7bdedd07f5ea4c799de9f23d9339986b88eb6 |
Hashes for pyodps-0.8.3-cp27-cp27m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e698bd0d9238a99f6cfcf349c2b74e66502752293667812ca1775ef8929caf1e |
|
MD5 | 7d33792cb326a78cd10bc4d1f4e2f191 |
|
BLAKE2b-256 | 6cff76af159615b422c551da34cc33948e8a22f45c7dd69be591bda3849677c7 |
Hashes for pyodps-0.8.3-cp27-cp27m-manylinux1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6bc9f5a1747eb5a5fc34f68db2ce08246568d1aa37fa94836f0355bb10eeb08d |
|
MD5 | 54f641bc52978d77599a4bcf3c0447d7 |
|
BLAKE2b-256 | 2f9e5eb1e00db65185aa801c8abf0ebe3fcd5611c97678c48d7c030984945aec |
Hashes for pyodps-0.8.3-cp27-cp27m-manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 860ae359280b630e820d269c1409015083ec05457ffcc8240c79f2710b246b6b |
|
MD5 | 6311b4606d1f0bbb95562d87b50cdacb |
|
BLAKE2b-256 | e4a7c264fc31596e5aff3d673e5334f1961e36eb73a58119e134dd23640fecef |
Hashes for pyodps-0.8.3-cp27-cp27m-macosx_10_13_x86_64.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 79652e07b3dd01103680d598755d6b9902f99f84c671612da72fc6be33e246ab |
|
MD5 | 1d131c4a4767c6c3c438266af5580973 |
|
BLAKE2b-256 | 0a0122eb545f42f2a6f1c8312ba77b52652f8d83dd385b8aceeeee57e731f605 |