pyanyapi

Tools for convenient interface creation over various types of data in a declarative way.

These details have not been verified by PyPI

Project links

Homepage

Project description

Tools for convenient interface creation over various types of data in a declarative way.

Installation

The current stable release:

pip install pyanyapi

or:

easy_install pyanyapi

or from source:

$ sudo python setup.py install

Usage

The library provides an ability to create API over various content. Currently there are bundled tools to work with HTML, XML, CSV, JSON and YAML. Initially it was created to work with requests library.

Basic setup

Basic parsers can be declared in the following way:

from pyanyapi import HTMLParser


class SimpleParser(HTMLParser):
    settings = {'header': 'string(.//h1/text())'}


>>> api = SimpleParser().parse('<html><body><h1>Value</h1></body></html>')
>>> api.header
Value

Or it can be configured in runtime:

from pyanyapi import HTMLParser


>>> api = HTMLParser({'header': 'string(.//h1/text())'}).parse('<html><body><h1>Value</h1></body></html>')
>>> api.header
Value

To get all parsing results as a dict there is parse_all method. All properties (include defined with @interface_property decorator) will be returned.

from pyanyapi import JSONParser

>>> JSONParser({
    'first': 'container > 0',
    'second': 'container > 1',
    'third': 'container > 2',
}).parse('{"container":["first", "second", "third"]}').parse_all()
{
    'first': 'first',
    'second': 'second',
    'third': 'third',
}

Complex setup

In some cases you may want to apply extra transformations to result list. Here comes “base-children” setup style.

from pyanyapi import HTMLParser


class SimpleParser(HTMLParser):
    settings = {
        'test': {
            'base': '//test',
            'children': 'text()|*//text()'
        }
    }


>>> api = SimpleParser().parse('<xml><test>123 </test><test><inside> 234</inside></test></xml>')
>>> api.test
['123 ', ' 234']

There is another option to interact with sub-elements. Sub parsers!

from pyanyapi import HTMLParser


class SubParser(HTMLParser):
    settings = {
        'href': 'string(//@href)',
        'text': 'string(//text())'
    }


class Parser(HTMLParser):
    settings = {
        'elem': {
            'base': './/a',
            'parser': SubParser
        }
    }

>>> api = Parser().parse("<html><body><a href='#test'>test</a></body></html>")
>>> api.elem[0].href
#test
>>> api.elem[0].text
test
>>> api.parse_all()
{'elem': [{'href': '#test', 'text': 'test'}]}

Also you can pass sub parsers as classes or like instances.

Settings inheritance

Settings attribute is merged from all ancestors of current parser.

from pyanyapi import HTMLParser


class ParentParser(HTMLParser):
    settings = {'parent': '//p'}


class FirstChildParser(ParentParser):
    settings = {'parent': '//override'}


class SecondChildParser(ParentParser):
    settings = {'child': '//h1'}


>>> FirstChildParser().settings['parent']
//override

>>> SecondChildParser().settings['parent']
//p

>>> SecondChildParser().settings['child']
//h1

>>> SecondChildParser({'child': '//more'}).settings['child']
//more

Results stripping

Parsers can automagically strip trailing whitespaces with strip=True option.

from pyanyapi import XMLParser


>>> settings = {'p': 'string(//p)'}
>>> XMLParser(settings).parse('<p> Pcontent </p>').p
 Pcontent
>>> XMLParser(settings, strip=True).parse('<p> Pcontent </p>').p
Pcontent

HTML & XML

For HTML and XML based interfaces XPath 1.0 syntax is used for settings declaration. Unfortunately XPath 2.0 is not supported by lxml. XML is about the same as HTMLParser, but uses a different lxml parser internally. Here is an example of usage with requests:

>>> import requests
>>> import pyanyapi
>>> parser = pyanyapi.HTMLParser({'header': 'string(.//h1/text())'})
>>> response = requests.get('http://example.com')
>>> api = parser.parse(response.text)
>>> api.header
Example Domain

If you need, you can execute more XPath queries at any time you want:

from pyanyapi import HTMLParser


>>> parser = HTMLParser({'header': 'string(.//h1/text())'})
>>> api = parser.parse('<html><body><h1>This is</h1><p>test</p></body></html>')
>>> api.header
This is
>>> api.parse('string(//p)')
test

XML Objectify

Lxml provides interesting feature - objectified interface for XML. It converts whole XML to Python object. This parser doesn’t require any settings. E.g:

from pyanyapi import XMLObjectifyParser


>>> XMLObjectifyParser().parse('<xml><test>123</test></xml>').test
123

JSON

Settings syntax in based on PostgreSQL statements syntax.

from pyanyapi import JSONParser


>>> JSONParser({'id': 'container > id'}).parse('{"container":{"id":"123"}}').id
123

Or you can get access to values in lists by index:

from pyanyapi import JSONParser


>>> JSONParser({'second': 'container > 1'}).parse('{"container":["first", "second", "third"]}').second
second

And executes more queries after initial parsing:

from pyanyapi import JSONParser


>>> api = JSONParser({'second': 'container > 1'}).parse('{"container":[],"second_container":[123]}')
>>> api.parse('second_container > 0')
123

YAML

Equal to JSON parser, but works with YAML data.

from pyanyapi import YAMLParser


>>> YAMLParser({'test': 'container > test'}).parse('container:\n    test: "123"').test
123

Regular Expressions Interface

In case, when data has wrong format or is just very complicated to be parsed with bundled tools, you can use a parser based on regular expressions. Settings are based on Python’s regular expressions. It is the most powerful parser, because of its simplicity.

from pyanyapi import RegExpParser


>>> RegExpParser({'error_code': 'Error (\d+)'}).parse('Oh no!!! It is Error 100!!!').error_code
100

And executes more queries after initial parsing:

from pyanyapi import RegExpParser


>>> api = RegExpParser({'digits': '\d+'}).parse('123abc')
>>> api.parse('[a-z]+')
abc

Also, you can pass flags for regular expressions on parser initialization:

from pyanyapi import RegExpParser


>>> RegExpParser({'test': '\d+.\d+'}).parse('123\n234').test
123
>>> RegExpParser({'test': '\d+.\d+'}, flags=re.DOTALL).parse('123\n234').test
123
234

CSV Interface

Operates with CSV data with simple queries in format ‘row_id:column_id’.

from pyanyapi import CSVParser


>>> CSVParser({'value': '1:2'}).parse('1,2,3\r\n4,5,6\r\n').value
6

Also, you can pass custom kwargs for csv.reader on parser initialization:

from pyanyapi import CSVParser


>>> CSVParser({'value': '1:2'}, delimiter=';').parse('1;2;3\r\n4;5;6\r\n').value
6

AJAX Interface

AJAX is a very popular technology and often use JSON data with HTML values. Here is an example:

from pyanyapi import AJAXParser


>>> api = AJAXParser({'p': 'content > string(//p)'}).parse('{"content": "<p>Pcontent</p>"}')
>>> api.p
Pcontent

It uses combination of XPath queries and PostgreSQL-based JSON lookups. Custom queries execution is also available:

from pyanyapi import AJAXParser


>>> api = AJAXParser().parse('{"content": "<p>Pcontent</p><span>123</span>"}')
>>> api.parse('content > string(//span)')
123

Custom Interface

You can easily declare your own interface. For that you should define execute_method method. And optionally perform_parsing. Here is an example of naive CSVInterface, which provides an ability to get the column value by index. Also you should create a separate parser for that.

from pyanyapi import BaseInterface, BaseParser


class CSVInterface(BaseInterface):

    def perform_parsing(self):
        return self.content.split(',')

    def execute_method(self, settings):
        return self.parsed_content[settings]


class CSVParser(BaseParser):
    interface_class = CSVInterface


>>> CSVParser({'second': 1}).parse('1,2,3').second
2

Extending interfaces

Also content can be parsed with regular Python code. It can be done with special decorators interface_method and interface_property.

Custom method example:

from pyanyapi import HTMLParser, interface_method


class ParserWithMethod(HTMLParser):
    settings = {'occupation': 'string(.//p/text())'}

    @interface_method
    def hello(self, name):
        return name + ' is ' + self.occupation


>>> api = ParserWithMethod().parse('<html><body><p>programmer</p></body></html>')
>>> api.occupation
programmer

>>> api.hello('John')
John is programmer

Custom property example:

from pyanyapi import HTMLParser, interface_property


class ParserWithProperty(HTMLParser):
    settings = {'p': 'string(.//p/text())', 'h1': 'string(.//h1/text())'}

    @interface_property
    def test(self):
        return self.h1 + ' ' + self.p


>>> api = ParserWithProperty().parse('<html><body><h1>This is</h1><p>test</p></body></html>')
>>> api.h1
This is

>>> api.p
test

>>> api.test
This is test

Certainly the previous example can be done with more complex XPath expression, but in general case XPath is not enough.

Complex content parsing

Combined parsers

In situations, when particular content type is unknown before parsing, you can create combined parser, which allows you to use multiply different parsers transparently. E.g. some server usually returns JSON, but in cases of server errors it returns HTML pages with some text. Then:

from pyanyapi import CombinedParser, HTMLParser, JSONParser


class Parser(CombinedParser):
    parsers = [
        JSONParser({'test': 'test'}),
        HTMLParser({'error': 'string(//span)'})
    ]

>>> parser = Parser()
>>> parser.parse('{"test": "Text"}').test
Text
>>> parser.parse('<body><span>123</span></body>').error
123

Another example

Sometimes different content types can be combined inside single string. Often with AJAX requests.

{"content": "<span>Text</span>"}

You can work with such data in the following way:

from pyanyapi import HTMLParser, JSONParser, interface_property


inner_parser = HTMLParser({'text': 'string(.//span/text())'})


class AJAXParser(JSONParser):
    settings = {'content': 'content'}

    @interface_property
    def text(self):
        return inner_parser.parse(self.content).text


>>> api = AJAXParser().parse('{"content": "<span>Text</span>"}')
>>> api.text
Text

Now AJAXParser is bundled in pyanyapi, but it works differently. But anyway, this example can be helpful for building custom parsers.

Python support

PyAnyAPI supports Python 2.6, 2.7, 3.2, 3.3, 3.4, 3.5, PyPy and partially PyPy3 and Jython. Unfortunately lxml doesn’t support PyPy3 and Jython, so HTML & XML parsing is not supported on PyPy3 and Jython.

Project details

These details have not been verified by PyPI

Project links

Homepage

Release history Release notifications | RSS feed

0.6.1

Nov 7, 2017

This version

0.6.0

Aug 9, 2016

0.5.8

Jul 14, 2016

0.5.7

Jan 28, 2016

0.5.6

Nov 24, 2015

0.5.5

Nov 23, 2015

0.5.4

Nov 15, 2015

0.5.3

Oct 30, 2015

0.5.2

Oct 20, 2015

0.5.1

Oct 19, 2015

0.5

Oct 4, 2015

0.4

Sep 29, 2015

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyanyapi-0.6.0.tar.gz (30.8 kB view details)

Uploaded Aug 9, 2016 Source

Built Distribution

pyanyapi-0.6.0-py2.py3-none-any.whl (9.5 kB view details)

Uploaded Aug 9, 2016 Python 2 Python 3

File details

Details for the file pyanyapi-0.6.0.tar.gz.

File metadata

Download URL: pyanyapi-0.6.0.tar.gz
Upload date: Aug 9, 2016
Size: 30.8 kB
Tags: Source
Uploaded using Trusted Publishing? No

File hashes

Hashes for pyanyapi-0.6.0.tar.gz
Algorithm	Hash digest
SHA256	`c9f43190509e4154944b31bd0855a85d525f23b4dc3d5867a4490c2ff9aed064`
MD5	`ab63d6eda0e097395e35ced73d466149`
BLAKE2b-256	`bbc710d95365bd0421715f2d57f9b002e41a7ccc6d87798e89258b0feb4cee70`

See more details on using hashes here.

File details

Details for the file pyanyapi-0.6.0-py2.py3-none-any.whl.

File metadata

Download URL: pyanyapi-0.6.0-py2.py3-none-any.whl
Upload date: Aug 9, 2016
Size: 9.5 kB
Tags: Python 2, Python 3
Uploaded using Trusted Publishing? No

File hashes

Hashes for pyanyapi-0.6.0-py2.py3-none-any.whl
Algorithm	Hash digest
SHA256	`570ccfc035a7fd606ca6b14e3d3943e7305888762abf69a7f5d3bb78e96d25ab`
MD5	`f3cf0a664112d0ddf64a7b9f6a491253`
BLAKE2b-256	`318321f26ebcdcf95189d1d71d616906991cc51b6f98e34880ef1fd46330a444`