Skip to main content
https://api.travis-ci.org/fgmacedo/raspador.png?branch=master https://coveralls.io/repos/fgmacedo/raspador/badge.png https://pypip.in/v/raspador/badge.png https://pypip.in/d/raspador/badge.png

Library to extract data from semi-structured text documents.

It’s best suited for data-processing in files that do not have a formal structure and are in plain text (or that are easy to convert). Structured files like XML, CSV and HTML doesn’t fit a good use case for Raspador, and have excellent alternatives to get data extracted, like lxml, html5lib, BeautifulSoup, and PyQuery.

The extractors are defined through classes as models, something similar to the Django ORM. Each field searches for a pattern specified by the regular expression, and captured groups are converted automatically to primitives.

The parser is implemented as a generator, where each item found can be consumed before the end of the analysis, featuring a pipeline.

The analysis is forward-only, which makes it extremely quick, and thus any iterator that returns a string can be analyzed, including infinite streams.

Install

Raspador works on CPython 2.6+, CPython 3.2+ and PyPy. To install it, use:

pip install raspador

or easy install:

easy_install raspador

From source

Download and install from source:

git clone https://github.com/fgmacedo/raspador.git
cd raspador
python setup.py install

Dependencies

There are no external dependencies.

Tests

To automate tests with all supported Python versions at once, we use tox.

Run all tests with:

$ tox

Tests depend on several third party libraries, but these are installed by tox on each Python’s virtualenv:

nose==1.3.0
coverage==3.6
flake8==2.0

Examples

Extract data from logs

from __future__ import print_function
import json
from raspador import Parser, StringField

out = """
PART:/dev/sda1 UUID:423k34-3423lk423-sdfsd-43 TYPE:ext4
PART:/dev/sda2 UUID:74928389-852893-sdfdf-g8 TYPE:ext4
PART:/dev/sda3 UUID:sdkj9d93-sdf9df-3kr3l-d8 TYPE:swap
"""


class LogParser(Parser):
    begin = r'^PART.*'
    end = r'^PART.*'
    PART = StringField(r'PART:([^\s]+)')
    UUID = StringField(r'UUID:([^\s]+)')
    TYPE = StringField(r'TYPE:([^\s]+)')


a = LogParser()

# res is a generator
res = a.parse(iter(out.splitlines()))

out_as_json = json.dumps(list(res), indent=2)
print (out_as_json)

# Output:
"""
[
  {
    "PART": "/dev/sda1",
    "TYPE": "ext4",
    "UUID": "423k34-3423lk423-sdfsd-43"
  },
  {
    "PART": "/dev/sda2",
    "TYPE": "ext4",
    "UUID": "74928389-852893-sdfdf-g8"
  },
  {
    "PART": "/dev/sda3",
    "TYPE": "swap",
    "UUID": "sdkj9d93-sdf9df-3kr3l-d8"
  }
]
"""

Release files for raspador 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for raspador 0.2.2
File Size Uploaded
raspador-0.2.2.zip 11.8 kB Details

Release files / raspador-0.2.2.zip

Download URL raspador-0.2.2.zip
Size 11.8 kB
Tags Source
SHA-256 checksum
How to use checksums
664cf4e6b1bd0ab60c4f5032d6aa6e0d221301bbadad067fc28f718af60efd94
BLAKE2b-256 checksum
How to use checksums
8ad53ff6ce348211782b047fb1649b943090ad68a2ae45a96ef9a26013024983
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

0.2.2 This release

1 release file

0.2.1

1 release file

0.2.0

1 release file

0.1.3

1 release file

0.1.2

1 release file

0.1.1

1 release file

0.1.0

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page