Skip to main content

Rebulk - Define simple search patterns in bulk to perform advanced matching on any string.

Project description

Latest Version MIT License Build Status Coveralls

ReBulk is a python library that performs advanced searches in strings that would be hard to implement using re module or String methods only.

It includes some features like Patterns, Match, Processor and Rule that allows developers to build a custom and complex string matcher using a readable and extendable API.

This project is hosted on GitHub: https://github.com/Toilal/rebulk

Install

$ pip install rebulk

Usage

Regular expression, string and function based patterns are declared in a Rebulk object. It use a fluent API to chain string, regex, and functional methods to define various patterns types.

>>> from rebulk import Rebulk
>>> bulk = Rebulk().string('brown').regex(r'qu\w+').functional(lambda s: (20, 25))

When Rebulk object is fully configured, you can call matches method with an input string to retrieve all Match objects found by registered pattern.

>>> bulk.matches("The quick brown fox jumps over the lazy dog")
[<brown:(10, 15)>, <quick:(4, 9)>, <jumps:(20, 25)>]

If multiple Match objects are found at the same position, only the longer one is kept.

>>> bulk = Rebulk().string('lakers').string('la')
>>> bulk.matches("the lakers are from la")
[<lakers:(4, 10)>, <la:(20, 22)>]

String Patterns

String patterns are based on str.find method to find matches, but returns all matches in the string. ignore_case can be enabled to ignore case.

>>> Rebulk().string('la').matches("lalalilala")
[<la:(0, 2)>, <la:(2, 4)>, <la:(6, 8)>, <la:(8, 10)>]

>>> Rebulk().string('la').matches("LalAlilAla")
[<la:(8, 10)>]

>>> Rebulk().string('la', ignore_case=True).matches("LalAlilAla")
[<La:(0, 2)>, <lA:(2, 4)>, <lA:(6, 8)>, <la:(8, 10)>]

You can define several patterns with a single string method call.

>>> Rebulk().string('Winter', 'coming').matches("Winter is coming...")
[<Winter:(0, 6)>, <coming:(10, 16)>]

Regular Expression Patterns

Regular Expression patterns are based on a compiled regular expression. re.finditer method is used to find matches.

If regex module is available, it will be used by rebulk instead of default re module.

>>> Rebulk().regex(r'l\w').matches("lolita")
[<lo:(0, 2)>, <li:(2, 4)>]

You can define several patterns with a single regex method call.

>>> Rebulk().regex(r'Wint\wr', 'com\w{3}').matches("Winter is coming...")
[<Winter:(0, 6)>, <coming:(10, 16)>]

All keyword arguments from re.compile are supported.

>>> import re  # import required for flags constant
>>> Rebulk().regex('L[A-Z]KERS', flags=re.IGNORECASE) \
...         .matches("The LaKeRs are from La")
[<LaKeRs:(4, 10)>]

>>> Rebulk().regex('L[A-Z]', 'L[A-Z]KERS', flags=re.IGNORECASE) \
...         .matches("The LaKeRs are from La")
[<La:(20, 22)>, <LaKeRs:(4, 10)>]

>>> Rebulk().regex(('L[A-Z]', re.IGNORECASE), ('L[a-z]KeRs')) \
...         .matches("The LaKeRs are from La")
[<La:(20, 22)>, <LaKeRs:(4, 10)>]

If regex module is available, it automatically supports repeated captures.

>>> # If regex module is available, repeated_captures is True by default.
>>> matches = Rebulk().regex(r'(\d+)(?:-(\d+))+').matches("01-02-03-04")
>>> matches[0].children # doctest:+SKIP
[<01:(0, 2)>, <02:(3, 5)>, <03:(6, 8)>, <04:(9, 11)>]

>>> # If regex module is not available, or if repeated_captures is forced to False.
>>> matches = Rebulk().regex(r'(\d+)(?:-(\d+))+', repeated_captures=False) \
...                   .matches("01-02-03-04")
>>> matches[0].children
[<01:(0, 2)>, <04:(9, 11)>]
  • abbreviations

    Defined as a list of 2-tuple, each tuple is an abbreviation. It simply replace tuple[0] with tuple[1] in the expression.

    >>> Rebulk().regex(r'Custom-separators', abbreviations=[("-", "[\W_]+")])\
    ...         .matches("Custom_separators using-abbreviations")
    [<Custom_separators:(0, 17)>]
    

Functional Patterns

Functional Patterns are based on the evaluation of a function.

The function should have the same parameters as Rebulk.matches method, that is the input string, and must return at least start index and end index of the Match object.

>>> def func(string):
...     index = string.find('?')
...     if index > -1:
...         return 0, index - 11
>>> Rebulk().functional(func).matches("Why do simple ? Forget about it ...")
[<Why:(0, 3)>]

You can also return a dict of keywords arguments for Match object.

You can define several patterns with a single functional method call, and function used can return multiple matches.

Patterns parameters

All patterns have options that can be given as keyword arguments.

  • validator

    Function to validate Match value given by the pattern. Can also be a dict, to use validator with pattern named with key.

    >>> def check_leap_year(match):
    ...     return int(match.value) in [1980, 1984, 1988]
    >>> matches = Rebulk().regex(r'\d{4}', validator=check_leap_year) \
    ...                   .matches("In year 1982 ...")
    >>> len(matches)
    0
    >>> matches = Rebulk().regex(r'\d{4}', validator=check_leap_year) \
    ...                   .matches("In year 1984 ...")
    >>> len(matches)
    1

Some base validator functions are available in rebulk.validators module. Most of those functions have to be configured using functools.partial to map them to function accepting a single match argument.

  • formatter

    Function to convert Match value given by the pattern. Can also be a dict, to use formatter with matches named with key.

    >>> def year_formatter(value):
    ...     return int(value)
    >>> matches = Rebulk().regex(r'\d{4}', formatter=year_formatter) \
    ...                   .matches("In year 1982 ...")
    >>> isinstance(matches[0].value, int)
    True
  • name

    The name of the pattern. It is automatically passed to Match objects generated by this pattern.

  • tags

    A list of string that qualifies this pattern.

  • value

    Override value property for generated Match objects.

  • validate_all

    By default, validator is called for returned Match objects only. Enable this option to validate them all, parent and children included.

  • format_all

    By default, formatter is called for returned Match values only. Enable this option to format them all, parent and children included.

  • disabled

    A function(context) to disable the pattern if returning True.

  • children

    If True, all children Match objects will be retrieved instead of a single parent Match object.

  • private

    If True, Match objects generated from this pattern are available internally only. They will be removed at the end of Rebulk.matches method call.

  • private_parent

    Force parent matches to be returned and flag them as private.

  • private_children

    Force children matches to be returned and flag them as private.

  • marker

    If true, Match objects generated from this pattern will be markers matches instead of standard matches. They won’t be included in Matches sequence, but will be available in Matches.markers sequence (see Markers section).

Match

A Match object is the result created by a registered pattern.

It has a value property defined, and position indices are available through start, end and span properties.

In some case, it contains children Match objects in children property, and each child Match object reference its parent in parent property. Also, a name property can be defined for the match.

If groups are defined in a Regular Expression pattern, each group match will be converted to a single Match object. If a group has a name defined ((?P<name>group)), it is set as name property in a child Match object. The whole regexp match (re.group(0)) will be converted to the main Match object, and all subgroups (1, 2, … n) will be converted to children matches of the main Match object.

>>> matches = Rebulk() \
...         .regex(r"One, (?P<one>\w+), Two, (?P<two>\w+), Three, (?P<three>\w+)") \
...         .matches("Zero, 0, One, 1, Two, 2, Three, 3, Four, 4")
>>> matches
[<One, 1, Two, 2, Three, 3:(9, 33)>]
>>> for child in matches[0].children:
...     '%s = %s' % (child.name, child.value)
'one = 1'
'two = 2'
'three = 3'

It’s possible to retrieve only children by using children parameters. You can also customize the way structure is generated with every, private_parent and private_children parameters.

>>> matches = Rebulk() \
...         .regex(r"One, (?P<one>\w+), Two, (?P<two>\w+), Three, (?P<three>\w+)", children=True) \
...         .matches("Zero, 0, One, 1, Two, 2, Three, 3, Four, 4")
>>> matches
[<1:(14, 15)+name=one>, <2:(22, 23)+name=two>, <3:(32, 33)+name=three>]

Match object has the following properties that can be given to Pattern objects

  • formatter

    Function to convert Match value given by the pattern. Can also be a dict, to use formatter with matches named with key.

    >>> def year_formatter(value):
    ...     return int(value)
    >>> matches = Rebulk().regex(r'\d{4}', formatter=year_formatter) \
    ...                   .matches("In year 1982 ...")
    >>> isinstance(matches[0].value, int)
    True
  • format_all

    By default, formatter is called for returned Match values only. Enable this option to format them all, parent and children included.

  • conflict_solver

    A function(match, conflicting_match) used to solve conflict. Returned object will be removed from matches by conflict_prefer_longer processor. If __default__ string is returned, it will fallback to default behavior keeping longer match.

Matches

A Matches object holds the result of Rebulk.matches method call. It’s a sequence of Match objects and it behaves like a list.

All methods accepts a predicate function to filter Match objects using a callable, and an index int to retrieve a single element from default returned matches.

It has the following additional methods and properties on it.

  • starting(index, predicate=None, index=None)

    Retrieves a list of Match objects that starts at given index.

  • ending(index, predicate=None, index=None)

    Retrieves a list of Match objects that ends at given index.

  • previous(match, predicate=None, index=None)

    Retrieves a list of Match objects that are previous and nearest to match.

  • next(match, predicate=None, index=None)

    Retrieves a list of Match objects that are next and nearest to match.

  • tagged(tag, predicate=None, index=None)

    Retrieves a list of Match objects that have the given tag defined.

  • named(name, predicate=None, index=None)

    Retrieves a list of Match objects that have the given name.

  • range(start=0, end=None, predicate=None, index=None)

    Retrieves a list of Match objects for given range, sorted from start to end.

  • holes(start=0, end=None, formatter=None, ignore=None, predicate=None, index=None)

    Retrieves a list of hole Match objects for given range. A hole match is created for each range where no match is available.

  • conflicting(match, predicate=None, index=None)

    Retrieves a list of Match objects that conflicts with given match.

  • chain_before(self, position, seps, start=0, predicate=None, index=None):

    Retrieves a list of chained matches, before position, matching predicate and separated by characters from seps only.

  • chain_after(self, position, seps, end=None, predicate=None, index=None):

    Retrieves a list of chained matches, after position, matching predicate and separated by characters from seps only.

  • at_match(match, predicate=None, index=None)

    Retrieves a list of Match objects at the same position as match.

  • at_span(span, predicate=None, index=None)

    Retrieves a list of Match objects from given (start, end) tuple.

  • at_index(pos, predicate=None, index=None)

    Retrieves a list of Match objects from given position.

  • names

    Retrieves a sequence of all Match.name properties.

  • tags

    Retrieves a sequence of all Match.tags properties.

  • to_dict()

    Convert to a standard dict, with Match.name as key and Match.value as value. If several distinct values are found for the same name, value will be a list.

    It’s a subclass of dict, that contains an additional matches property which is a dict with Match.name as key and list of Match objects as value.

  • markers

    A custom Matches sequences specialized for markers matches (see below)

Markers

If you have defined some patterns with markers property, then Matches.markers points to a special Matches sequence that contains only markers matches. This sequence supports all methods from Matches.

Markers matches are not intended to be used in final result, but can be used to implement custom Processor or Rule.

Processors

Processors are functions that can be registered to Rebulk object with processor method.

All registered processors will be executed sequentially to modify the default sequence of Match returned by patterns, at the beginning of the Rebulk.matches process.

Rebulk embeds some processors in processors module.

conflict_prefer_longer (enabled by default) is used to keep only longer matches when several matches shares the same characters.

Default processors can be disabled when creating Rebulk object with default argument set to False.

>>> bulk = Rebulk(default=False).string('la', 'lakers')
>>> bulk.matches("the lakers are from la")
[<la:(4, 6)>, <la:(20, 22)>, <lakers:(4, 10)>]

You can also register processors with post_processor method. They will be executed at the very end of the Rebulk.matches process, after Rules execution.

Rules

Rules are a convenient and readable way to implement advanced conditional logic involving several Match objects. When a rule is triggered, it can perform an action on Matches object, like filtering out, adding additional tags or renaming.

Rules are implemented by extending the abstract Rule class. They are registered using Rebulk.rule method by giving either a Rule instance, a Rule class or a module containing Rule classes only.

For a rule to be triggered, Rule.when method must return True, or a non empty list of Match objects, or any other truthy object. When triggered, Rule.then method is called to perform the action with when_response parameter defined as the response of Rule.when call.

Instead of implementing Rule.then method, you can define consequence class property with a Consequence classe or instance, like RemoveMatch, RenameMatch or AppendMatch. You can also use a list of consequence when required : when_response must then be iterable, and elements of this iterable will be given to each consequence in the same order.

When many rules are registered, it can be useful to set priority class variable to define a priority integer between all rule executions (higher priorities will be executed first). You can also define dependency to declare another Rule class as dependency for the current rule, meaning that it will be executed before.

For all rules with the same priority value, when is called before, and then is called after all.

>>> from rebulk import Rule, RemoveMatch

>>> class FirstOnlyRule(Rule):
...     consequence = RemoveMatch
...
...     def when(self, matches, context):
...         grabbed = matches.named("grabbed", 0)
...         if grabbed and matches.previous(grabbed):
...             return grabbed

>>> rebulk = Rebulk()

>>> rebulk.regex("This match(.*?)grabbed", name="grabbed")
<...Rebulk object ...>
>>> rebulk.regex("if it's(.*?)first match", private=True)
<...Rebulk object at ...>
>>> rebulk.rules(FirstOnlyRule)
<...Rebulk object at ...>

>>> rebulk.matches("This match is grabbed only if it's the first match")
[<This match is grabbed:(0, 21)+name=grabbed>]
>>> rebulk.matches("if it's NOT the first match, This match is NOT grabbed")
[]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

rebulk-0.5.0.tar.gz (248.1 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page