Rule-based Text Labeling Framework Aiming at Flexibility
Project description
seqlabel: Flexible Rule-based Text Labeling
seqlabel is a rule-based text labeling framework aiming at flexibility.
Installation
To install seqlabel:
pip install seqlabel
Requirements
- Python 3.8+
Usage
For a normal text
First, import some classes.
from seqlabel import Text
from seqlabel.matchers import DictionaryMatcher
from seqlabel.entity_filters import LongestMatchFilter, MaximizedMatchFilter
from seqlabel.serializers import IOB2Serializer
Initialize Text
by giving it a text you want to label over.
text = Text("Tokyo is the capital of Japan.")
Prepare matcher
matching supplied patterns. You can supply patterns via Hash Map mapping string sequences to the corresponding labels. You can define your own matcher by inheriting seqlabel.matchers.Matcher
.
Then, apply matcher.match
to text
.
# Preparing Matcher
matcher = DictionaryMatcher()
# Adding patterns
matcher.add({"Tokyo": "LOC", "Japan": "LOC"})
# Matching
entities = matcher.match(text)
Filter unwanted entities. LongestMatchFilter
removes overlapping entities and leaves longer entity. MaximizedMatchFilter
removes overlapping entities and leaves as many entities as possible. You can define your own filter by inheriting seqlabel.entity_filters.EntityFilter
.
filter_a = LongestMatchFilter()
filtered_entities_a = filter_a(entities)
filter_b = MaximizedMatchFilter()
filtered_entities_b = filter_b(entities)
Convert entities to IOB2 format after matching and filtering. Check seqlabel.serializers
out if you want to use other formats.
serializer = IOB2Serializer()
serializer.save(text, filtered_entities_a)
For a tokenized text
If you want to process a tokenized text, you need to use TokenizedText
instead of Text
. You could import it as follows:
from seqlabel import TokenizedText
Initialize TokenizedText
by giving it tokens
and space_after
you want to label over. tokens
is a list of strings and space_after
is a list of boolean indicating whether each token has a subsequent space.
tokenized_text = TokenizedText(
["Tokyo", "is", "the", "captial", "of", "Japan", "."],
[True, True, True, True, True, False, False]
)
You can use matcher
, filter
, and serializer
just like a normal text, as shown above.
# Mathcing
entities = matcher.match(tokenized_text)
# Filtering
filtered_entities = filter_a(entities)
# Serializing
serializer.save(tokenized_text, filtered_entities)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.