Skip to main content

An experimental diff library for generating operation deltas that represent the difference between two sequences of comparable items.

Project description

Deltas

An open licensed (MIT) library for performing generating deltas (A.K.A sequences of operations) representing the difference between two sequences of comparable tokens.

This library is intended to be used to make experimental difference detection strategies more easily available. There are currently two strategies available:

deltas.sequence_matcher.diff(a, b):
A shameless wrapper around difflib.SequenceMatcher to get it to work within the structure of deltas.

deltas.segment_matcher.diff(a, b, segmenter=None):
A generalized difference detector that is designed to detect block moves and copies based on the use of a Segmenter.

Example:

from deltas import segment_matcher, text_split
a = text_split.tokenize("This is some text. This is some other text.")`|
b = text_split.tokenize("This is some other text. This is some text.")
operations = segment_matcher.diff(a, b)

for op in operations:
 print(op.name, repr(''.join(a[op.a1:op.a2])),
  repr(''.join(b[op.b1:op.b2])))

equal 'This is some other text.' 'This is some other text.'
insert ' ' ' '
equal 'This is some text.' 'This is some text.'
delete ' ' ''

Tokenization

By default Deltas performs tokenization by regexp text splitting. We included CJK tokenization functionality. If text consists of at least 1/4 (default value) Japanse or Korean symbols it is tokenized by language specific Tokenizer. Else, Chinese Tokenizer is used.

  • Chinese Tokenizer - Jieba
  • Japanese Tokenizer - Sudachi
  • Korean Tokenizer - KoNLPy(Okt)

Tokenization example:

import mwapi
import deltas
import deltas.tokenizers

# example title ["China", "Haiku", "Kimchi"]: "中国" - Chinese(zh), "俳句" - Japanese(ja), "김치" - Korean(ko)
session = mwapi.Session("https://zh.wikipedia.org")
doc = session.get(action="query", prop="revisions", titles="中国", rvprop="content", rvslots="main",formatversion=2)
text = doc['query']['pages'][0]['revisions'][0]['slots']['main']['content']

# text processed only by regexp tokenizer
tokenized_text = deltas.tokenizers.wikitext_split.tokenize(text)
# text processed regexp tokenizer with cjk post processing
tokenized_text_cjk = deltas.tokenizers.wikitext_split_w_cjk.tokenize(text)

FOR IMPROVED JAPANESE TOKENIZER ACCURACY PLEASE INSTALL FULL DICTIONARY:

pip install sudachidict_full
# and link sudachi to dict
sudachipy link -t full

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for deltas, version 0.6.2
Filename, size File type Python version Upload date Hashes
Filename, size deltas-0.6.2-py2.py3-none-any.whl (35.3 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size deltas-0.6.2.tar.gz (25.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page