Skip to main content

Binary delta encoding tools.

Project description

buildstatus appveyor coverage codecov

About

Binary delta encoding in Python 3 and C.

Based on http://www.daemonology.net/bsdiff/, with the following differences:

  • BZ2, LZMA, Zstandard, heatshrink or CRLE compression.

  • Linear patch file access pattern to allow streaming and less RAM usage.

  • SA-IS instead of qsufsort.

  • Variable length size fields.

  • Incremental apply patch implemented in C, suitable for memory constrained embedded devices.

  • Normal or in-place (resumable) updates.

  • Optional experimental data format aware algorithm for potentially smaller patches. I don’t recommend anyone to use this functionality as the gain is small in relation to memory usage and code complexity!

    There is a risk this functionality uses patent https://patents.google.com/patent/EP1988455B1/en. Anyway, this patent expires in August 2019 as I understand it.

    Supported data formats:

    • ARM Cortex-M4
    • AArch64

Project homepage: https://github.com/eerimoq/detools

Documentation: http://detools.readthedocs.org/en/latest

Installation

pip install detools

Statistics

“LZMA ref.” is the target binary compressed with lzma --best.

The percentages are calculated as “patch size” / “to size”. Lower is better.

Update To size LZMA ref. LZMA heatshrink CRLE
upy v1.9.4 -> v1.10 615388 59.8 % 11.7 % 15.7 % 26.2 %
python 3.5 -> 3.6 4568920 30.7 % 31.8 % - -
foo old -> new 2780 69.5 % 4.5 % 4.5 % 6.8 %

Example usage

Examples in C are found in src/c.

Command line tool

The create patch subcommand

Create a patch foo.patch from tests/files/foo/old to tests/files/foo/new.

$ detools create_patch tests/files/foo/old tests/files/foo/new foo.patch
Successfully created patch 'foo.patch'!
$ ls -l foo.patch
-rw-rw-r-- 1 erik erik 127 Mar  1 19:18 foo.patch

Create the same patch as above, but without compression.

$ detools create_patch --compression none \
      tests/files/foo/old tests/files/foo/new foo-no-compression.patch
Successfully created patch 'foo-no-compression.patch'!
$ ls -l foo-no-compression.patch
-rw-rw-r-- 1 erik erik 2792 Mar  1 19:18 foo-no-compression.patch

Create an in-place patch foo-in-place.patch.

$ detools create_patch --type in-place --memory-size 3000 --segment-size 500 \
      tests/files/foo/old tests/files/foo/new foo-in-place.patch
Successfully created patch 'foo-in-place.patch'!
$ ls -l foo-in-place.patch
-rw-rw-r-- 1 erik erik 672 Mar 16 08:49 foo-in-place.patch

Create a bsdiff patch foo-bsdiff.patch, compatible with the original bsdiff program.

$ detools create_patch --type bsdiff \
      tests/files/foo/old tests/files/foo/new foo-bsdiff.patch
Successfully created patch 'foo-bsdiff.patch'!
$ ls -l foo-bsdiff.patch
-rw-rw-r-- 1 erik erik 261 Apr 22 18:20 foo-bsdiff.patch

The apply patch subcommand

Apply the patch foo.patch to tests/files/foo/old to create foo.new.

$ detools apply_patch tests/files/foo/old foo.patch foo.new
$ ls -l foo.new
-rw-rw-r-- 1 erik erik 2780 Mar  1 19:18 foo.new

The in-place apply patch subcommand

Apply the in-place patch foo-in-place.patch to foo.mem.

$ cp tests/files/foo/old foo.mem
$ detools apply_patch_in_place foo.mem foo-in-place.patch
$ ls -l foo.mem
-rwxrwxr-x 1 erik erik 2780 Mar 16 08:51 foo.mem

The bsdiff apply patch subcommand

Apply the patch foo-bsdiff.patch to tests/files/foo/old to create foo.new.

$ detools apply_patch_bsdiff tests/files/foo/old foo-bsdiff.patch foo.new
$ ls -l foo.new
-rw-rw-r-- 1 erik erik 2780 Mar  1 19:18 foo.new

The patch info subcommand

Print information about the patch foo.patch.

$ detools patch_info foo.patch
Type:               normal
Patch size:         127 bytes
To size:            2.71 KiB
Patch/to ratio:     4.6 % (lower is better)
Diff/extra ratio:   9828.6 % (higher is better)
Size/data ratio:    0.3 % (lower is better)
Compression:        lzma

Number of diffs:    2
Total diff size:    2.69 KiB
Average diff size:  1.34 KiB
Median diff size:   1.34 KiB

Number of extras:   2
Total extra size:   28 bytes
Average extra size: 14 bytes
Median extra size:  14 bytes

Contributing

  1. Fork the repository.

  2. Install prerequisites.

    pip install -r requirements.txt
    
  3. Implement the new feature or bug fix.

  4. Implement test case(s) to ensure that future changes do not break legacy.

  5. Run the tests.

    make test
    
  6. Create a pull request.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for detools, version 0.32.0
Filename, size File type Python version Upload date Hashes
Filename, size detools-0.32.0.tar.gz (10.9 MB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page