imohash-rs 0.1.0

Fast hashing for large files

imohash_rs

imohash_rs is a fast, constant-time hashing library for Python, and application to hash files from the command line. It uses file size and sampling to calculate hashes quickly, regardless of file size.

---

imohash_rs is built on top of Rust component: hiql/imohash.

Alternative implementations

• Java: https://github.com/dynatrace-oss/hash4j
• JavaScript/TypeScript: https://github.com/clockwork-dog/imohash
• Python: https://github.com/kalafut/py-imohash
• Rust: https://github.com/hiql/imohash
  • Python: https://github.com/unsektor/py-imohash (this one)

Algorithm

Consult the documentation for more information.

Misuses

Because imohash only reads a small portion of a file's data, it is not suitable for:

• file verification or integrity monitoring
• cases where fixed-size files are manipulated
• anything cryptographic

The original project created by Jim Kalafut, check out https://github.com/kalafut/imohash

Installation

pip install imohash_rs

Architecture overview

Benchmark

Comparison with imohash

imohash is a package that provides native python implementation of imohash algorithm (see https://github.com/kalafut/py-imohash).

See bench.ipynb for details:

Initial launch

Graphics analyze reveals, that native implementation (imohash) is faster in case of sequent files processing.

Usage

Component provides Python API (for usage in code), and CLI application.

Python API

import imohash_rs

imohash = imohash_rs.Imohash()
# Adjust hash settings with `sample_threshold` and `sample_size` parameters:
# imohash = imohash_rs.Imohash(sample_threshold=imohash_rs.SAMPLE_THRESHOLD, sample_size=imohash_rs.SAMPLE_SIZE)

if True:  # ... calculate hash for file 
   hash_ = imohash.get_for_file(path=imohash_rs.__file__)
   # Hash(value=173061995066865536926914513679577055882)

   hash_.hexdigest()
   # '0c246e9df1ac5b0d4536efbe96244aa8'

if True:  # ... calculate hash for bytes data
   hash_ = imohash.get(data=b'example data')
   # Hash(value=223695275369458893910658172564464870412)

   hash_.hexdigest()
   # '0c246e9df1ac5b0d4536efbe96244aa8'

   hash_.digest()
   # b'\x0c$n\x9d\xf1\xac[\rE6\xef\xbe\x96$J\xa8'

CLI application

Component provides a CLI sample application to hash files, similar to md5sum:

python3 -m imohash_rs  # ... options and arguments

Application options:

• -t / --sample-threshold — Sample threshold value.
• -s / --sample-size — Sample size value. The entire file will be hashed (i.e. no sampling), if sample_size < 1
• -f / --format of { int | bytes | hex } — Hash representation format. Default hex
• -i / --interactive — Interactive hash computation mode. Conflicts with [file_path ...] argument
• [file_path ...] — File paths to compute hash of. Conflicts with -i/--interactive argument

Usage example:

1. Compute hash sum of file(s):

   # echo example > /tmp/my_file
   python3 -m imohash_rs /tmp/my_file
   

   prints:

   0877d8731ad98e5ee1cc09c0a87772bf  /tmp/my_file
   

2. Compute hash sum of file(s) with find application result:

   # dd if=/dev/random of=/tmp/1.iso bs=1M count=64
   # dd if=/dev/random of=/tmp/2.iso bs=1M count=64
   # cp /tmp/1.iso /tmp/3.iso
   
   find /tmp -type f -iname '*.iso' -exec python3 -m imohash_rs {} \+
   

   prints:

   808080203afea9085df78cd992f28546  /tmp/1.iso
   8080802011cdd41fbddd9c1f853c1330  /tmp/2.iso
   808080203afea9085df78cd992f28546  /tmp/3.iso  # as same as #1 !
   

3. Compute hash of string content (as bytes data) interactively:

   python3 -m imohash_rs -i  # ... or implicitly: python3 -m imohash_rs
   

   Interactive mode (format: hex)
   > example
   07ce528a343b2b99d4bd1bcdd648d138
   > example 2
   09b17440da02c7feb0b54f89d4d7b142
   >
   

Development

## initial setup
python3 -m venv venv
python3 -m pip install -r requirements.lock

## development
source venv/bin/activate
maturin develop
# maturin develop --release --strip

## build 
maturin build --release

Motivation

This was primarily written for me to learn Rust.

Changelog

License (MIT)