Small string compression using smaz, supports Python 3.
Small string compression using smaz compression algorithm.
This library wraps the original C code, so it should be quite fast. It also has a testsuite that uses hypothesis based property testing - a fancy way of saying that the tests are run with randomly generated strings using most of unicode, to better guard against edge cases.
Why do I need this?
You are working with tons of short strings (text messages, urls,...) and want to save space.
According to the original code and notes, it achieves best compression with english strings (up to 50%) that do not contain a ton of numbers. However, any other language might just work as well (allegedly still up to 30%).
Note that in certain cases it is possible that the compression increases the size. Keep that in mind and maybe first run some tests. Measuring size is explained in the example below as well.
How do I use this?
$ pip install smaz-py3
-py3 is important. There is an original release, kudos to Benjamin
Sergeant, but it does not work with Python 3+.
Now, a usage example.
import smaz # First we compress our example sentence. compressed = smaz.compress("The quick brown fox jumps over the lazy dog.") # The output is raw bytes. As can be seen in the decompress() call below. # Now, we decompress these raw bytes again. This should return our example sentence. decompressed = smaz.decompress(b'H\x00\xfeq&\x83\xfek^sA)\xdc\xfa\x00\xfej&-<\x95\xe7\r\x0b\x89\xdbG\x18\x06;n') # This does not fail, which means we have successfully compressed and decompressed # without damaging anything. assert decompressed == "The quick brown fox jumps over the lazy dog."
How much did we compress?
# First, we get the actual byte size of our example string. original_size = len("The quick brown fox jumps over the lazy dog.".encode("utf-8")) # 44 bytes # As `compressed` is already raw bytes, we can also call len() on this compressed_size = len(compressed) # 31 bytes compression_ratio = 1 - compressed_size / original_size # 0.295
So we saved about 30% (0.295 * 100 and some rounding 😉).
If the compression ratio would be below 0, we would have actually increased the string. Yes, this can happen. Again, smaz works best on small strings.
A small note about NULL bytes
smaz-py3 does not support strings with NULL bytes (
\x00) in compression:
>>> import smaz >>> smaz.compress("The quick brown fox\x00 jumps over the lazy dog.") Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: embedded null character
My reasoning behind this is that in most scenarios you want to clean that away beforehand anyways. If you think this is wrong, please open up an issue on github. I am happy for further input!
Migrating from Python 2
If you have been using the Python 2
this Python 3 version exposes the same API, so it is a drop-in replacement.
Important: While developing this extension, I think I found a bug in the original library. Using Python 2.7.16:
>>> import smaz >>> smaz.compress("The quick brown fox jumps over the lazy dog.") 'H' # this is wrong. >>> small = smaz.compress("The quick brown fox jumps over the lazy dog.") >>> smaz.decompress(small) 'The' # information lost.
So, if you are actually upgrading from this, please make sure that you are not
affected by this.
smaz-py3 is not prone to this bug.
Behind the scenes, smaz uses NULL bytes in compression. However, when converting from
C back to a Python string object, NULL is used to mark the end of the string. The
above sentence, compressed, has the NULL byte right after the
That's why it stops right then and there. Again,
smaz-py3 is not affected by this,
mostly because I got lucky in choosing this example sentence.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size smaz-py3-1.0.1.tar.gz (20.6 kB)||File type Source||Python version None||Upload date||Hashes View hashes|