Fast low color mask read/write
Project description
pyfastmask - Fast image segmentation format
This is a simple format for storing single channel images with low-frequency data (e.g. semantic segmentation masks).
It has a size similar to PNG, but is much faster (up to 20x) to read.
Installation
From PyPI:
The easiest way to install the latest version is by using pip:
pip install pyfastmask
From source:
git clone git@github.com:JIy3AHKO/pyfastmask.git
cd pyfastmask
pip install -e .
Usage
For image reading and writing, use the read
and write
functions:
import numpy as np
import pyfastmask as pf
img = np.random.randint(0, 256, (100, 100), dtype=np.uint8)
pf.write('mask.pfm', img)
img2 = pf.read('mask.pfm')
np.testing.assert_array_equal(img, img2)
Benchmark
See BENCHMARK.md for more detailed information.
Image | pyfastmask | opencv png | cv2_bmp | qoi |
---|---|---|---|---|
Median Read Time | 0.09 ms | 1.71 ms | 0.35 ms | 0.81 ms |
Average Size | 217.35 KiB | 149.36 KiB | 1146.64 KiB | 498.24 KiB |
Format Description
The pyfastmask efficiently compresses and stores image segmentation masks using Run-Length Encoding (RLE) and line-differential encoding. All values are stored with different bit widths, depending on the number of unique symbols and the mask size - it helps to reduce the size of encoded data.
Storage Structure
The format organizes data into three main sections: header, symbol mapping, and line-by-line encoded data.
1. Header
- Magic Byte: Format identifier.
- Version Byte: Format version.
- Symbol Bit Width: Bits for each symbol.
- Count Bit Width: Bits for run lengths.
- Line Count Bit Width: Bits for the number of runs per line.
- Unique Symbols Count: Number of unique symbols.
- Mask Height: Mask height in pixels.
- Mask Width: Mask width in pixels.
2. Symbol Mapping
Lists unique symbols in the mask, each encoded with 8 bits.
Semantic segmentation masks usually have a small number of unique symbols, so we can use a small number of bits to encode each symbol.
3. Line-by-Line Encoding
Encodes mask data line by line:
- First Line: Encoded with standard RLE.
- Subsequent Lines: Encoded with sparse RLE on the difference between the current and previous lines.
First line is represented as: (Number of runs), (Symbol, Run Length), (Symbol, Run Length), ...
Subsequent lines are represented as: (Number of runs), (Offset, Symbol, Run Length), (Offset, Symbol, Run Length), ...
Where:
- Number of runs: Number of runs in the line.
- Symbol: Symbol index from the symbol mapping.
- Run Length: Number of pixels with the same symbol.
- Offset: Number of pixels to skip from the previous line.
Encoding Process
- Encode Lines:
- First Line: Standard RLE.
- Subsequent Lines: Sparse RLE.
- Estimate Bit Widths: Calculate the number of bits required to store each value.
- Write Header and Symbol Mapping.
- Pack Data: Combine all encoded data into a byte stream.
Decoding Process
- Read Header and Symbol Mapping.
- Decode first line with standard RLE.
- On subsequent lines:
- copy the previous line
- apply sparse RLE to the symbols which are differ from the previous line
Testing
To run tests, use the following command:
python -m unittest discover tests/
Contributing
Contributions are welcome! If you want to contribute, please create an issue or a pull request.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyfastmask-1.1.1-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15a16f90476c6f71dc85ae07a850b5c2b31b13ad283202b0ce90ca0826d89338 |
|
MD5 | b85a4195be2f52eb117b8689b0db25f2 |
|
BLAKE2b-256 | 3c10b995908bdeabc9027c37d47081d5100dbd67ba671576d2a028033c961a3c |
Hashes for pyfastmask-1.1.1-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7817d104493e203317d896c1be13d56e96ccdfdf22addf2aa0c68101396ffc18 |
|
MD5 | 5755b18b27a5640a7ba1a265d99f3de4 |
|
BLAKE2b-256 | 7cec0dd0a2a479b4e4629dc2bdae50a32ff48eaeae215a125baae0aba7d2a40a |
Hashes for pyfastmask-1.1.1-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a64900312d413051faa3a914ad4a5ece589d1037561f65a8520fed53eb2010e |
|
MD5 | ec0cb81ff575f8b827b5e729e8aefedf |
|
BLAKE2b-256 | 7889015a501131796d145c94388123620069a22b388a040f6febe7afe765f2fd |
Hashes for pyfastmask-1.1.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e52ac104e03365e4ddfcc422f51630aaffd32f11fff8a455ecb57b71449448b6 |
|
MD5 | d7850b31bba755029ff1e33560bd4d36 |
|
BLAKE2b-256 | b3e3d9311401d4aa363365b99e6fb17debc538c0ad9c7ecb4383bf862145c6c0 |
Hashes for pyfastmask-1.1.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 682af3f5b14ea3c120c973421baf9c76e622913c990a026035c11a02fd2e4214 |
|
MD5 | 4d8f6865f69f433e102423ae2bc625e3 |
|
BLAKE2b-256 | f0186efea656d709f8547bd67aa96b8a1aede04fb98f987473aa38aca045b315 |
Hashes for pyfastmask-1.1.1-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25a8d65cdf0faf7a2afb5cb411da3a9c56ff1de2e7df81a524157a3209295134 |
|
MD5 | 3b27c494cad1b036b9dd0da5253d2128 |
|
BLAKE2b-256 | 33f0e8a8b5b5eaf915aaae46a0ff9f92b91f76ab41f7bb5d06dc337138a93fa5 |
Hashes for pyfastmask-1.1.1-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be4994873055bc742fd8313a71a49f70d8dd121f87a3ffa8057c32984da0a563 |
|
MD5 | 92ed071e46b1224fc4297a8c281b08f6 |
|
BLAKE2b-256 | 43774d00788ad1ce99aad90cdd02db98c698fe35276b4042b5e985003bba33d8 |
Hashes for pyfastmask-1.1.1-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43758c9a166cf13a6d2e721a8bd0925dfa03c19f11b658f1cc62aa403a2a886a |
|
MD5 | 14dc20d6955e8c608197368e398efb37 |
|
BLAKE2b-256 | 3dc4b442cd75c0fa4e8f4619094c7ff940ac1bd068715c211c454bc881f9edba |
Hashes for pyfastmask-1.1.1-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | acbfb18aaea1ecdc3555a57af4ba74c67eec63f906df9380ac5b38ed87c5ce75 |
|
MD5 | 102a9856389e7c0711a9f645088c606a |
|
BLAKE2b-256 | c5aab76996a436a78569a29e51e54abf0969e1fcfa1f899f348997ee4a3d7ad7 |
Hashes for pyfastmask-1.1.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e1e6811c5d4c76498807301c38ca80e6160471ba722425729c63b91727d3c76 |
|
MD5 | ff1667c5392506f6c2969d6a2997e4d3 |
|
BLAKE2b-256 | 48a9d3b847244ad0deeb339d8ee1f8ec6013353c6d25ecf429f882b9b6dda743 |
Hashes for pyfastmask-1.1.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 303cd1ac94738c7c356d72b96bec94921702a1b92da3079556877342a5fce955 |
|
MD5 | 2daea4a4b6434cea03bf32c2faf42665 |
|
BLAKE2b-256 | 238556e6fc8fcb83d3cd6139dba34158e06165bedbea358065c65749061ed458 |
Hashes for pyfastmask-1.1.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed629ea167bede0315ccebd73153e14fe07e0cf2294cb93f7e536b4be2bb2437 |
|
MD5 | 2bbc117145510f9a6e3ab5fa4c9837c9 |
|
BLAKE2b-256 | c25cb27fbb9ccf12d4fe20e067bc68f554a6211822f70dcdf9dc298ec7e45d8d |
Hashes for pyfastmask-1.1.1-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef7282e644e7c010b65168f3a0e87d5d284af8f00285bccd5ad334bcd94c58a2 |
|
MD5 | e25accab28a9441462c873f878083622 |
|
BLAKE2b-256 | ee26315102da69e43b4473576bab284ecb5c6f489e562915394055c2ec0e6a72 |
Hashes for pyfastmask-1.1.1-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13b3993155336ec71b4234404729d13accd37a4cf82a8eed0754b978bc9e391a |
|
MD5 | f43dfa650d3b850f7d3bc9a670569759 |
|
BLAKE2b-256 | f92371594a1d2fe5e2145fa21a25962addb6109fcf945c414226771b7848387c |
Hashes for pyfastmask-1.1.1-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94aebb2293ba5231d4069cc735277f9ba888969a6198189ba09526e1e5a34dc3 |
|
MD5 | b701b28c2b4b5b2e860dd3127cfa177c |
|
BLAKE2b-256 | dbff58d6ffb1f6bc14eacd163b6511dd6ad9b2c4c140272382591bc33b7b94aa |
Hashes for pyfastmask-1.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7c2f47c6a94c809773f75fd2202423917aa83f2aa088602d8af6913729f17c2 |
|
MD5 | 63f51300749cdf7966f521719155dd33 |
|
BLAKE2b-256 | 4fde743d407cd1a1a3427db54355fb28f69139996c096108e6143c623dbc523d |
Hashes for pyfastmask-1.1.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 586e83c0da16c715fedccda6c19141a65dc85c6ccdcb1c0fc239a0244fdc6908 |
|
MD5 | 2c7a77e6676d07917788d717708cd6eb |
|
BLAKE2b-256 | 12c23fb6c4dbea0862a3fd56b094c6acee23c66cfd6dc7a2abfc0dd2151d362f |
Hashes for pyfastmask-1.1.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33fb1e16f2ffaf61b0139c00e3e089003f231bfdd45b460bc37977be91cca023 |
|
MD5 | 9d7caac5a5d8c38370443641586807b3 |
|
BLAKE2b-256 | 57471071128325fa6cb987edc4ee47ed7119778f1120d2900714cd1cfa35b029 |
Hashes for pyfastmask-1.1.1-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ef13f14022b4fab1be5c554e075e448591ac693031aaa38531c11c59ee580baf |
|
MD5 | 0478c5f5c498449e9e3f7791b22a590f |
|
BLAKE2b-256 | ffe8e85b4e84a43c5f91863c16e0ddf07463b81b8e6a62dea54516b738233f47 |
Hashes for pyfastmask-1.1.1-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d4ae1a0385bc54ff217ce8a4d6ba203cebc7c18b589a4ef64d08837436b02a0d |
|
MD5 | 199c9a753b060048153ff48ef76a5ff4 |
|
BLAKE2b-256 | dccb30ad0eb3d9076a40eedb7bec832f8643d163f2168b4a0a5503409addd41f |
Hashes for pyfastmask-1.1.1-cp39-cp39-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec1d7fb8c87fe50fe98b7dd0da871d0140ccd4816aebec845356a864fed862e1 |
|
MD5 | 2cacade8c2cc0f595b38d54c06694f7e |
|
BLAKE2b-256 | 272ff2af8d7595bf3644fa5a9f546155755a5710afaa5d57129890bab08442bc |
Hashes for pyfastmask-1.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 648bee3aeceff950424abcc9f501c9f756ef6ad4e2cce9b71ad16fff54bbe674 |
|
MD5 | e824333367ec607482e19ecb7f236058 |
|
BLAKE2b-256 | 773bb5ed27b96773dae17427fea688588e698bc17bbb40575f16a235427d75a4 |
Hashes for pyfastmask-1.1.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3a2aac48ad1afe1502a23d90a336c39e1125d45c8e3f4b7089f775cf9c16d94 |
|
MD5 | 92db9a16561fd0b0ce29de6747a7983e |
|
BLAKE2b-256 | ada42324824a299aca3e4e550cf7bfcc021b5187e182b18e86e237639c5f8ffb |
Hashes for pyfastmask-1.1.1-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c00ddb4b3fd193226c9cad7912d49c7af835d12c369d6d03fd4bf9d1d4ffb8f |
|
MD5 | 08327ae0f25aac3b5163b8b019a467e5 |
|
BLAKE2b-256 | 7619e549a43bc28583921a464af21c3f41038aad0e55a43b15e68e696d99a93c |