Fast low color mask read/write
Project description
pyfastmask - Fast image segmentation format
This is a simple format for storing single channel images with low-frequency data (e.g. semantic segmentation masks).
It has a size similar to PNG, but is much faster (up to 20x) to read.
Installation
From PyPI:
The easiest way to install the latest version is by using pip:
pip install pyfastmask
From source:
git clone git@github.com:JIy3AHKO/pyfastmask.git
cd pyfastmask
pip install -e .
Usage
For image reading and writing, use the read
and write
functions:
import numpy as np
import pyfastmask as pf
img = np.random.randint(0, 256, (100, 100), dtype=np.uint8)
pf.write('mask.pfm', img)
img2 = pf.read('mask.pfm')
np.testing.assert_array_equal(img, img2)
Benchmark
See BENCHMARK.md for more detailed information.
Image | pyfastmask | opencv png | cv2_bmp | qoi |
---|---|---|---|---|
Median Read Time | 0.09 ms | 1.71 ms | 0.35 ms | 0.81 ms |
Average Size | 217.35 KiB | 149.36 KiB | 1146.64 KiB | 498.24 KiB |
Format Description
The pyfastmask efficiently compresses and stores image segmentation masks using Run-Length Encoding (RLE) and line-differential encoding. All values are stored with different bit widths, depending on the number of unique symbols and the mask size - it helps to reduce the size of encoded data.
Storage Structure
The format organizes data into three main sections: header, symbol mapping, and line-by-line encoded data.
1. Header
- Magic Byte: Format identifier.
- Version Byte: Format version.
- Symbol Bit Width: Bits for each symbol.
- Count Bit Width: Bits for run lengths.
- Line Count Bit Width: Bits for the number of runs per line.
- Unique Symbols Count: Number of unique symbols.
- Mask Height: Mask height in pixels.
- Mask Width: Mask width in pixels.
2. Symbol Mapping
Lists unique symbols in the mask, each encoded with 8 bits.
Semantic segmentation masks usually have a small number of unique symbols, so we can use a small number of bits to encode each symbol.
3. Line-by-Line Encoding
Encodes mask data line by line:
- First Line: Encoded with standard RLE.
- Subsequent Lines: Encoded with sparse RLE on the difference between the current and previous lines.
First line is represented as: (Number of runs), (Symbol, Run Length), (Symbol, Run Length), ...
Subsequent lines are represented as: (Number of runs), (Offset, Symbol, Run Length), (Offset, Symbol, Run Length), ...
Where:
- Number of runs: Number of runs in the line.
- Symbol: Symbol index from the symbol mapping.
- Run Length: Number of pixels with the same symbol.
- Offset: Number of pixels to skip from the previous line.
Encoding Process
- Encode Lines:
- First Line: Standard RLE.
- Subsequent Lines: Sparse RLE.
- Estimate Bit Widths: Calculate the number of bits required to store each value.
- Write Header and Symbol Mapping.
- Pack Data: Combine all encoded data into a byte stream.
Decoding Process
- Read Header and Symbol Mapping.
- Decode first line with standard RLE.
- On subsequent lines:
- copy the previous line
- apply sparse RLE to the copied line
Testing
To run tests, use the following command:
pip install -r test-requirements.txt
python -m unittest discover tests/
Contributing
Contributions are welcome! If you want to contribute, please create an issue or a pull request.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyfastmask-1.1.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e3f23d6f03babb80c67ded7d8e188a9f9754c70d0d77d7aa8f94eda4869fc80 |
|
MD5 | caf66801752c1c80cbdec04b735928cc |
|
BLAKE2b-256 | 313fecd198d747ab55ebf268ca0d641d31ec492f8dcef317363d02f7116a87eb |
Hashes for pyfastmask-1.1.0-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d64b800843426b690f1981b84aeca181e7cf29e5179af1bca9c054f6ace48c8 |
|
MD5 | a2ea4db81d2b62f6522cbfb673c31c8a |
|
BLAKE2b-256 | 50295baa96a0f08f6561a6de29aa7bf9be6c68d98b2a4f9bd6cff709e629eb10 |
Hashes for pyfastmask-1.1.0-cp312-cp312-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5965b1fae3c9648bfaba4d5c9d7944d162054cf8cdef983452629a31b78775ef |
|
MD5 | 89cccb9a34514e48484b0ab899062cb8 |
|
BLAKE2b-256 | 87b80a85d9efc27b076b654d08524696ae134143e5cfe78e725da72296f0789b |
Hashes for pyfastmask-1.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff15f85d7fd0b9e2456dad4652d718347f019005cc94c230d4de333dd055e175 |
|
MD5 | 02de1f9cb10f034b33b7476adf9b9d0f |
|
BLAKE2b-256 | 9d2f40793713d6c66a76a74e8c5727322299810b8231a7c1a7564785c5859d6f |
Hashes for pyfastmask-1.1.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5019b728ab16ddc297f6f495a3522ebbb278563807b7dda1dce85e84075f35a6 |
|
MD5 | 772cecec9d5820c8ec55b33b9bd6602e |
|
BLAKE2b-256 | 25e156b327b55e118feb36edb3c52a8010985b2d7f24cbc91dceaaafeb7a1760 |
Hashes for pyfastmask-1.1.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf63ddc5d3ca41322d2410972ba8da195677a48213f060b3615a8b69b8d7be35 |
|
MD5 | 78f493ee1d8b845206525a8d52cc6959 |
|
BLAKE2b-256 | 22641545f8c3c25f801d8dd3c02c5fbaf78f7066c7d55c383cffc32318bc7e56 |
Hashes for pyfastmask-1.1.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb6f93c65336f93a2c6a5c97b4e8ce6682445ccea402b52b3aa8d7b2868a3500 |
|
MD5 | d58d2036ce18182f422e08ce9fb6f430 |
|
BLAKE2b-256 | 55a94157d7c25eca9a645a127f1b5b51a05a554534cb7804668b241cfbb2ae72 |
Hashes for pyfastmask-1.1.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f8905737a471befe11b27c4b5b654004380d675247657ff6095d147638bcd63 |
|
MD5 | 5b0fdfc269222f5228ae4052d81ba5e5 |
|
BLAKE2b-256 | 1ac8192c2cc10052e1c7aa05e587f96abcd2c7cf4590fc2ebb94ea58597a51fe |
Hashes for pyfastmask-1.1.0-cp311-cp311-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d10111186eceb1521eecd5b0432fac054c10338c381a459cc30237397b0dfd31 |
|
MD5 | d916dd537e63fb34f32c41679284b9ad |
|
BLAKE2b-256 | b0064624e81c8cb804f14bbe3c7095d8af63e25b4475d871ec1a61741fbb85d0 |
Hashes for pyfastmask-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4341cbd66910b91034d5a0933c3b21e112b88a2b18c7cec78997b88843068f33 |
|
MD5 | 7b48ec6a78cdd6cb56473d17a372a1ec |
|
BLAKE2b-256 | 4b915101d736bfe56fda585f76498dad9ea0998459102c0aef11f2bb4ed5a4b7 |
Hashes for pyfastmask-1.1.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 580f7427a8355119326af06345a644dd8ab9bb50713b6df26e7a6958c0d98878 |
|
MD5 | d3837bb5a5f0a4b75c734f21f127e286 |
|
BLAKE2b-256 | a06b42094aae7daf2717402d38406a7c09fac6cc1942518b75acee090dbe057d |
Hashes for pyfastmask-1.1.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c3c4490c576c896b8c5643c515512efd3b5580a5f391e754a0c54185faada39e |
|
MD5 | 98b205b0eefa4d36507c2b9cf5b2681b |
|
BLAKE2b-256 | 990997baf00df174f28cb74254cdd9c6946756352575d1bf9acbf302ee410b8a |
Hashes for pyfastmask-1.1.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | da42981aebd8cb2ea1c84e7f00d64024a9dd9ed4db271c16e7f30253fa01af92 |
|
MD5 | 9f7a10244ab630da13b05dbf6e57995d |
|
BLAKE2b-256 | b5a7a9c636eef61cb80dc6d1e22460df058d84383195130d831dbfafac9e7c3e |
Hashes for pyfastmask-1.1.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f42c198ecdacab58bd6f5af0568ed0a03f605e5ba5bdcf8ddfdaa41b47cb2aa5 |
|
MD5 | 9a4f43c62e6e93065bc2e56cb0e8be73 |
|
BLAKE2b-256 | 2f7c9ed8cae2f6914a6dd0d6a562060594268a0860b674b228ccc29b21c876cd |
Hashes for pyfastmask-1.1.0-cp310-cp310-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b53107072da78c1953e4c3547f8b7572379aaa2f68bace1129e9596a63397c90 |
|
MD5 | 2f1d1386dcc70ab4107cca78ab331c28 |
|
BLAKE2b-256 | 4b1aa63dd8dda4f5390bb54a8016429f577491f3d70e25035593568a9d1dfb06 |
Hashes for pyfastmask-1.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e81f77b7fe99eee94c2a028a9fcea21867ea97287bd47882d1e03e998751044a |
|
MD5 | ff20084311f026e9ee4433c5621be560 |
|
BLAKE2b-256 | fe353460a47a971cb2875ce74b2fbd5741fb640132c0e603ea91f7d7be7911d2 |
Hashes for pyfastmask-1.1.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ddb51268206bc76955fc1c3136b00e31823b73c8d7c2750e4b490591c211154 |
|
MD5 | 8c31666e20de5bcfc3551bf0c1e425b9 |
|
BLAKE2b-256 | 61b584f2a41573847fc6bd3545ec34018321f4464f25de77f675a992b6c457f8 |
Hashes for pyfastmask-1.1.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8c1f1e928d304139976fca1524eddd6f746789a0e0ac010ae41e738159f2045 |
|
MD5 | e4f812467aa2d1d6f6f19a2b7fb4827a |
|
BLAKE2b-256 | d7250f0340e734ec9bc33bd1aaddb4d220359fd1308435b25e85b87019aec4e6 |
Hashes for pyfastmask-1.1.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 285d630f31cbe16ede8bb997a2aff8a6d14f4837f4a2d7e33ea6851a4948268a |
|
MD5 | 57d94a23a009dc2d2d5ed0b48a8e5337 |
|
BLAKE2b-256 | f6d541a3a3871191e3d98780231705e4111f20906189803c8b3dced088603897 |
Hashes for pyfastmask-1.1.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f51c8497545d1e25c3e9df141b4a15327af0988f6ee761ca59d907bb5d6c87e7 |
|
MD5 | 2549b078c04d7b413c3640eb239d3ca8 |
|
BLAKE2b-256 | 5c652975e54e0317dddca23dc6b6a9b7d75af165fa3ecc8853b14641623a7f3d |
Hashes for pyfastmask-1.1.0-cp39-cp39-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3922a814ba35696e8a8ee14ecfdcfa8cea0656502286a01b79e26ba46104b4b0 |
|
MD5 | f95222faa9a166685348d4fb150d4443 |
|
BLAKE2b-256 | c88b4206298c5944851a04a32274f4bc9f712a68429a729c9ab80f341899c193 |
Hashes for pyfastmask-1.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ca85261bf690b02f07a550d4e48ceaa04a14cd786bae422449452f30ea915ae |
|
MD5 | 8be173c900690d0900aab282316fffda |
|
BLAKE2b-256 | c06bc4636b8c93a774bec37704ef9fb65810c534c33d099f62b195cab4b1ef03 |
Hashes for pyfastmask-1.1.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa33d620690db5fdbb8b022e145bb43f37b5f85110a1f61b307f6080dcb61e18 |
|
MD5 | 30337162563b87d66e4c636f6cfe1064 |
|
BLAKE2b-256 | 4b775b21aeb79b79e80f1e2556283252e023d93c49691f5e64ffc702722185ee |
Hashes for pyfastmask-1.1.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d369f3ed70ccfb3962288cc942693f4e7e93aa16adf4de95295854cd8e41e643 |
|
MD5 | 07bf7a6ccc7c27771fd3815fa9538f1c |
|
BLAKE2b-256 | a61e4e3c5f7b1a3c899c26de65b491066c4c509f473ca5e9fff0dbbdc70d0969 |