This module provides an IndexedBzip2File class, which can be used to seek inside bzip2 files without having to decompress them first.
It's based on an improved version of the bzip2 decoder bzcat from toybox, which was refactored and extended to be able to export and import bzip2 block offsets and seek to them.
Seeking inside a block is only emulated, so IndexedBzip2File will only speed up seeking when there are more than one blocks, which should almost always be the cause for archives larger than 1 MB.
You can simply install it from PyPI:
pip install indexed_bzip2
fromindexed_bzip2importIndexedBzip2Filefile=IndexedBzip2File("example.bz2")# You can now use it like a normal filefile.seek(123)data=file.read(100)
The first call to seek will ensure that the block offset list is complete and therefore might create them first.
Because of this the first call to seek might take a while.
The creation of the list of bzip2 blocks can take a while because it has to decode the bzip2 file completely.
To avoid this setup when opening a bzip2 file, the block offset list can be exported and imported.
fromindexed_bzip2importIndexedBzip2Filefile=IndexedBzip2File("example.bz2")blockOffsets=file.block_offsets()# can take a whilefile.close()file2=IndexedBzip2File("example.bz2")blockOffsets=file2.set_block_offsets(blockOffsets)# should be fastfile2.seek(123)data=file2.read(100)file2.close()
Tracing the decoder
Performance profiling and tracing is done with Score-P for instrumentation and Vampir for visualization.
This is one way, you could install Score-P with most of the functionalities on Debian 10.
# Install PAPI
tar -xf papi-5.7.0.tar.gz
sudo make install
# Install Dependencies
sudo apt-get install libopenmpi-dev openmpi gcc-8-plugin-dev llvm-dev libclang-dev libunwind-dev libopen-trace-format-dev otf-trace
# Install Score-P (to /opt/scorep)
tar -xf scorep-6.0.tar.gz
./configure --with-mpi=openmpi --enable-shared
# Add /opt/scorep to your path variables on shell start
cat <<EOF >> ~/.bashrcif test -d /opt/scorep; then export SCOREP_ROOT=/opt/scorep export PATH=$SCOREP_ROOT/bin:$PATH export LD_LIBRARY_PATH=$SCOREP_ROOT/lib:$LD_LIBRARY_PATHfiEOF# Check whether it works
# Actually do the tracingcd tools