Skip to main content

Fast random access to bzip2 files

Project description

indexed_bzip2

PyPI version Downloads License Build Status

This module provides an IndexedBzip2File class, which can be used to seek inside bzip2 files without having to decompress them first. It's based on an improved version of the bzip2 decoder bzcat from toybox, which was refactored and extended to be able to export and import bzip2 block offsets and seek to them. Seeking inside a block is only emulated, so IndexedBzip2File will only speed up seeking when there are more than one blocks, which should almost always be the cause for archives larger than 1 MB.

Installation

You can simply install it from PyPI:

pip install indexed_bzip2

Usage

Example 1

from indexed_bzip2 import IndexedBzip2File

file = IndexedBzip2File( "example.bz2" )

# You can now use it like a normal file
file.seek( 123 )
data = file.read( 100 )

The first call to seek will ensure that the block offset list is complete and therefore might create them first. Because of this the first call to seek might take a while.

Example 2

The creation of the list of bzip2 blocks can take a while because it has to decode the bzip2 file completely. To avoid this setup when opening a bzip2 file, the block offset list can be exported and imported.

from indexed_bzip2 import IndexedBzip2File

file = IndexedBzip2File( "example.bz2" )
blockOffsets = file.block_offsets() # can take a while
file.close()

file2 = IndexedBzip2File( "example.bz2" )
blockOffsets = file2.set_block_offsets( blockOffsets ) # should be fast
file2.seek( 123 )
data = file2.read( 100 )
file2.close()

Tracing the decoder

Performance profiling and tracing is done with Score-P for instrumentation and Vampir for visualization. This is one way, you could install Score-P with most of the functionalities on Debian 10.

# Install PAPI
wget http://icl.utk.edu/projects/papi/downloads/papi-5.7.0.tar.gz
tar -xf papi-5.7.0.tar.gz
cd papi-5.7.0/src
./configure
make -j
sudo make install

# Install Dependencies
sudo apt-get install libopenmpi-dev openmpi gcc-8-plugin-dev llvm-dev libclang-dev libunwind-dev libopen-trace-format-dev otf-trace

# Install Score-P (to /opt/scorep)
wget https://www.vi-hps.org/cms/upload/packages/scorep/scorep-6.0.tar.gz
tar -xf scorep-6.0.tar.gz
cd scorep-6.0
./configure --with-mpi=openmpi --enable-shared
make -j
make install

# Add /opt/scorep to your path variables on shell start
cat <<EOF >> ~/.bashrc
if test -d /opt/scorep; then
    export SCOREP_ROOT=/opt/scorep
    export PATH=$SCOREP_ROOT/bin:$PATH
    export LD_LIBRARY_PATH=$SCOREP_ROOT/lib:$LD_LIBRARY_PATH
fi
EOF

# Check whether it works
scorep --version
scorep-info config-summary

# Actually do the tracing
cd tools
bash trace.sh

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for indexed-bzip2, version 1.1.2
Filename, size File type Python version Upload date Hashes
Filename, size indexed_bzip2-1.1.2-cp35-cp35m-manylinux1_i686.whl (296.8 kB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp35-cp35m-manylinux1_x86_64.whl (309.8 kB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp35-cp35m-manylinux2010_x86_64.whl (390.3 kB) File type Wheel Python version cp35 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp36-cp36m-manylinux1_i686.whl (296.4 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp36-cp36m-manylinux1_x86_64.whl (310.0 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp36-cp36m-manylinux2010_x86_64.whl (392.8 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp37-cp37m-manylinux1_i686.whl (296.5 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp37-cp37m-manylinux1_x86_64.whl (312.1 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp37-cp37m-manylinux2010_x86_64.whl (392.6 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp38-cp38-manylinux1_i686.whl (291.1 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp38-cp38-manylinux1_x86_64.whl (303.8 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp38-cp38-manylinux2010_x86_64.whl (411.2 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp39-cp39-manylinux1_i686.whl (292.7 kB) File type Wheel Python version cp39 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp39-cp39-manylinux1_x86_64.whl (306.6 kB) File type Wheel Python version cp39 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2-cp39-cp39-manylinux2010_x86_64.whl (404.2 kB) File type Wheel Python version cp39 Upload date Hashes View
Filename, size indexed_bzip2-1.1.2.tar.gz (75.6 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page