lzo-indexer 0.0.1

Library for indexing LZO compressed files

python-lzo-indexer
==================


Python library for indexing block offsets within LZO compressed files. The implementation is largely based on that of the [Hadoop Library](https://github.com/twitter/hadoop-lzo). Index files are used to allow Hadoop to split a single file compressed with LZO into several chunks for parallel processing.

Since LZO is a block based compression algorithm, we can split the file along the lines of blocks and decompress each block on it's own. The index is a file containing byte offsets for each block in the original LZO file.


Example
-------

The python code below demonstrates how easy it is to index an LZO file. This library also supports indexing a string, and a method to return the individual block offsets should you need to create a file of your own format.

```python
import lzo_indexer

with open("my-file.lzo", "r") as f:
with open("my-file.lzo.index", "rw") as index:
lzo_indexer.index_lzo_file(f, index)
```


Command-line Utility
--------------------

This library also includes a utility for indexing multiple lzo files, using the python indexer. This is a much faster alternative to the command line utility built into the hadoop-lzo library as it avoids the JVM.

```
$ bin/lzo-indexer --help

usage: lzo-indexer [-h] [--verbose] [--force] lzo_files [lzo_files ...]

positional arguments:
lzo_files List of LZO files to index

optional arguments:
-h, --help show this help message and exit
--verbose, -v Enable verbose logging
--force, -f Force re-creation of an index even if it exists
```


Contributions
-------------

I welcome any contributions, though I request that any pull requests come with test coverage.