This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Library for indexing LZO compressed files

Project Description
python-lzo-indexer
==================

![](https://travis-ci.org/duedil-ltd/python-lzo-indexer.png)

Python library for indexing block offsets within LZO compressed files. The implementation is largely based on that of the [Hadoop Library](https://github.com/twitter/hadoop-lzo). Index files are used to allow Hadoop to split a single file compressed with LZO into several chunks for parallel processing.

Since LZO is a block based compression algorithm, we can split the file along the lines of blocks and decompress each block on it's own. The index is a file containing byte offsets for each block in the original LZO file.


Example
-------

The python code below demonstrates how easy it is to index an LZO file. This library also supports indexing a string, and a method to return the individual block offsets should you need to create a file of your own format.

```python
import lzo_indexer

with open("my-file.lzo", "r") as f:
with open("my-file.lzo.index", "rw") as index:
lzo_indexer.index_lzo_file(f, index)
```


Command-line Utility
--------------------

This library also includes a utility for indexing multiple lzo files, using the python indexer. This is a much faster alternative to the command line utility built into the hadoop-lzo library as it avoids the JVM.

```
$ bin/lzo-indexer --help

usage: lzo-indexer [-h] [--verbose] [--force] lzo_files [lzo_files ...]

positional arguments:
lzo_files List of LZO files to index

optional arguments:
-h, --help show this help message and exit
--verbose, -v Enable verbose logging
--force, -f Force re-creation of an index even if it exists
```


Contributions
-------------

I welcome any contributions, though I request that any pull requests come with test coverage.
Release History

Release History

This version
History Node

0.0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
lzo-indexer-0.0.1.tar.gz (8.0 kB) Copy SHA256 Checksum SHA256 Source Aug 6, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting