ciff-toolkit 0.2.2

Toolkit for working with Common Index File Format (CIFF) files.

CIFF Toolkit

This repository contains a Python toolkit for working with Common Index File Format (CIFF) files.

Specifically, it provides a CiffReader and CiffWriter for easily reading and writing CIFF files. It also provides a handful of CLI tools, such as merging a CIFF file or dumping its contents.

Installation

To use the CIFF toolkit, install it from PyPI:

$ pip install ciff-toolkit

Usage

Reading

To read a CIFF file, you can use the CiffReader class. It returns the posting lists and documents as lazy generators, so operations that need to process large CIFF files do not need to load the entire index into memory.

The CiffReader can be used as a context manager, automatically opening files if a path is supplied as a str or pathlib.Path.

from ciff_toolkit.read import CiffReader

with CiffReader('./path/to/index.ciff') as reader:
    header = reader.read_header()

    for pl in reader.read_postings_lists():
        print(pl)

    for doc in reader.read_documents():
        print(doc)

Writing

The CiffWriter offers a similar context manager API:

from ciff_toolkit.ciff_pb2 import Header, PostingsList, DocRecord
from ciff_toolkit.write import CiffWriter

header: Header = ...
postings_lists: list[PostingsList] = ...
doc_records: list[DocRecord] = ...

with CiffWriter('./path/to/index.ciff') as writer:
    writer.write_header(header)
    writer.write_postings_lists(postings_lists)
    writer.write_documents(doc_records)

Command Line Interface

A couple of CLI commands are supported:

• ciff_dump INPUT

  Dumps the contents of a CIFF file, in order to inspect its contents.

• ciff_merge [-d,--description DESCRIPTION] INPUT... OUTPUT

  Merges two or more CIFF files into a single CIFF file. Ensures documents and terms are ordered correctly, and will read and write in a streaming manner (i.e. not read all data into memory at once). If a document identifier appears in multiple CIFF files, only the last one will be retained.

Development

This project uses Poetry to manage dependencies, configure the project and publish it to PyPI.

To get started, use Poetry to install all dependencies:

$ poetry install

Then, either activate the virtual environment to execute all Python code in the virtual environment, or prepend every command with poetry run.

$ poetry shell
(venv) $ ciff_dump index.ciff

or:

$ poetry run ciff_dump index.ciff