hical 0.2.2

Python-first active learning engine backed by libhicalengine

hical

hical helps you review large document collections more efficiently. Instead of working through a corpus uniformly, you give it a few relevant examples and review feedback, and it keeps reranking the remaining documents so the next batch is more likely to matter.

Its primary use is helping reviewers find more relevant documents faster. It is also useful for building labeled datasets and evaluation or test collections without judging the entire corpus by hand.

In practice, hical gives you a Python workflow to build a corpus dataset from raw documents or ir_datasets, open it, and run interactive review sessions.

The source lives in the CALEngine repository. The intended user-facing surface is the hical Python package.

Installation

Install the package from PyPI:

python -m pip install hical

If you want ir_datasets support:

python -m pip install "hical[datasets]"

If you are building from a source checkout, see docs/developer/development.md.

Quickstart

Create a tiny JSONL corpus:

cat > docs.jsonl <<'EOF'
{"id": "doc-1", "title": "Florida citrus", "body": "Oranges and groves across central Florida."}
{"id": "doc-2", "title": "Coastal cleanup", "body": "Shoreline cleanup and beach restoration projects."}
EOF

Create a minimal config:

cat > corpus.yaml <<'EOF'
input:
  format: jsonl
  path: ./docs.jsonl
  doc_id_field: id
  text_fields:
    - title
    - body
output:
  path: ./docs.bin
  min_df: 1
  build_threads: 2
  parallel_docs_per_chunk: 50000
  optimize_for_fast_load: true
EOF

Build the corpus:

hical-build-corpus --config corpus.yaml

Open the dataset and start reviewing:

import hical

dataset = hical.open_dataset("docs.bin")
session = dataset.start_session(
    relevant_seeds=["florida oranges"],
    review_batch_size=2,
    retraining="auto",
    retrain_every_n_judgments=2,
    session_id="demo-review",
)

batch = session.next_batch()
for item in batch:
    print(item.doc_id, item.score)

session.judge_relevant(batch[0])

The normal flow is:

1. build a .bin
2. open it as a Dataset
3. start a Session
4. fetch documents and record judgments

For the purpose and workflow at a higher level, see docs/overview.md.

Common Tasks

Build your own corpus

Use hical-build-corpus with JSONL, CSV, TSV, archive, or ir_datasets input. For working configs and sample inputs, see:

• docs/build-your-own-corpus.md
• preprocessing/python/config_examples/
• examples/custom_data/

Use with ir_datasets

Build directly from a dataset id:

hical-build-ir-dataset --dataset-id cranfield --output ./cranfield.bin

Inspect fields first when the document type has multiple useful fields:

hical-build-ir-dataset --dataset-id beir/msmarco --list-fields

Then choose specific fields to combine:

hical-build-ir-dataset \
  --dataset-id beir/msmarco \
  --text-field title \
  --text-field text \
  --output ./msmarco.bin

For more, see docs/ir-datasets.md.

Use the Python API

The main public entry points are:

• hical.build_corpus
• hical.build_ir_dataset_corpus
• hical.inspect_ir_dataset
• hical.build_fast_load_index
• hical.open_dataset
• dataset.start_session
• session.next_batch

For the fuller dataset/session workflow, see docs/python-api.md.

Documentation

• docs/README.md: user guide index
• docs/overview.md: what hical is for and how the review loop works
• docs/concepts.md: core objects and terminology
• docs/build-your-own-corpus.md: corpus build workflows and config examples
• docs/ir-datasets.md: direct ir_datasets corpus building
• docs/python-api.md: dataset and session usage from Python
• docs/cli.md: bmi_cli usage
• docs/troubleshooting.md: setup and runtime issues
• examples/README.md: runnable examples

Supported Platforms

Published wheels are smoke-tested on:

• Linux x86_64
• macOS x86_64
• macOS arm64

Contributing

If you want to work on the repository internals rather than just use the Python package, start here:

• docs/developer/README.md
• docs/developer/development.md
• docs/developer/architecture.md