No project description provided
Project description
CMIP - Conditional Mutual Information with the logging Policy
CMIP implementation from the 2023 SIGIR paper: An Offline Metric for the Debiasedness of Click Models
.
The metric quantifies the mutual information between a new click model policy and the production system that collected the train dataset (logging policy), conditional on human relevance judgments. CMIP quantifies the degree of debiasedness (see paper for details). A policy is said to be debiased w.r.t. its logging policy with a cmip <= 0
.
Example
import numpy as np
n_queries = 1_000
n_results = 25
# Human relevance annotations per query-document pair
y_true = np.random.randint(5, size=(n_queries, n_results))
# Relevance scores of the logging policy
y_logging_policy = y_true + np.random.randn(n_queries, n_results)
# Relevance scores of a new policy (in this case, strongly dependent on logging policy)
y_predict = y_logging_policy + np.random.randn(n_queries, n_results)
# Number of documents per query, used for masking
n = np.full(n_queries, n_results)
from cmip_metric import CMIP
metric = CMIP()
metric(y_predict, y_logging_policy, y_true, n)
> 0.2687 # The policy predicting y_predict is not debiased w.r.t. the logging policy.
Installation
pip install cmip-metric
Reference
Note: To be published at:
@inproceedings{Deffayet2023Debiasedness,
author = {Romain Deffayet and Philipp Hager and Jean-Michel Renders and Maarten de Rijke},
title = {An Offline Metric for the Debiasedness of Click Models},
booktitle = {Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR`23)},
organization = {ACM},
year = {2023},
}
License
This project uses the MIT license.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
cmip_metric-0.1.2.tar.gz
(5.0 kB
view hashes)
Built Distribution
Close
Hashes for cmip_metric-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 081a4a45a230275d7579250292e19ff3ced629a180d2c8fa5f0f1caee7ca1599 |
|
MD5 | c9fd96d420404dc7a3a7196d32e6239a |
|
BLAKE2b-256 | 61546649787acbbf7186a00422bfdd2b9121806977f7a9829e3ac242739e5996 |