fewlab

Buy the labels that buy the smallest standard error.

The problem

You have a usage matrix: rows are units, usually people, and columns are the things they use. You want to regress something about the columns on features of the rows, but finding out what a column is costs money, and there are too many columns to label them all.

The outcome for each row is a usage-weighted average of the column labels: the share of a person's browsing that goes to news sites, the share of their spending at small businesses. fewlab says which columns to label, labels as few as the precision you need allows, and keeps the estimate exactly unbiased with a standard error you can report.

Worked example: a browsing panel

On a real browsing panel, labeling 250 of 34,078 domains recovers the same regression coefficients as labeling all of them, with a ninth of the error that spending the same 250 labels at random would give you. At 2,000 domains the error falls to 11% of the coefficient's own size, from scanning 6% of what was actually scanned.

Those numbers come from the benchmark, which runs against a population whose labels are all known, so every design is scored against the same fixed truth. They are regenerated by python benchmarks/make_exhibits.py, which also writes every figure quoted in the manuscript; none of them is typed in by hand.

To work through the same decision with a budget slider, without installing anything, run the notebook in your browser.

Why a label's value depends on who used it

Cost and value sit at different levels. You pay once to label a column, whether one person used it or a hundred thousand did, and that single label then enters every one of those people's rows. So what a label is worth depends both on where the entity appears and on what you intend to estimate.

That has an exact answer. Write V for row shares, so V_ij is person i's share of activity on entity j, and A for the matrix of entity labels. The outcome is Y = VA and the complete-label regression coefficient is

B = (X'X)⁻¹X'VA = HA,        H = (X'X)⁻¹X'V

so, writing h_j for column j of H,

B = Σⱼ hⱼ aⱼ'

exactly, with no error term and no model. Entity j's entire contribution to the answer is the rank-one matrix hⱼaⱼ', and hⱼ is computable from usage counts and features alone, before you buy a single label.

Which entities to label

Two goals, and neither is traded for the other. Unbiasedness is a constraint: every entity that contributes must have a positive chance of being labeled, and must be weighted by the reciprocal of that chance. Minimum standard error is what you optimize subject to it.

from fewlab import Study

# usage: one row per (person, item) pair with a count; features: one row per person
study = Study.from_long(usage, features, unit="person", entity="item", count="n")

design = study.design(budget=250)  # π_j = min(1, λ√(b_j/c_j))
sample = study.draw(design, random_state=0)
sample.save("label_sample.json")

labels = your_labeling_pipeline(sample.items)

from fewlab import LabelSample

sample = LabelSample.load("label_sample.json")
est = study.estimate(sample, labels)
print(est.summary())

Step by step, what that does:

1. Compute b_j = h_j'h_j, each entity's contribution to the risk. No labels are needed. For a mutually exclusive taxonomy, a_j'a_j = 1, so this is exact with no pilot.
2. Find the λ that exhausts the budget in π_j = min(1, λ√(b_j/c_j)).
3. Label with certainty every entity where the cap binds. You do not pick this head; the constraint produces it. In a heavy-tailed population it absorbs most of the budget, which is why the design pays.
4. Draw each remaining entity independently at π_j. The budget is the expected number of labels, so the realized count varies. If an exact count is required, study.draw(design, fixed_size=True) uses the cube method to balance on the influence totals.
5. Weight by 1/π_j. Head entities get weight 1; tail entities get more, standing in for the tail you did not label.

Unbiasedness comes from step 4, positive probability everywhere, plus the weighting in step 5. Minimum standard error comes from the square root rule in step 2. You need both.

The rule also solves the reverse question. row_se_min_labels gives the fewest labels that hold every row's standard error under a ceiling you name, rather than the smallest error for a budget you name.

To set a regression-risk ceiling instead, use study.design_for_risk(risk_limit=0.01**2, contrast=contrast) and study.draw(design, fixed_size=False). This buys the least expected labeling cost for the chosen contrast's label-sampling variance. Omit contrast to bound the sum of coefficient variances. Unit label norms give exact aggregate risk for a complete one-hot taxonomy and a conservative bound for a binary indicator. The ceiling excludes regression uncertainty and realized error; see the quickstart for a complete example.

Inference for the observed frame

est.design_se measures uncertainty from labeling only some entities. It conditions on the observed rows, their features, and the entity frame. The default est.summary() uses this standard error. Independent draws use the Poisson variance estimator; cube draws use a balanced residual approximation. The estimator is selected from the recorded draw method. The Poisson risk shown by a design is exact for independent draws and only a planning criterion for cube draws.

Save the draw before sending its selected IDs for labeling. The JSON manifest stores every entity ID and inclusion probability, which IDs were selected, the draw method, the random seed when one was supplied, and a fingerprint of the counts and features. Keep the input counts and features too: they define the finite frame and are needed to resume estimation. Study.estimate rejects a sample loaded against different study data. Labels should be indexed by the selected entity IDs; their order does not matter. The manifest is an audit record, not a substitute for the labels or the study data.

summary() names the variance method and marks the variance estimator as design-unbiased variance, approximate, or unavailable. A Poisson draw with the matching Poisson variance estimator gets the first status. Cube draws use a balanced approximation.

Entity labels are the sampling units here: a single label can affect every row's outcome. Applying ordinary survey weights to the regression rows would describe a different sampling design.

Row-level model uncertainty and when to stop paying

If the observed rows are themselves viewed as a sample from a larger population, Pass row_model_se=True to study.estimate when this model is appropriate. regression_se adds a homoskedastic row-level model assumption. total_se combines that component with label-sampling uncertainty. Neither is part of the finite-frame design SE.

Var(β̂) = σ²(X'X)⁻¹  +  Σⱼ (1−πⱼ)/πⱼ · bⱼ
         your n units      your labeling budget

Only the second term moves when you spend more. Under the row-level model, Estimate.diagnostics["variance_ratio"] reports the share of combined variance owed to labeling; it can guide a budget decision, but it is not a design-based uncertainty measure for rows that were not sampled by a known design.

Using a cheap classifier without giving up unbiasedness

If you can guess a label cheaply for every entity, as a domain name predicts a site category or a merchant name its line of business, pass it as proxy and fewlab uses

B̂ = Σⱼ hⱼ μⱼ' + Σⱼ (Rⱼ/πⱼ) hⱼ (aⱼ − μⱼ)'

This is exactly unbiased whatever the classifier says, because the proxy term is a known constant and the sampled term corrects it. A bad proxy costs variance and never bias, so there is no risk in supplying one. Acquisition then targets influence times residual label uncertainty, and for calibrated class probabilities that uncertainty is 1 − ‖μⱼ‖², known without labels.

Buying bias on purpose

Shrinking the weights trades bias for variance, and at small budgets it wins: on the benchmark, shrinking cuts mean squared error by 46% at 250 labels. Pass alpha to estimate and it reports the worst case bias alongside the variance, so the trade is priced rather than assumed. The default is alpha=1, which is unbiased.

Setting alpha=0 is what zero-filling unlabeled entities amounts to. On the benchmark its bias runs from 14% to 53% of the coefficient's own size, and it is never the best point on the frontier.

Checking that a design can help you at all

Some populations have nothing to optimize. If every entity influences the regression in the same direction and differs only in magnitude, every loss function ranks entities identically and the choice of estimand stops mattering. Study.diagnostics["direction_spectrum"] reports this. Read it against a noise floor of roughly 1/√(typical count) rather than a fixed cutoff.

Installation

pip install fewlab

Requires Python 3.12 or later, numpy and pandas. Counts can be handed over in long format, one row per unit and entity, which is how usage logs arrive and what keeps a 0.3% dense matrix from being materialized.

pip install -e ".[dev]"   # tests, linting, pre-commit
pip install -e ".[docs]"  # documentation

Upgrading from 2.x

3.0 changes the API throughout and has no compatibility shims. The changelog lists what was removed, what replaces it, and which 2.x results should be re-drawn.

License

MIT