fewlab

Buy the labels that buy the smallest standard error.

On a real browsing panel, labeling 250 of 34,078 domains recovers the same regression coefficients as labeling all of them, with a ninth of the error that spending the same 250 labels at random would give you. The estimate stays exactly unbiased and reports a standard error you can put in a table. At 2,000 domains the error falls to 11% of the coefficient's own size, from scanning 6% of what was actually scanned.

Those numbers come from the benchmark, which runs against a population whose labels are all known, so every design is scored against the same fixed truth. They are regenerated by python benchmarks/make_exhibits.py, which also writes every figure quoted in the manuscript; none of them is typed in by hand.

The problem

You have a usage matrix: rows are people, columns are things they used. Websites, merchants, products, employers, surnames. You want to regress something about the column on features of the row, but finding out what the column is costs money, and you cannot afford all of them.

The awkward part is that cost and value sit at different levels. You pay once to label a website, whether one panelist visited it or a hundred thousand did, and that single label then enters every one of those people's rows. So what a label is worth depends both on where the entity appears and on what you intend to estimate.

That has an exact answer. Write V for row shares, so V_ij is person i's share of activity on entity j, and A for the matrix of entity labels. The outcome is Y = VA and the complete-label regression coefficient is

B = (X'X)⁻¹X'VA = HA,        H = (X'X)⁻¹X'V

so, writing h_j for column j of H,

B = Σⱼ hⱼ aⱼ'

exactly, with no error term and no model. Entity j's entire contribution to the answer is the rank-one matrix hⱼaⱼ', and hⱼ is computable from usage counts and features alone, before you buy a single label.

Which entities to label

Two goals, and neither is traded for the other. Unbiasedness is a constraint: every entity that contributes must have a positive chance of being labeled, and must be weighted by the reciprocal of that chance. Minimum standard error is what you optimize subject to it.

from fewlab import Study

study = Study.from_long(visits, features, unit="user", entity="domain", count="visits")

design = study.design(budget=250)  # π_j = min(1, λ√(b_j/c_j))
sample = study.draw(design, random_state=0)

labels = your_labeling_pipeline(sample.items)

est = study.estimate(sample, labels)
print(est.summary())

Step by step, what that does:

1. Compute b_j = h_j'h_j, each entity's contribution to the risk. No labels are needed. For a mutually exclusive taxonomy, a_j'a_j = 1, so this is exact with no pilot.
2. Find the λ that exhausts the budget in π_j = min(1, λ√(b_j/c_j)).
3. Label with certainty every entity where the cap binds. You do not pick this head; the constraint produces it. In a heavy-tailed population it absorbs most of the budget, which is why the design pays.
4. Sample the rest at π_j, not uniformly. The cube method lands exactly on budget and balances on the influence totals.
5. Weight by 1/π_j. Head entities get weight 1; tail entities get more, standing in for the tail you did not label.

Unbiasedness comes from step 4, positive probability everywhere, plus the weighting in step 5. Minimum standard error comes from the square root rule in step 2. You need both.

The rule also solves the reverse question. row_se_min_labels gives the fewest labels that hold every row's standard error under a ceiling you name, rather than the smallest error for a budget you name.

To set a regression-risk ceiling instead, use study.design_for_risk(risk_limit=0.01**2, contrast=contrast) and study.draw(design, fixed_size=False). This buys the least expected labeling cost for the chosen contrast's label-sampling variance. Omit contrast to bound the sum of coefficient variances. Unit label norms give exact aggregate risk for a complete one-hot taxonomy and a conservative bound for a binary indicator. The ceiling excludes regression uncertainty and realized error; see the quickstart for a complete example.

Two standard errors, and when to stop paying

Var(β̂) = σ²(X'X)⁻¹  +  Σⱼ (1−πⱼ)/πⱼ · bⱼ
         your n units      your labeling budget

Only the second term moves when you spend more. Estimate.diagnostics reports the share of total variance the budget is responsible for, which is a stopping rule: once the design term falls well below the regression term, more labels buy almost nothing.

Using a cheap classifier without giving up unbiasedness

If you can guess a label cheaply for every entity, a domain name predicts a site category fairly well, pass it as proxy and fewlab uses

B̂ = Σⱼ hⱼ μⱼ' + Σⱼ (Rⱼ/πⱼ) hⱼ (aⱼ − μⱼ)'

This is exactly unbiased whatever the classifier says, because the proxy term is a known constant and the sampled term corrects it. A bad proxy costs variance and never bias, so there is no risk in supplying one. Acquisition then targets influence times residual label uncertainty, and for calibrated class probabilities that uncertainty is 1 − ‖μⱼ‖², known without labels.

Buying bias on purpose

Shrinking the weights trades bias for variance, and at small budgets it wins: on the benchmark, shrinking cuts mean squared error by 46% at 250 labels. Pass alpha to estimate and it reports the worst case bias alongside the variance, so the trade is priced rather than assumed. The default is alpha=1, which is unbiased.

Setting alpha=0 is what zero-filling unlabeled entities amounts to. On the benchmark its bias runs from 14% to 53% of the coefficient's own size, and it is never the best point on the frontier.

Checking that a design can help you at all

Some populations have nothing to optimize. If every entity influences the regression in the same direction and differs only in magnitude, every loss function ranks entities identically and the choice of estimand stops mattering. Study.diagnostics["direction_spectrum"] reports this. Read it against a noise floor of roughly 1/√(typical count) rather than a fixed cutoff.

Installation

pip install fewlab

Requires Python 3.12 or later, numpy and pandas. Counts can be handed over in long format, one row per unit and entity, which is how panels arrive and what keeps a 0.3% dense matrix from being materialized.

pip install -e ".[dev]"   # tests, linting, pre-commit
pip install -e ".[docs]"  # documentation

Upgrading from 2.x

3.0 changes the API throughout and has no compatibility shims. The changelog lists what was removed, what replaces it, and which 2.x results should be re-drawn.

License

MIT