twiser 0.0.1

Setup a predictor as a control variate

First, we need to define a regression model. We can use anything that fits the sklearn idiom of fit and predict methods. This predictor is used to take the n x d array of treatment unit covariates x_covariates and predict the treatment outcomes n-length outcome array x. Likewise, it makes predictions from the m x d array of control unit covariates y_covariates to the control m-length outcome array y.

predictor = RandomForestRegressor(criterion="squared_error", random_state=0)

Basic z-test

First, we apply the basic two-sample z-test included in Twiser. This works basically the same as scipy.stats.ttest_ind.

estimate, (lb, ub), pval = twiser.ztest(x, y, alpha=0.05)
show_output(estimate, (lb, ub), pval)

ATE estimate: 0.80 in (-0.14, 1.75), CI width of 1.89, p = 0.0954

Variance reduction with held out data

Next, we apply variance reduction where the predictor was trained on a held out 30% of the data. This is the easiest to show validity, but some of the added power is lost because not all data is used in the test.

estimate, (lb, ub), pval = twiser.ztest_held_out_train(
  x,
  x_covariates,
  y,
  y_covariates,
  alpha=0.05,
  train_frac=0.3,
  predictor=predictor,
  random=np.random.RandomState(123),
)
show_output(estimate, (lb, ub), pval)

ATE estimate: 1.40 in (0.20, 2.59), CI width of 2.39, p = 0.0217*

Variance reduction with cross validation

To be more statistically efficient we train and predict using 10-fold cross validation. Here, no data is wasted. As we can see it is a more significant result.

estimate, (lb, ub), pval = twiser.ztest_cross_val_train(
  x,
  x_covariates,
  y,
  y_covariates,
  alpha=0.05,
  k_fold=10,
  predictor=predictor,
  random=np.random.RandomState(123),
)
show_output(estimate, (lb, ub), pval)

ATE estimate: 1.38 in (0.51, 2.25), CI width of 1.74, p = 0.0019*

Variance reduction in-sample

In the literature it is popular to train the predictor in the same sample as the test. This often gives the most power. However, any overfitting in the predictor can also invalidate the results.

estimate, (lb, ub), pval = twiser.ztest_in_sample_train(
  x,
  x_covariates,
  y,
  y_covariates,
  alpha=0.05,
  predictor=predictor,
  random=np.random.RandomState(123),
)
show_output(estimate, (lb, ub), pval)

ATE estimate: 0.86 in (0.24, 1.49), CI width of 1.24, p = 0.0065*

Other interfaces

It is also possible to call these methods using raw control predictions instead of training the predictor in the Twiser method. It also supports a sufficient statistics interface for working with large datasets. See the documentation for details.