What does CTR stand for?
click-through rate
What is the A group in A/B test?
the control group
What is the B group in A/B test?
the treatment group
How would we compute conversion rates in an A/B test?
the number of times A or B was chosen divided by number of users shown either A or B
What would be the null hypothesis and alternate hypothesis for an A/B test with click through rates?
H_0: p_a = p_b, no difference
H_1: p_a != p_b, there is a difference
after identifying the null hypothesis for an a/b test, what should we do?
compute the pooled proportion, compute the standard error and then use that to compute z-score
what is the equivalent of 5% significance level in z values?
the critical z values for two tailed tests are += 1.96
what can we conclude if our z score was larger than 2?
we’d reject h_0 and conclude B performs better
When would we use a t-test?
use it if we’re comparing averages
what is an example of a metric that tells us we need to use a t-test?
average time spent, purchase amount
what is an example of a metric that tells us we need to use a two-proportion z-test?
click-through rate, conversion rate
what is an example of a metric that tells us we need to use a mann-whitney U test?
non-normal data or small samples
When would we use a mann-whitney u test?
non-parametric alternative
What is another time we would use a two-proportion z-test?
when the data are binary/categorical like clicked(1) or did not click(0)
What does a two-sample t-test measure?
whether the means differ significantly
What is an example that a two-sample t-test would be good for?
comparing the average time on a site for version A vs. version B
What is the chi-squared test of independence?
tests whether two categorical variables are related
what is an example that we would use chi-squared test of independence?
does gender affect click behavior?
what is the chi-squared goodness-of-fit test?
tests whether one categorical variable follows an expected distribution
what is an example of when we would use a chi-squared goodness-of-fit test?
are clicks evenly distributed across 3 button colors?
What is an easy way to tell if we need to use chi-squared test?
are these two things connected?
what is a true positive?
you said yes and you were right
what is a false positive?
you said yes but were wrong
what is a false negative?
you said no but were wrong
