Family-wise Error
Each time you test at = 0.05, you increase your chance of error
Types of multiple comparisons - Planned comparisons (contrasts)
- Before study
- A priori
- Break down variance into component parts
- Test specific hypotheses
Types of multiple comparisons - Post-hoc analyses
- After study
- Compare all groups using stricter alpha values
- This reduces Type I error rate
Planned Contrasts
Involves breaking down the variance according to hypotheses made ‘a priori’ (i.e., before the data were collected)
RULES
* Once a group has been singled out – it cannot be used in another contrast
* Each contrast must only compare 2 “chunks” of variation
* There should always be 1 less comparison than the number of groups (i.e., number of contrasts = k-1)
Planned Contrasts - Rules
- Compare positive against negative weights
- The sum of weights for a comparison should be zero
- If a group is not in a comparison it should be assigned zero
- For any contrast, the weights assigned to the groups or group in one chunk of variation should be equal to the number of groups in the opposite chunk of variation
Planned Contrasts - Orthogonal Contrasts
Compare unique “chunks” of variance
Planned Contrasts - Non-orthogonal Contrasts
Overlap or use the same “chunks” of variance in multiple
comparisons
Require careful interpretation
Lead to increased type 1 error rate
Planned Contrasts - Standard Contrasts
Orthogonal: Helmert and Difference
Non-Orthogonal: Deviation, Simple, Repeated
Planned Contrasts - Polynomial Contrasts
Linear, Quadratic, Cubic and Quartic trends
Planned Contrasts - Helmert
Compare each category to the mean of subsequent categories (based on the order they are coded in SPSS, which might be alphabetical!)
Planned Contrasts - Difference
Compare each category to the mean of previous categories
Planned Contrasts - Polynomial contrasts
Used only when your IV is ordinal
Research Scenario
Researcher interested in exploring the influences of drawing conditions on drawing quality
* Total sample of 60 with 20 participants in each group
* Independent Variable: Drawing condition
- 3 levels: Normal, Non-dominant Hand, Blindfolded
* Dependent Variable: Drawing quality
- Rated by an independent group of observers with the average score being the drawing quality
- The possible range of scores is 0-10 with higher scores indicating higher drawing quality
Research Questions
Does the conditions in which you draw an object affect the quality of your drawings?
Hypotheses:
* H1: Drawing with your dominant hand will produce higher quality images than other conditions
* H2: Drawing while blindfolded will be more difficult than drawing with your non-dominant hand
Post-hoc Tests
Involves comparing all possible differences between pairs of means
* Good approach with exploratory research or where there are no pre-defined specific hypotheses
* Simplest post-hoc test is Bonferroni
* Bonferroni correction means:
𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 𝛼= 𝛼 /𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠
Post-hoc Tests - Tukey’s HSD (Honestly Significant Difference)
- The cumulative probability of a type 1 error never exceeds the specified level of significance (p < .05)
- Supplies a single critical value (HSD) for evaluating the ‘significance’ of each pair of means
- The critical value (HSD) increases with (i.e., each additional group mean)
- It becomes more difficult to reject the null hypothesis as a greater number of group means are compared
- If the absolute (i.e., obtained) difference between two means exceeds the critical value for HSD, the null hypothesis for that pair of means can be rejected
Module Summary
ANOVAs tell us if there is a main effect of an independent variable on a dependent variable, but not where that effect is
Running multiple t-tests isn’t sensible, because of family- wise error
Multiple comparisons allow us to compare levels of the independent variable
Planned contrasts are designed a priori to test specific hypotheses
Post-hoc tests compare all conditions to each other
You should know how to run these tests, and how to report them
