1
Q
Pearson correlation
A
- linear and monotonic data only, ratio or interval data
- describes direction (positive/negative), form (linear), consistency (strength, degree of correlation) of a relationship
- correlation values are ordinal: r of .8 is not twice as strong as .4 (does not increase in equal increments)
2
Q
Spearman correlation
A
- Spearman rho (rs) used for ordinal data
- for monotonic non-linear data
- compute the rank order of scores (order from smallest to largest, assign a rank, plot the ranks which corrects for nonlinearities)
- you need at least 5 pairs of data (ideally more than 8 pairs)
- Spearman correlation will be 1 when data are monotonically related even if not linear (will pick up on this relationship better than Pearson)
- less sensitive to outliers than Pearson (ranks cannot be outliers because they will fall into the same range as the rest of the data)
3
Q
monotonic
A
two variables increase or decrease or stay the same together, but the changes are not consistently the same size
4
Q
point-biserial correlation
A
- when one variable is non-numerical and has two levels, you can convert those levels into 0 and 1
- the sign is meaningless, the idea of a linear correlation is meaningless
5
Q
chi-square test
A
- both variables are non-numerical, so you organize the data into a matrix according to the frequency of individuals in each cell
6
Q
phi-coefficient
A
- both variables are non-numerical and each have two levels - code them both as 0 and 1
- sign and linearity are meaningless
7
Q
coefficient of determination
A
- r2: how much variability in one variable is explained by its relationship with the other variable (shared variance)
- like a Venn diagram showing the degree of overlap (the more overlap, the stronger the relationship)
- if only given the r2 value, you can tell the strength of the correlation (square root), but not the direction (a square root can be negative or positive)
8
Q
what can you use the correlational strategy for
A
- predictions about future behaviour or the other variable (using regression, a predictor variable and a criterion variable)
- determine test-retest reliability and concurrent validity
- often used for preliminary research to indicate further research is required
9
Q
problems with correlational strategy
A
- third-variable problem: an unidentified variable is controlling the levels of both variable
- directionality problem: cannot determine which variable is the cause and which is the effect
10
Q
small, medium, large correlations
A
- no relationship: 0 - 0.1
- small/weak: r = 0.1 - 0.3, r2 = 0.01
- medium/moderate: r = 0.3 (or 0.3 - 0.7), r2 = 0.09
- large: r = 0.5 (or 0.7 - 1), r2 = 0.25 (or 0.49)
11
Q
multiple regression
A
- for multivariate relationships: an individual variable relates to a multitude of other variables
- one criterion variable can be better predicted by a set of variables than just one at a time (but you can still examine the individual relationships)
12
Q
correlational strategy
A
- describes the direction and degree of the relationship between two or more variables
- data collected is only measured, not manipulated (observations, surveys, physiological)
- high external validity
- unit of analysis is often time or a person - measuring both X and Y
- if data is numerical, represented in a scatterplot (each point is independent; one point per unit of analysis) with a regression line (line of best fit)
13
Q
spearman rho interpretation
A
- weak: 0.21 - 0.41
- moderate: 0.41 - 0.60
- strong: 0.61 - 0.80
- very strong: 0.81 - 1.00
14
Q
what relies on monotonicity?
A
- both Pearson and Spearman
15
Q
what relies on linearity
A
- Pearson
16
Q
which is robust to outliers
A
- Spearman
17
Q
outliers
A
- data point that differs significantly from others in the set
- defined by the variance in the variable (2-3 standard deviations from the mean)
18
Q
correlation significance
A
- typically p < 0.05 (less than 5% chance that this result is due to chance alone)
- df = n - 2 (number of variables)
- consult a table: to be significant r must be equal or larger than the corresponding value of df and alpha level
- small sample sizes are prone to producing larger correlations, so the criteria for statistical significance are more stringent (when n = 2, r is always +/- 1)
- if the sample size is larger enough, a small r could still be meaningful (in this case, evaluate for practical significance)
19
Q
practical significance
A
- related to meaningful, real-world consequences of the observed correlation
20
Q
advantages of correlations
A
- quick and efficient
- often the only method available (for practical or ethical reasons)
- high external validity (reflects natural events being examined)
21
Q
limitations of correlations
A
- cannot say why two variables are related (correlations are often misinterpreted by the media which assumes causality)
- low internal validity
- very sensitive to outliers
- directionality and third variable problem
22
Q
experimental research
A
- establishing a cause-and-effect relationship between two variables
- manipulation: changing level of the IV to create two or more treatment conditions (allowing us to determine directionality and causes through temporal ordering)
- control: all extraneous variables must be controlled to ensure they can’t be responsible for the change in DV (ensuring they don’t become confounds and produce a third variable problem - internal validity)
- extraneous variables only become confounds if they have an effect on the DV and vary systematically with the IV (only focus on controlling the important variables)
- creating an artificial/unnatural situation
23
Q
hold constant method
A
- active method of control
- standardizing environment and procedures means that those variables are held constant for all participants (if they don’t vary, they can’t become confounds)
- often limiting to a range (like ages) for practical reasons
- limits external validity because you won’t be able to generalize beyond this range of values
24
Q
matching method
A
- active method of control
- match subjects on pre-existing variables that may be related to differences in DV
- across levels of IV (make sure ages are balanced across treatment conditions or make sure the average ages are the same)
- can also be used to control environmental or time-related factors (counterbalancing)
- can be time consuming and impossible for all extraneous variables
25
randomization method
- passive method of control
- disrupting any systematic relationship between extraneous and independent variables by distributing extraneous variables across treatment conditions using a random process (random assignment)
- can also be used for environmental variables
- makes it unlikely to see systematic relationships between variables, but not impossible (small samples are more likely to be biased, should create groups of at least 20 participants per condition)
- variables that are especially important should be held constant or matched
26
control conditions
- no-treatment (waitlist control) or placebo control (for showing effects beyond the placebo effect)
- not a necessary component to be considered experimental research (but control of extraneous variables is an essential component)
27
manipulation check
- measures whether the IV had the intended effect on the participant (how they interpreted/perceived the intervention)
- could be an explicit measure of the IV, or part of the exit questionnaire
- especially important in participant manipulations (was the intervention successful), subtle interventions (if it could go unnoticed by the participant), placebo controls (Ps must believe the placebo is real), simulation of a real-world situation (depends on their perception and acceptance)
28
conditions for establishing causality
1) time order: IV occurred before the effects in DV
2) co-variation: changes in the IV value must be accompanied by changes in the DV value
3) rationale/explanation: logical and compelling reason for the two variables being related (the mechanism behind the association)
4) non-spuriousness: only the IV caused the changes found in DV; rival explanations must be ruled out
29
between-groups design
- two or more samples/groups are formed randomly (each group composed of different participants) and assigned randomly to a different condition (level of IV)
- participation in only one level = one score on DV per participant (independent measures)
- lots of individual differences = more variability in the scores and potential confound (significant threat to internal validity)
- uses systematic and non-systematic variance to calculate the F-ratio (treatment index)
- use randomization, matching, holding constant to make sure groups are as similar as possible (distribute characteristics equally between groups, holding constant restricts external validity)
30
within-groups design
- only one sample/group is formed and each person participates in all conditions, which can be administered sequentially or all at once (values are compared across conditions within participants)
- possibility of time-related and environmental threats to internal validity
31
four basic elements of the experimental design
1) manipulation: creating conditions (levels of IV) to determine the direction of the effect and helps control the influence of other variables (ensuring that the IV isn't changing with other another variable)
2) measurement (not unique to experiments)
3) comparison (not unique to experiments)
4) control: ruling out alternative explanations for changes in DV by not letting extraneous variables become confounds (if the variable affects all conditions equally, it's just an extraneous variable - if only one condition is affected and it's possible the variable is affecting the DV, it's a confound)
32
categories of extraneous variables
- environmental: testing environment, time of day, etc.
- participant variables: gender, age, personality, IQ, etc.
- time-related variables: history, maturation, instrumentation, testing effects, regression to the mean
33
ways to control possible confounds
- remove them (not possible for all variables)
- hold them constant (include them in all conditions, can limit external validity)
- use a placebo control (or waitlist): if the experimental method itself is a confound (like delivering medication by injection - inject saline to the control group)
- match them across conditions (can limit external validity): creating balanced groups, averages, counterbalancing
- randomize them: powerful for controlling participant and environmental variables all at once, rather than individually
34
possible reasons an experiment didn't work
- IV isn't sensitive enough (not a wide enough variety of conditions)
- DV isn't sensitive enough (scale not sensitive)
- IV/DV have floor or ceiling effects
- measurement error: methods you used are prone to error (you can control this)
- insufficient power: not enough participants
- hypothesis is wrong: compare with other studies
35
threats to internal validity in experiments
- history: a current event affected change in the DV
- maturation: changes in DV due to normal development processes
- statistical regression: subjects came from low- or high-performing groups, and subsequent testing generated scores closer to the mean
- selection: self-selected or randomly assigned?
- experimental attrition: more people dropped out of one condition than the other
- testing: previous testing affected behaviour at later testing (should counterbalance)
- instrumentation: measurement method changed during research
- design contamination: participants changing their behaviour according to their own hypotheses
36
other threats to external validity in experiments
- unique program features: experimenter in one condition creating a unique environment that isn't present in the other condition
- effects of selection: was recruitment and assignment to conditions successful
- effects of environment/setting: can the results be replicated in other labs/environments
- effects of history: can the results be replicated in different time periods
37
external validity strategies in experiments
- simulation: trying to bring the real world into the lab (mundane realism: how close the lab environment is to the real world / experimental realism: only bringing the psychological aspects of the situation to create immersion)
- field studies: bringing the experimental strategy into the real world (examining bx that are difficult to replicate in a lab)
- strengths: testing hypotheses in realistic environments
- limitations: field studies make it difficult to control all extraneous factors, simulations are dependent on whether the participant believes the simulation is real
38
original simulations
- pre-virtual reality
- Ps have a preference for hypothetical situations as IVs (vivid descriptions) and qualitative self-reports as DVs
39
current-day simulations
- VR headsets
- using realistic immersive stimuli as IVs and quantitative response measures as DVs
- VR influences emotional responses (will be more similar to the emotional response you might see in the real world)
40
perils for experimental designs
- much current research is atheoretical, but without theories, hypotheses are made ad hoc which can be illogical or meaningless
- many measurement instruments have not been tested for their reliability and validity and can be incomparable across studies - use pre-validated tasks when available
- sometimes uses inappropriate research designs = lacking internal validity - should conduct pilot tests with small samples to ensure the roles of IV and DV
- conditions or tasks may be inappropriate = threat to external validity (other participants would have responded differently) because you can't compare across studies or generalize - instead use simple and familiar tasks
- should always conduct manipulation checks
41
individual differences effect on variability
- F ratio compares between-group differences to variances
- with large individual differences, variance increases which can obscure a significant result
- big differences between groups are good (treatment effect), but big differences within groups are bad (variance)
42
ways to reduce within-group variance in between-group designs
- standardize procedures, keep environment constant
- limit individual differences by creating a more homogeneous group (reducing variance and threat of confounds, but also limits external validity - instead, use a factorial design)
- random assignment DOES NOT affect within group variance
- a large sample size can reduce variance (but does so in relation to the sqrt(n) so you need a very large increase)
43
systematic variance in between-groups
- difference in DV between groups
- composed of treatment effect + experimental error (determine if between-group DV differences are due to only experimental error)
44
non-systematic variance in between-groups
- any differences between subjects who are treated alike
- scores varying within groups (individual differences occurring by chance)
- source of error to be minimized
- experimental error: chance factors not being controlled
45
treatment index for between-groups
- F = (between-group variance) / (within-groups variance)
- F = (treatment + experimental error) / experimental error
- F = systematic variance / non-systematic variance
- between-group variance > within-group = large and positive F ratio (good!)
- between-group variance < within-group = F ratio is near 0 (bad)
46
single-factor multiple-group design
- for comparing multiple groups with one IV (single-factor ANOVA)
- provides stronger causal evidence than a two-group design (looking at components individually)
47
major sources of confounds in between-groups
- as a function of the design
- individual differences: want to make sure that groups are as similar as possible, except the IV (assignment bias: process of assignment produces groups with different characteristics like age = threat to internal validity)
- environmental variables: subjects are only tested once in one set of conditions, those characteristics could differ between groups = extraneous variable can become a confound
48
limiting confounds in between-groups designs
- randomization: most powerful way to ensure groups are as equal as possible before treatment (spreading them evenly)
49
random sampling vs. randomization
- sampling: random selection of participants from a larger population
- randomization: assignment of participants to experimental or control groups in a study
50
free random assignment
- each P has an equal chance of being in any condition (like a coin toss)
- theoretically should lead to equal groups, but no guarantee
- improbable that groups will be perfectly matched, but small differences are random and insignificant (neutralizing nuisance differences and maximizing between-group differences)
- with small samples, no guarantees
51
4 steps for matching
1. identify the variables to be matched (potential confounds)
2. measure and rank participants on the variables (pretest)
3. segregate subjects into matched pairs
4. randomly assign pair-members to conditions
52
matching across blocks
- extended to units larger than pairs
- groups of individuals are matched in blocks
- random assignment into groups from blocks
53
threats to internal validity between-groups
- differential attrition: participants leaving one group at a higher rate than for another group (may be affected by individual differences like motivation or environmental like time of day) = groups are no longer equal
- communication between groups: diffusion - treatment effects spreading from one group to another, treatment effects masked by shared information, resentful demoralization - perceived inequity between groups
54
advantages of between-subjects
- simple designs (score are independent of each other)
- clean and uncontaminated by other treatment factors (no carryover or time-related)
- less time required for each participant
- establishes causality
55
disadvantages of between-subjects
- requires many participants (esp. if many treatment conditions)
- difficult to recruit from special populations
- individual and environmental differences
- generalization (external validity) if holding extraneous variables constant
- assignment bias, experimenter expectancy, subject expectancy (try to keep participants, experimenter, analyst blind)
56
when to use between-subjects designs?
- when carryover or comparison effects are expected
- when participants should be using similar anchoring across conditions
- when participants could become sensitized to measurement over time
- to conserve ecological validity when participants would normally be exposed to all levels of IV in real life
- if changes in measurement properties/tests over time are expected
57
threats to internal validity for within-subjects
- environmental: changing from one session to the other (control by holding constant, matching, randomization)
- time-related: history, maturation, instrumentation (bias or decay) (control by shortening the time between measurements, but this can increase the chance of order effects OR control by counterbalancing in time so that these will affect each condition equally)
- regression to the mean
- order effects: progressive error and carry-over/contrast (control by counterbalancing)
58
history
- time-related variable
- events other than treatment that change over time and may affect the scores in one condition differently than another condition
- things that occur in participants' lives (must affect enough participants in one condition to have an effect, like a snowstorm)
59
maturation
- time-related variable
- systematic changes in participant physiology or psychology that occurs during a study and affects scores
- especially concerning in younger or older participants/over a long period of time
60
instrumentation bias/decay
- time-related variable
- changes in a measurement instrument over time (esp. concerning in behavioural observation/when measurements are taken over a long period of time)
61
regression to the mean
- tendency for extreme scores on any measurement to move toward the mean (regress) when the measurement procedure is repeated
- because scores are dependent on both stable (skill) and unstable (chance) factors
- concerning when participants are selected on the basis of a first measurement (exceptionally high or low scores)
62
order effects
- participation in one treatment may have an influence on the participants’ scores in the following treatments
- progressive error, carry-over/contrast
- confound when order effects vary systematically with the treatments + can distort the results
63
progressive error
- order effect due to the general experience of being in a study, not due to a specific treatment
- fatigue: progressive decline in performance as you advance through the conditions
- practice: progressive improvement in performance as you advance through the conditions
64
carry-over effect
- order effect: a specific treatment causes changes in the participants so that the lingering aftereffects of the treatment carry over to the next treatment
- contrast effect: subjective perception of a treatment condition is influenced by its contrast with the previous treatment
65
counterbalancing
- Ps undergo the conditions in different orders to balance/disrupt any systematic relationship between time and order of conditions
- requires separate groups = combining within- and between-group designs
- distributing the order effects equally in all conditions, but doesn't eliminate them (will inflate treatment means, but leave the differences between treatments unaffected)
- it's possible that one treatment has more order effects than another (or certain individuals could be more/less affected by order effects) = asymmetrical carryover effects (if this is the case, the counterbalancing order can interact with the IV to influence DV)
- complete or partial
66
complete counterbalancing
- presenting treatments in every possible sequence
- becomes complex (and impossible) if you have many conditions (which will require more participants to have equal groups)
- number of possible sequences is N! (n-1 * n-2, etc.)
67
partial/Latin-square counterbalancing
- uses enough different orderings to ensure that each treatment condition occurs first in the sequence for one group of participants, occurs second for another group, third for another group, and so on (each Tx occurring at least once in every ordinal position)
- can use a Latin square to decide which sequences to select
- but each condition is preceded and followed by the same conditions
- so you can adjust the Latin square to balance the conditions that precede and follow each condition
68
advantages of within-subjects
- requires fewer participants
- eliminates problems from individual differences because participants serve as their own baseline
- you can examine individual differences = easy to see treatment effects (more statistical power)
69
disadvantages of within-subjects
- time-related threats to internal validity
- order effects
- participant attrition: shrinks sample size and exacerbates volunteer bias (which also affects external validity)
- counterbalancing does not eliminate order effects, just adds them to the groups equally
70
when to use within-subjects
- anticipating large individual differences
- not expecting time-related factors
- difficult to find/recruit participants
71
matched-subject designs
- separate group for each treatment condition, but each individual in one group is matched one-to-one with an individual in every other group (on variables related to the study)
- approximating the advantages of between- and within- designs (eliminating individual differences and using separate groups = no time-related differences)
- can be tedious or impossible depending on how many variables you're matching
72
two-treatment design
- simplest application of within-subjects design
- easy to maximize the difference between conditions, but not demonstrating a functional difference between variables
- easy to counterbalance
- single-factor repeated-measures ANOVA (ratio or interval data)
- Wilcoxon Signed-Ranks test (ordinal data)
73
multiple-treatment design
- within-group design application
- functional relationship between variables
- more convincing cause-and-effect
- too many conditions = differences between them are too small to reveal an effect
- many conditions = more attrition
- difficult to counterbalance
74
F-ratio for within-subjects
- between-condition variance (which is based on within-subject comparisons) / within-condition variance
- individual differences contribute equally to the numerator and denominator, so can be removed
- no assumption of independence between scores
75
time trade-off in within-subjects
- increasing the amount of time between conditions increases the risk of time-related threats to internal validity (history, maturation, instrumentation)
- decreasing the amount of time increases the risk of order effects
76
reversibility in within-subjects
- IVs that permanently affect the development or state of participants in **irreversible carryover effects** (skills that cannot be unlearned, brain lesions, some medications)
- if this is the case, you can't use within-subjects (like ABA')
77
Reversal design
- ABA' design: measuring the presence/absence of carryover effects (how is behaviour affected after the intervention is removed - does it go back to baseline?)
78
single-subject design
- only using one participant, but makes unambiguous cause-and-effect conclusions
- uses elements of descriptive case studies and quasi-experimental time-series designs
- experimental = control of extraneous variables + manipulation of IV (the timing of the phase switches is the manipulation)
- baseline phase measurement + repeated measurement + replication are the three components
- minimum of 3 observations to constitute a phase (usually 5-6 to establish a pattern, more if data is highly variable)
- the type of IV and DV you're measuring will determine the required length of observation during baseline + repeated measurement (varies daily? varies seasonally?)
- critical factor is stability of the data to establish a level/trend of behaviour
- unit of analysis is the single participant undergoing the observations
79
level of behaviour (single-subject)
- magnitude of the participant's responses (to demonstrate stability during phases)
- a horizontal line on a graph
80
trend of behaviour (single-subject)
- consistent increase/decrease in the magnitude of behaviour in a phase (still a consistent pattern)
- doesn't need to be linear, but linear is the easiest for subsequent analysis
81
how to deal with unstable data (single-subject)
- keep observing/measuring and hope that the behaviour will stabilize (maybe the participant is reacting to being measured and could habituate)
- average over a set of two or more observations (there should be a rationale for which observations you average)
- look for patterns within the inconsistency
82
phase change (single-subject)
- manipulation (implementation, withdrawal, change treatment)
- only do this when a pattern has been established in the previous phase
- if baseline phase is showing an improvement trend in behaviour = DON'T implement the treatment (unnecessary and will lead to ambiguous results), maybe the improvement is part of a cycle and they will regress again
- if baseline phase is showing declining/dangerous level/trend in behaviour = implement treatment immediately (reassess what to do when behaviour has improved)
- if a treatment produces an immediate deterioration in bx = stop/modify the treatment
- phase changes are always based on the participant's response
83
causal results in single-subject
- change in behaviour is immediate and large = convincing evidence
- change in average level (clear difference in average levels)
- immediate change in level (comparing the last observation to the first observation)
- change in trend: from no trend (stable level) to consistent trend or a change in direction of trend
- latency of change: a delay undermines the causal interpretation
- when interpreting a graph, visual aids like representing 2SDs from the mean can help determine if the change is important
- visual aids like regression lines for trends
84
single-subject reversal design
- ABAB (more causally valid than AB)
- evidence for treatment = pattern of behaviour in each treatment phase is different from patterns in each baseline phase + the same changes for each phase change point
- only the second phase (from B back to A) change provides causal evidence (the first one could be due to chance)
- the last phase change should replicate the first = good causal evidence
- withdrawing treatment may not produce a change in bx (treatment having lasting effects?)
- withdrawing a treatment that appears to be working has ethical ramifications (but it's a necessary practical necessity and is only a temporary withdrawal)
- adding/modifying treatments undermines causal explanations and threatens internal validity (you can go back to baseline, then re-starting the successful modified treatment to replicate)
85
multiple-baseline single-subject
- begin with two simultaneous baseline phases but initiate treatments at different times
- no return to baseline (good for treatments with long-lasting effects), but replicating the phase change for another participant or another behaviour
- clear and immediate change in pattern of behaviour + at least TWO replications (3 total participants showing the same results)
- clarity of results can be impacted by individual differences between participants or behaviours (one behaviour showing large/immediate change, the other smaller/gradual or if one participant has variability in their baseline that makes it difficult to see the treatment effect)
86
multiple-baseline across participants
- initiate the treatment for one participant, continue baseline for another and initiate treatment later
87
multiple baseline across behaviours
- initial baseline phases correspond to two different behaviours for the same participant
- targeting each behaviour individually as long as the behaviours are independent of one another
- if behaviours are too similar, the treatment could generalize to the other behaviour
88
multiple baseline across situations
- initial baseline phases correspond to the same behaviour in different situations
- repeat for the same behaviour that is manifesting in various situations; administer the same treatment at different times for the two situations
89
component analysis design
- parts of the treatments are added or withdrawn during different phases (to evaluate how each component contributes to the overall treatment)
- can start with full treatment, then remove components one by one or add them until you reach the full treatment
- reversal design: baseline + series of phases adding treatments one by one + return to baseline + same series of treatment additions
- multiple baseline: repeating the effects for a different participant/behaviour/situation with different components added sequentially
90
AB design
- not causally valid, but the simplest single-case experimental design
- baseline (A phase) observations - treatment (B phase) observations (you can also have additional treatment phases like C)
91
advantages of single-case designs
- cause-and-effect relationships with only one participant
- flexibility: free to modify treatment or change to a new one if the participant isn't responding (designing your treatment according to the individual)
- no need to standardize treatments across participants (only 1 participant is used)
- much cheaper, useful for rare population, less time-consuming
92
disadvantages of single-case designs
- relationship between variables may be specific to your participant (threat to external validity)
- multiple continuous observations are required (con if you're using something expensive of time-consuming)
- according to the textbook, absence of statistical controls/analysis/reliance on a graph (according to lecture, there are statistical tests of significance)
93
factorial design and applications
- using multiple IVs (factors) or quasi-independent, each of which has levels that make up a matrix
- 2 x 3 x 2 is a three-factor design in which the first factor has two levels, the second has 3 levels, the third has 2 levels (total of 12 conditions because it is fully-crossed)
- useful to examine a real-world behaviour because behaviour is usually affected by a multitude of variables in interaction (higher ecological validity than one-IV designs)
- ANOVA for statistical significance (a two-way has three null hypotheses)
- can be within- (repeated-measures ANOVA) or between- (independent-measures ANOVA) or mixed
- can expand and replicate previous research: are treatment effects (factor A) still present under different conditions (factor B)
- can be used to reduce variance in between-subjects designs: instead of holding a variable constant (and limiting external validity), introduce it as an additional factor so that nay differences become between-group instead of within-group
- can be used to evaluate order effects
94
main effects and interactions factorial designs
- main effects: treatment differences between levels of a given factor (comparing the means of columns together + comparing the means of rows together)
- interaction: combined effect of factors - comparing cell means (the effects of one factor depends on the level of the other factor so the combination of factors produces a unique result than just looking at individual factors), indicated by non-parallel lines in a graph
- interactions in a table: size or direction of the difference between cell means of rows is difference across columns
- always consider an interaction effect before considering main effects because interactions can distort main effects
95
combined factorial design strategy
- one factor is experimental, the other is nonmanipulated (quasi-independent) like a pre-existing characteristic (person-by-environment design) OR time (which is not manipulated)
96
pretest-posttest control group design (factorial)
- example of a two-factor mixed design (treatment group + control group is between-subjects, and each participant is measured twice so within-subjects)
- can be quasi-experimental when using nonequivalent groups
- can be experimental if you're using random assignment to treatment and control groups
97
higher-order factorial designs
- involving three or more factors
- 3 main effects + 3 two-way interactions + 1 three-way interaction (7 hypotheses)
- three-way: the two-way interaction depends on the level of the third factor
- four-way: 4 main effects + 6 two-way interactions + 4 three-way interactions + 1 four-way interaction (interpretation can be very difficult/not practical, so should avoid four-way factorial designs unless you have clear predictions beforehand)
98
factorial designs for order effects
- mixed design with treatment as within-subjects factor and the order of treatments as between-subjects factor
- makes it possible to examine order effects that may exist & to reduce variance
- could lead to no order effects: no interaction
- could lead to symmetrical order effects: interaction (lines crossing exactly at the middle)
- could lead to asymmetrical order effects: interaction (lines not crossing at their midpoints)
99
variance in factorial designs
- small overlap in variances = easier to assume means differ significantly
- large overlap in variances: means of X, Y, 2 Factors will likely not differ significantly
- ANOVA is required to determine if the difference in mean values exceeds the variance in the factors sufficiently
100
pure factorial designs
- all factors are being manipulated, between-groups design (participants are randomly assigned to each cell/condition)
- guaranteed no carryover effects, no order effects
- can require many participants (especially if you have many factors with many levels)
- individual differences can become confounds
- best to use when many Ps are available, individual differences are small, order effects could be a problem
101
within-subjects factorial designs
- single group participants in all cells/conditions
- many factors = Ps experience many conditions = high risk of attrition because it's time-consuming
- higher chance of testing effects (practice, fatigue)
- difficult to counterbalance orders to control order effects
- fewer Ps required, reduces individual differences (best when individual differences are large and order effects won't be a problem)
102
mixed factorial designs
- combines between- and within- (used when one factor is expected to threaten validity, so you want the advantages of between- for one factor and the advantages of within- for the other factor)
- compromise in the number of participants required while still being able to test the interaction
103
factorial designs advantages
- highly efficient: effects of many factors simultaneously + interactions + can replicate and expand prior research at the same time
- instead of holding individual differences constant, they can become another factor
- high external validity
104
factorial designs disadvantages
- more chance of having confounds than single-IV designs, but still the same difficulties when controlling for them
- if factors aren't manipulated, your interpretation is no better than for correlational research
- too many factors can make interpretations confusing
- multiple statistical tests = more stringent alpha level (less power)
105
nonexperimental and quasi-experimental research strategies
- non: failing to control at least one variable that is a threat to internal validity (that cannot be removed by design)
- quasi: an attempt is made to minimize the threat to internal validity (usually a control group or multiple measures to control for confounds)
- usually the problematic variable is pre-existing groups (groups defined in terms of participant characteristics) or time variable (pre/post)
- not using random assignment = not a true experiment
- use a quasi-independent variable: not manipulated by the experimenter, so not truly an experiment (the variable used to differentiate groups of scores being compared)
106
differential design
- NONEXPERIMENTAL nonequivalent group design (simplest between-group design)
- ex-post-facto (after the fact) design
- looking a differences between pre-existing groups (based on individual differences/variable like gender/other individual differences) - no manipulation or control over assignment to groups, just comparing differences on DV
- similar to correlational research (same interpretation) but here a variable is used to separate people into groups (in correlational research, you're just measuring individuals as one group)
107
nonequivalent control group design
- NONEXPERIMENTAL
- evaluating the effectiveness of a treatment given to a pre-existing group and using a similar (but nonequivalent) group as a control
- lacks random assignment
- post-test only nonequivalent control group design is one type
108
posttest only nonequivalent control group design
- NONEXPERIMENTAL nonequivalent group design
- one group is given a treatment and measured afterwards + compared with a nonequivalent control group that is only measured after not receiving the treatment (no manipulation, no control over assignment = assignment bias)
X O
O
- often comparing clusters of people receiving the same treatment like the same algebra class (not administering treatment individually)
- if random assignment is used, then the groups are no longer nonequivalent
R X O
R O
109
pretest-posttest nonequivalent control group design
- QUASI-EXPERIMENTAL design with nonequivalent groups
- including a pretest for all groups before the treatment is administered to help with internal validity
O X O (treatment group)
O O (nonequivalent control group)
- allows the researcher to evaluate individual differences beforehand to see if groups are similar (measuring one variable, so the threat to internal validity is reduced, but not eliminated)
- time-related variables are also reduced because if groups are similar beforehand, then go through the same length of time, they should have changed similarly (except if they undergo differential history effects - some event affects one group, but not the other, differential maturation, instrumentation, regression, testing effects)
- still no random assignment, so you can't say the groups are equivalent
110
pre-post design
- evaluating the influence of a treatment of event by comparing prior to treatment to after treatment scores for one group (observations over time for one group of participants)
- no control group
- time-related variables are threats to internal validity
- no counterbalancing, but otherwise similar to within-subjects
- includes nonexperimental pretest-posttest design & quasi-experimental pre-post time series design
111
one-group pretest-posttest design
- NONEXPERIMENTAL
- O X O (one group only, one observation before and one after)
- no attempt made to control internal validity
- goal: evaluate an intervention by comparing observations before and after
- no assignment bias (there's only one group)
- cannot counterbalance treatments = threatened by time-related variables (history, instrumentation, maturation, regression, testing effects)
112
pre-post time series design
- QUASI-EXPERIMENTAL
- using a series of observations pretest and posttest (O O O X O O O) to add some control
- no random assignment, so not experimental
- treatment can be either manipulated (treatment) or nonmanipulated (like a natural event or law = interrupted time-series design)
- pretest observations allow the researcher to identify trends before the treatment (practice/fatigue, instrumentation, maturation, regression) so that you can reasonably assume that post-test differences are caused by the intervention
- history is a threat to internal validity IF it occurs simultaneously with the treatment administration (if it occurs at another time, you can remove its effects in analysis)
- posttest observations allows you to identify trends/if the effect remains stable over time (or if it fades)
- this design is often applied to individuals in single-case designs (using at least 3 observations)
- stats: time series analysis (similar to single-case designs)
113
developmental research designs
- NONEXPERIMENTAL research used to study changes in behaviour/other variables related to age
- longitudinal: one group measured 2+ times (within-groups)
- cross-sectional: 2+ groups measured once (between-groups)
- cross sectional longitudinal: 2+ groups measured 2+ times (between and within groups)
114
developmental longitudinal design
- measuring a variable in the same group of individuals over a period of time (every few months or years)
- individuals are usually cohorts of people roughly the same age/same environment
- like a within-subjects NONEXPERIMENTAL one-group pretest-posttest design (but no treatment administered, just age)
- like a time series design (observation, then period of treatment/aging, then observation +)
- no cohort effects and you can see how behaviour changes with age
- very time-consuming + expensive, high risk of participant attrition (differential attrition) or participant mortality = threat to internal validity
- Ps are measured many times = risk of practice/testing effects due to experience with measurement
115
developmental cross-sectional design
- between-subjects NONEXPERIMENTAL design that uses a different group for each age being studied, then compare the groups based on one measurement
- nonequivalent groups (using preexisting groups based on age)
- exactly the same as a differential design, but the variable is age, so it's cross-sectional
- can't see how participants change over time, susceptible to cohort effects (participants are different in more ways than age because they grew up in different environments which is a threat to internal validity)
- time-efficient and no risk of attrition (not long-term)
116
developmental cross-sectional longitudinal design
- mixed developmental design (NONEXPERIMENTAL)
- comparing results obtained from different samples (cross-sectional) and obtained at different times (longitudinal)
- usually examining some other factor than aging - how behaviours develop according to factors other than age
- can draw comparisons both between groups and across time
117
between-groups non/quasi-experimental designs
- differential design: NONEXPERIMENTAL
- posttest only nonequivalent group design: NONEXPERIMENTAL
- pretest-posttest nonequivalent group design: QUASI-EXPERIMENTAL
118
nonequivalent
- pre-existing groups used: groups differentiated by one specific factor (that you expect to be responsible for DV differences between groups)
- experimenter cannot control group membership (or individual differences)
- problem of assignment bias: groups may differ based on other factors than group membership = possible confound that clouds cause-and-effect relationship
119
within-group non/quasi-experimental designs
- one-group pre-post design: NONEXPERIMENTAL
- time series design: QUASI-EXPERIMENTAL
120
replicability
- whether a study's published findings can be repeated in a different lab using the same or similar methods (using a whole new set of participants)
- often the gold standard for the accuracy of a finding
121
reproducibility
- the ability of a different researcher to reproduce a different researcher's published analyses, given the original data set and computer code for statistics used
122
replication crisis
- failure for published research findings to be replicated in other labs with similar methods
- only about 36% of 100 studies that were replicated reached significance p<.05
- 50% of cognitive psychology, 26% of social psychology
- also in other sciences: 70% failed to replicate in other labs, 50% failed to replicate by the same researchers
- psychology especially important: use of human participants introduces lots of variability
123
enhancing replicability
- document the research methods carefully
- run the study again before publishing
- ask a lab member to run the study
124
social psych experiments that failed to replicate
- Stanford prison experiment
- bystander effect (situational and contextual factors have a more important role than assumed)
- stereotype threat (especially in real-world setting)
125
cognitive psych experiments that failed to replicate
- spotlight attention: individuals focusing attention on a part of their visual field - may be more distributed/flexible than assumed
- dual-process model of memory: unconscious vs. conscious memory
- mirror neurons: seen some inconsistencies
126
developmental psych experiments that failed to replicate
- 'critical period' for language acquisition: more individual variation than assumed
- attachment theory/types: more fluid and context-dependent than assumed
- mozart effect on infant intelligence: not always leading to cognitive enhancement
127
clinical psych experiments that failed to replicate
- dodo bird verdict: the therapeutic relationship doesn't exclusively account for positive outcomes, like was assumed
- power of positive thinking in health outcomes
- memory recovery techniques: hypnosis and guided imagery to recover repressed memories = more unreliable 'recovered' memories than assumed
128
neuroscience experiments that failed to replicate
- amygdala in fear processing: maybe less involved than assumed
- left/right brain distinction: both hemispheres involved in a variety of functions
- 'brain-training' effect
129
causes of replication crisis
1) ignoring or misunderstanding statistics: misunderstanding null hypotheses or the meaning of p values, small sample sizes, effect sizes and power
2) publication bias: conducting, publishing, funding science
3) falsifying data
4) quality of replication (failure to follow original procedures or due to incomplete method descriptions in the original study)
130
poor hypothesis practices replication crisis
- Hypothesizing After Results are Known: formulating or changing hypotheses after analyzing the data (could be exploring data without a hypothesis and then coming up with one after = confirmation bias)
- HARKing goes against the scientific principle that the IV causes (precedes) the DV (you shouldn't be looking for another IV after seeing results and then saying that it caused your results)
131
SHARKing
- secretly HARKing: “publicly presenting in the Introduction section of an article hypotheses that emerged from post hoc analyses and treating them as if they were a priori.”
- not making hypotheses based on existing research (literature), but based on the data at hand
- never justified in science, ethically wrong
132
THARKing
- transparently HARKing: “clearly and transparently presenting new hypotheses that were derived from post hoc results in the Discussion section of an article.”
- promoting effectiveness of science, may even be ethically required
- helps to reduce the motivation to engage in SHARKing + provides a more compelling narrative
- presenting new post hoc analyses with healthy skepticism so that the reader decides how to treat them, may spark new research into new hypotheses
- necessary to capture complexities that human deduction can miss, allows us to discover things accidentally
- could increase power and lower Type II rates while also preventing Type I error rates that occur from SHARKing
- no deception + providing avenues for research
133
meaning of p values replication crisis
- null hypothesis significance testing (NHST): assumption that there is no significant difference between groups/conditions
- with a large enough N, every study would yield a significant result (any stimuli is likely to have some effect, so if you have enough power to detect that effect it will be significant)
- p-hacking
134
p-hacking
- manipulating statistical analyses to achieve statistically significant results = false positive findings
- researchers should ethically only compare/analyze what they said they would beforehand (also define your outliers beforehand)
135
methods of p-hacking
- continue collecting data and only stop once you reach p<.05
- analyze many measures but only report those with p<.05
- collect and analyze many conditions, but only report those with p<.05
- use covariates (other variables that could account for differences) until you reach p<.05
- exclude participants to get p<.05
- transform the data to get p<.05
136
cherry-picking data
- selectively reporting only the data or results that support their hypotheses and disregard data that is contradictory = biased research findings
137
data fabrication and falsification
intentionally creating or altering data to support desired outcomes - clearly ethically wrong
138
publication biases replication crisis
- file drawer problem: studies with non-significant results are less likely to be published or reported (solution: publish a meta-analysis that uses all data, including non-significant results)
- selective reporting: researchers/institutions tend to favour publications of significant or novel findings (solution: have journals that agree to publish null results when research methods are strong)
- incomplete knowledge: if non-significant results aren't available in the literature, the scientific community gets a biased view of a topic (solution: larger sample sizes, reducing confounds)
- replication challenges: researchers attempt to replicate findings and encounter difficulties when null results are in file drawer and aren't being considered
139
solutions to publication biases
- open science practices
- pre-registration of research methods: detailed plan filed ahead of data collection (your hypotheses, specific methodologies - sampling participants, IVs, DVs, design materials, procedure, analyses) so you get a unique DOI (you can also do this for studies using secondary data)
- registered reports: peer review of research methods prior to data collection (if they're quality, they're accepted for publication regardless of results as long as you follow your plan)
- pre-registration is much more common that registered reports
- both encourage a priori plans, help reduce p-hacking and HARKing, reduce selection bias, encourage open data
140
pros of registered reports
- you get feedback earlier and you can incorporate it into your design to make for a stronger study (traditional publications only get reviewed after collecting and analyzing data)
- reduces publication bias, gives you guaranteed publication (which pre-registration doesn't do)
141
pre-registration vs. registered reports
- pre-registration on its own doesn't improve replication rates (but it does improve the perception of higher quality research)
- registered reports may improve replication rates (but won't solve the publication bias)
142
course of action to promot THARKing
- journals having a policy for mandatory post hoc analyses in the discussion section or a section to include them in reports (especially when results are non-significant and alternative hypotheses can be helpful)
- action editors and reviewers should continue to suggest post hoc analyses to be run, but explicitly say to include them in the discussion section
- reviewers should spot when introduction sections don't make sense according to the current literature to stop HARKing
WINTER 2024
flashcards
Decks in class (28)
# Cards
-
412 Chapter 141
-
412 Chapter 245
-
412 Statistics14
-
412 Research38
-
ADHD33
-
Disruptive Behaviour Disorders54
-
337 DSM criteria26
-
412 DSM criteria49
-
412 DSM criteria midterm 233
-
412 Substance Use14
-
412 Anxiety33
-
412 Depression38
-
412 IDD34
-
412 ASD34
-
337 M1118
-
337 criteria M220
-
337 suicide and NSSI42
-
337 SUD/AUD49
-
337 personality26
-
337 BPD & ASPD54
-
337 Schizophrenia53
-
412 post-M284
-
412 pre-M1217
-
412 pre-M2187
-
306 final142
-
337 final criteria45
-
337 final342
-
337 post-M248
