Adding More Terms

When we add more terms to our model, we can ask two different questions about the percent of variance explained by a predictor - how much variance is accounted by the predictor in total, and how much is accounted when controlling for any other predictors. The latter questions is answered by the partial-Eta squared (η_p²), which is the percent of the partial variance (after accounting for other predictors in the model) associated with a term:

$$ \eta^2_p = \frac{SS_{effect}}{SS_{effect} + SS_{error}} $$ which can also be accessed via the eta_squared() function:

m <- lm(value ~ gender + phase + treatment, data = obk.long)

eta_squared(m, partial = FALSE)

> # Effect Size for ANOVA (Type I)
> 
> Parameter |       η² |       95% CI
> -----------------------------------
> gender    |     0.03 | [0.00, 1.00]
> phase     | 9.48e-03 | [0.00, 1.00]
> treatment |     0.25 | [0.11, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

eta_squared(m) # partial = TRUE by default

> # Effect Size for ANOVA (Type I)
> 
> Parameter | η² (partial) |       95% CI
> ---------------------------------------
> gender    |         0.04 | [0.00, 1.00]
> phase     |         0.01 | [0.00, 1.00]
> treatment |         0.26 | [0.12, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

(phase is a repeated-measures variable, but for simplicity it is not modeled as such.)

In the calculation above, the SSs were computed sequentially - that is the SS for phase is computed after controlling for gender, and the SS for treatment is computed after controlling for both gender and phase. This method of sequential SS is called also type-I test. If this is what you want, that’s great - however in many fields (and other statistical programs) it is common to use “simultaneous” sums of squares (type-II or type-III tests), where each SS is computed controlling for all other predictors, regardless of order. This can be done with car::Anova(type = ...):

eta_squared(car::Anova(m, type = 2), partial = FALSE)

> # Effect Size for ANOVA (Type II)
> 
> Parameter |       η² |       95% CI
> -----------------------------------
> gender    |     0.05 | [0.00, 1.00]
> phase     | 9.22e-03 | [0.00, 1.00]
> treatment |     0.24 | [0.11, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

eta_squared(car::Anova(m, type = 3)) # partial = TRUE by default

> # Effect Size for ANOVA (Type III)
> 
> Parameter | η² (partial) |       95% CI
> ---------------------------------------
> gender    |         0.07 | [0.01, 1.00]
> phase     |         0.01 | [0.00, 1.00]
> treatment |         0.26 | [0.12, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

η_p² will always be larger than η². The idea is to simulate the effect size in a design where only the term of interest was manipulated. This terminology assumes some causal relationship between the predictor and the outcome, which reflects the experimental world from which these analyses and measures hail; However, η_p² can also simply be seen as a signal-to-noise- ratio, as it only uses the term’s SS and the error-term’s SS.[^in repeated-measure designs the term-specific residual-SS is used for the computation of the effect size].

(Note that in a one-way fixed-effect designs η² = η_p².)

Adding Interactions

Type II and type III treat interaction differently. Without going into the weeds here, keep in mind that when using type III SS, it is important to center all of the predictors; for numeric variables this can be done by mean-centering the predictors; for factors this can be done by using orthogonal coding (such as contr.sum for effects-coding) for the dummy variables (and NOT treatment coding, which is the default in R). This unfortunately makes parameter interpretation harder, but only when this is does do the SSs associated with each lower-order term (or lower-order interaction) represent the SS of the main effect (with treatment coding they represent the SS of the simple effects).

# compare
m_interaction1 <- lm(value ~ treatment * gender, data = obk.long)

# to:
m_interaction2 <- lm(
  value ~ treatment * gender,
  data = obk.long,
  contrasts = list(
    treatment = "contr.sum",
    gender = "contr.sum"
  )
)

eta_squared(car::Anova(m_interaction1, type = 3))

> Type 3 ANOVAs only give sensible and informative results when covariates
>   are mean-centered and factors are coded with orthogonal contrasts (such
>   as those produced by `contr.sum`, `contr.poly`, or `contr.helmert`, but
>   *not* by the default `contr.treatment`).

> # Effect Size for ANOVA (Type III)
> 
> Parameter        | η² (partial) |       95% CI
> ----------------------------------------------
> treatment        |         0.12 | [0.02, 1.00]
> gender           |     9.11e-03 | [0.00, 1.00]
> treatment:gender |         0.20 | [0.07, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

eta_squared(car::Anova(m_interaction2, type = 3))

> Type 3 ANOVAs only give sensible and informative results when covariates
>   are mean-centered and factors are coded with orthogonal contrasts (such
>   as those produced by `contr.sum`, `contr.poly`, or `contr.helmert`, but
>   *not* by the default `contr.treatment`).

> # Effect Size for ANOVA (Type III)
> 
> Parameter        | η² (partial) |       95% CI
> ----------------------------------------------
> treatment        |         0.27 | [0.13, 1.00]
> gender           |         0.12 | [0.03, 1.00]
> treatment:gender |         0.20 | [0.07, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

If all of this type-III-effects-coding seems like a hassle, you can use the afex package, which takes care of all of this behind the scenes:

library(afex)

> Loading required package: lme4

> Loading required package: Matrix

> ************
> Welcome to afex. For support visit: http://afex.singmann.science/

> - Functions for ANOVAs: aov_car(), aov_ez(), and aov_4()
> - Methods for calculating p-values with mixed(): 'S', 'KR', 'LRT', and 'PB'
> - 'afex_aov' and 'mixed' objects can be passed to emmeans() for follow-up tests
> - Get and set global package options with: afex_options()
> - Set sum-to-zero contrasts globally: set_sum_contrasts()
> - For example analyses see: browseVignettes("afex")
> ************

> 
> Attaching package: 'afex'

> The following object is masked _by_ '.GlobalEnv':
> 
>     obk.long

> The following object is masked from 'package:lme4':
> 
>     lmer

m_afex <- aov_car(value ~ treatment * gender + Error(id), data = obk.long)

> Contrasts set to contr.sum for the following variables: treatment, gender

eta_squared(m_afex)

> # Effect Size for ANOVA (Type III)
> 
> Parameter        | η² (partial) |       95% CI
> ----------------------------------------------
> treatment        |         0.27 | [0.13, 1.00]
> gender           |         0.12 | [0.03, 1.00]
> treatment:gender |         0.20 | [0.07, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

Unbiased Effect Sizes

These effect sizes are unbiased estimators of the population’s η²:

• Omega Squared (ω²)
• Epsilon Squared (ϵ²), also referred to as Adjusted Eta Squared.

omega_squared(m_afex)

> # Effect Size for ANOVA (Type III)
> 
> Parameter        | ω² (partial) |       95% CI
> ----------------------------------------------
> treatment        |         0.24 | [0.10, 1.00]
> gender           |         0.10 | [0.02, 1.00]
> treatment:gender |         0.17 | [0.05, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

epsilon_squared(m_afex)

> # Effect Size for ANOVA (Type III)
> 
> Parameter        | ε² (partial) |       95% CI
> ----------------------------------------------
> treatment        |         0.25 | [0.11, 1.00]
> gender           |         0.11 | [0.02, 1.00]
> treatment:gender |         0.18 | [0.06, 1.00]
> 
> - One-sided CIs: upper bound fixed at [1.00].

Both ω² and ϵ² (and their partial counterparts, ω_p² & ϵ_p²) are unbiased estimators of the population’s η² (or η_p², respectively), which is especially important is small samples. Though ω² is the more popular choice (Albers and Lakens 2018), ϵ² is analogous to adjusted-R² (Allen 2017, 382), and has been found to be less biased (Carroll and Nordholm 1975).

Generalized Eta²

Partial Eta squared aims at estimating the effect size in a design where only the term of interest was manipulated, assuming all other terms are have also manipulated. However, not all predictors are always manipulated - some can only be observed. For such cases, we can use generalized Eta squared (η_G²), which like η_p² estimating the effect size in a design where only the term of interest was manipulated, accounting for the fact that some terms cannot be manipulated (and so their variance would be present in such a design).

eta_squared(m_afex, generalized = "gender")

> # Effect Size for ANOVA (Type III)
> 
> Parameter        | η² (generalized) |       95% CI
> --------------------------------------------------
> treatment        |             0.21 | [0.08, 1.00]
> gender           |             0.10 | [0.02, 1.00]
> treatment:gender |             0.18 | [0.06, 1.00]
> 
> - Observed variables: gender
> - One-sided CIs: upper bound fixed at [1.00].

η_G² is useful in repeated-measures designs, as it can estimate what a within-subject effect size would have been had that predictor been manipulated between-subjects (Olejnik and Algina 2003).

Cohen’s f

Finally, we have the forgotten child - Cohen’s f. Cohen’s f is a transformation of η_p², and is the ratio between the term-SS and the error-SS.

$$\text{Cohen's} f_p = \sqrt{\frac{\eta^2_p}{1-\eta^2_p}} = \sqrt{\frac{SS_{effect}}{SS_{error}}}$$

It can take on values between zero, when the population means are all equal, and an indefinitely large number as the means are further and further apart. It is analogous to Cohen’s d when there are only two groups.

cohens_f(m_afex)

> # Effect Size for ANOVA (Type III)
> 
> Parameter        | Cohen's f (partial) |      95% CI
> ----------------------------------------------------
> treatment        |                0.61 | [0.38, Inf]
> gender           |                0.37 | [0.17, Inf]
> treatment:gender |                0.50 | [0.28, Inf]
> 
> - One-sided CIs: upper bound fixed at [Inf].