This chapter will help you to:
The great benefit of the general linear model is the ability to account for various designs into a single statistical model. For this course, the pennacle of this generality is the “mixed factorial ANOVA.” As the term “factorial” implies, we will be including multiple indpendent variables. The term “mixed” tells us that we will be combining both within- and between-subjects factors.
The best thing about ending the course on this topic is that it is almost all review. We really are just combining the two types of factorial ANOVAs already discussed.
We see that wit the assumptions.
The mixed factorial ANOVA has three assumptions that we’ll need to verify.
Normality of DV within each combination of levels of IVs. This was a common assumption for both between- and within-subjects designs.
Homogeneity of variance for between-subjects IVs. This will appear slightly differently than before because we will have to check this assumption for across the DV scores in each level of the wtihin-subjects factor.
Sphericity or equality of variance within-subjects IV. As in the original within-subjects ANOVA, we’ll want to verify that the difference scoress across the combined levels of the variables are roughly equal.
The example we’ll be working with this week involves the changes to SAT scores across the number of test attempts and the impact of a music education background. Table 1 shows a sample of a data set to illustrate the design.
Table 1
Split Plot Design of Example
| Test Attempt | ||||
|---|---|---|---|---|
| First | Second | Third | ||
| Music Education Background | Yes |
P1 = 1261 P2 = 1275 P3 = 1232 P4 = 1314 P5 = 1238 |
P1 = 1349 P2 = 1361 P3 = 1316 P4 = 1396 P5 = 1326 |
P1 = 1415 P2 = 1430 P3 = 1384 P4 = 1470 P5 = 1390 |
| No |
P6 = 1216 P7 = 1179 P8 = 1188 P9 = 1181 P10 = 1203 |
P6 = 1221 P7 = 1181 P8 = 1190 P9 = 1186 P10 = 1210 |
P6 = 1223 P7 = 1182 P8 = 1192 P9 = 1185 P10 = 1211 |
|
Pay particular attention to the participants. The same particiapnt are in all three attempts for having no music education background but another group of participants are in the three attempts without having a music education background.
The difference in participants will be important when we calculate (or rather, have SPSS calculate) the error terms.
Please navigate to the “Dataset” folder in the “Files” section of our Canvas site and download the “SAT.sav” file.
Figure 2 is a screenshot of the dataview of the SPSS file. We see five columns. The first, “Participant,” is just a marker to tell us from who each set of data belongs. The second column is likely dummy coded. Columns 3 through 5 have what look live continuous data (perhaps our DV).
Figure 2
Data View of SAT.sav
An examination of the variable view will help us to determine which variables contain the DV and which contain the IVs.
As we see in Figure 3, the last three variables have a common phrase in the “label” column that suggests to us the DV. “SAT Score” seemes to be what was repeatedly measured across the three test conditions.
As such, “SAT scrore” is the DV (review the factorial within-subjects ANOVA for more on the process of DV identification).
Figure 3
Variable View of SAT.sav
It is important to be specific as we identify the independent variables. That is, we need to specify which IV is a between-subjects IV and which is a within-subjects IV because they will need to be entered into our GLM procedure differently.
Between-subjects factors, by definition, change levels between subjects. Because the SPSS data file is organized in rows by participant, this means each row should only have one value for a between-subjects variable.
This is the case for “MusicEd” as participants either have a 1 or 0 recorded (see the data view). When we check the variable view (see Figure 3), the label informs us that this variable is binary (“Yes” or “No”) regarding the participants Music Education Background. That is, this variable tells us if a person has some music education in their background (before taking the SAT) or not.
The within-subjects factors change levels within subjects. In SPSS, that means we should have multiple levels of a within-subjects variable per row. It does not seem like that is the case if we just check the values in each row. We actually need to check the variable labels to help us determine the within-subjects IV.
Recall that another term for a within-subjects design is a repeated measures design. We saw that we repeatedly measured SAT scores, but why? We measured SAT scores for multiple test attempts. The labels for the last three variables (see Figure 3) indicate just that; “SAT Score on First Test, SAT Score on Second Test, SAT Score on Third Test.” Our within-subjects IV is thus “Test attempt.”
With our variables identified, we can surmise the research question to be: “Does having a music education background influence the change on SAT Scores across mutliple test attempts?”
Before we check our distirbutions, we need to ensure that we have the correct level of analysis. That is, we need to be sure that we are looking at the distributions of the DV produced by the intersection of the IVs.
We neeed to first split the file by the between-subjects IV (“MusicEd”) by clicking on “Data” in the menu bar and then clicking “Split File”. Drag “MusicEd” to the “Groups based on” box. I like to click the “Compare groups” option to keep my output organized. Click paste to generate the syntax.
To help us judge normality (and flag some potential outliers), we’ll produce Q-Q plots.
Click on the “Analyze” menu, then hover over “Descriptive Statistics”, then click on “Q-Q Plots” (See Figure 4).
Figure 4
Navigating to Q-Q Plots in Menu Bar
The Q-Q plot window will open. You will need to move each variable that contains the DV scores (i.e., the within-subjects factors variables) from the box on the left to the box on the right (see Figure 5).
Figure 5
Q-Q Plots Window
Click “Paste” to generate the syntax.
Outliers can lead to deviations from normality. We can easily check for outliers by asking SPSS to convert our DV scores to z-scores. We have already split the file before creating Q-Q plots, so we need only go to the “Descriptives…” menu under “Analyze” and “Descriptives.”
Drag the same variables that you used in creating the Q-Q plots (i.e., “test1”,“test2” and “test3”) to the “Variable(s):” box. Click on the “save standardized values as variables” box below the list of variables.
Click paste to generate the syntax.
Navigate to the syntax editor window then type “SPLIT FILE OFF.” at the end of your syntax (see Figure 6). Select all the syntax and run it.
Figure 6 contains the Q-Q plots, arranged by IV levels.
Figure 6
Q-Q plots of SAT scores by Music Education and Test Attempt
| Music Education | Test Attempt | ||
|---|---|---|---|
| First | Second | Third | |
| No |
|
|
|
| Yes |
|
|
|
All of these plots seem to indicate that our samples are roughly normally distributed. Although there is some slight bowing in the no music background samples and some very slight snaking in the music background samples, points hang very closely to the reference line.
We would notice some outliers at the end of the distribution. They would stand out to us because they would deviate largely from the reference line. None of the Q-Q plot indicate any outliers but we’ll verify that with the z-scores.
Figure 7 is a screenshot of some of the values in our newly created z-score variables. We’ll want to scan these variables for values that are either less than -3 or greater than +3.
Figure 7
Newly Created Z-Scores
A quick sort (right click on “Ztest1” in data view, then click “Sort Ascending”), reveals that we have no z-scores below -3 and no z-scores above +3. Therefore, we have no concerns for outliers in these samples.
We’ll let SPSS handle this assumption check with Mauchly’s Test of Sphericity in our GLM output.
We are ready to set up the GLM procedure. Navigate to the “Analyze” menu in the menu bar. Hover over “General Linear Model,” then select “Repeated Measures” from the menu. The “Repeated Measures Define Factor(s)” window should appear.
The first screen of the repeated measures GLM is used to define factors. We’ll only need to put in the within-subjects factor and the DV associated with that factor.
Type “Test” into the “Within-Subjects Factor Name” box and 3 for the “Number of Levels.” Figure 8 demonstrates the set up for adding the “test” factor.
Figure 8
Adding a Factor
Click “Add” to add the “test” factor to the model.
Add the dependent variable (“SAT”) to the “Measure Name” box and click “Add”.
Figure 9 shows the completed “Repeated Measures Define Factor(S)” window.
Figure 9
Completed Define Factors Window
Click “Define” in the bottom of the window to move to the next window.
Move each of the three “test” variables to the “Within-Subjects Variables” box.
Move the “musci education” variable to the "Between-Subjects Factor(s) box.
See Figure 10 for the completed alignement of both within- and between-subjects variables.
Figure 10
Assigned Variables in GLM
Click the “Plots” button on the right side of the “Repeated Measures” window.
We’ll need to ask for both main effects plots and an interaciton plots.
To create a main effect plot, move an IV from the “Factors” box on the left to the “Horizontal Axis” box on the right. Figure 11 shows the set up for the main effect of “MusicEd”
Figure 11
Setting up Main Effect Profile Plot
Click the “Add” button to ensure that the plot gets made.
Do the same procedure for the main effect of “Test”
To create an interaction plot, we’ll want to include both of our IVs. Let’s move “Test” to the “Horizontal Axis” box and “MusicEd” to the “Separate Lines” box (see Figure 12). Click the “Add” button to complete the process.
Figure 12
Setting up Interaction Effect Profile Plot
Let’s ensure that our charts are helpful by choosing the “line chart” option for “Chart Type” and to “include error bars” under the “Error Bars” section.
Click “Continue” to return to the “Repeated Measures” window.
To compare the levels of any between-subjects variables that have significant main effects, we’ll want to perform the Tukey. Two important notes. First, this step is only necessary IF we have a between-subjects factor with more than 2 levels and the main effect is statistically significant. Second, we may be focusing on the interaction effect, rendering this step moot.
Given those two notes, we do no know beforehand if the main effects for the between-subjects variable will be statistically significant or if the interaction effect will be significant. As such, we set up this test in case we need to reference it.
Click on the “Post Hoc” button.
In the “Post Hoc Multiple Comparisons” window, move “MusicEd” over to the “Post Hoc Tests for:” box then check the box next to “Tukey” (see Figure 13). Click “Continue” to return to the main window.
Figure 13
Tukey HSD in Post Hoc Analyses
We’ll also want some post hoc comparisons for the within-subjects variables (and associated interaction effects). Let’s ask SPSS for marginal means with the Bonferroni-adjusted confidence intervals.
Click on the “EM Means” button to open the “Repeated Measures: Estimated Marginal Means” window.
Drag all but the “(OVERALL)” factor from the “Factor(s) and Factor Interactions” box to the “Display Means for” box. Click the “Compare main effects” box and choose “Bonferroni” from the “Confidence interval adjustment” boxt (see Figure 14).
Figure 14
Setting up Marginal Means Factors
If your brain is throwing a red flag right now, it is because we had used the Tukey HSD post hoc for the between-subjects factors and the Bonferroni adjustment for the within-subjects variables since the t-tests. Now we are asking SPSS for a Bonferroni-adjusted comparison for a between-subjects factor? Why?
Although we are getting to post hoc comparisons, we will reference the Tukey HSD for “MusicEd” (if needed, of course), because that is the more appropriate test. We are only getting the Bonferroni-adjusted comparison because we want the means and confidence intervals that are produced for “MusicEd” in this step.
Click the “Continue” button to return to the main “Repeated Measures” window.
Before we wrap up this GLM, we need to ask SPSS for Levene’s test for the homogeneity of variance. SPSS includes that in the output for the between-subjects designs (i.e., the univariate GLM) but does not do so for the repeated measures procedure. This is because the between-subjects factor(s) are an optional component in the this procedure.
Click on the “Options” button in the main “Repeated Measures” window. Check the box next to “Homogeneity tests” (see Figure 15).
Figure 15
Homogeneity Test Option
Click “Continue” to return to the main “Repeated Measures” window.
We now have all of the options set for our repeated measures GLM. Navigate to the syntax editor to select and run the GLM syntax (see Figure 16 for complete syntax).
Figure 16
Syntax for Repeated Measures GLM
We’ll get to the interpretation of our model right after we address the “Warnings” at the begining of our output for the GLM. It states:
Post hoc tests are not performed for Music Education Background because there are fewer than three groups.
As we noted above, we only need Tukey Post hoc comparisons if our between-subjects factor has more than 2 levels. This is not the case for “Music Education Background”, which as “yes” or “no” as levels.
We do not need the post hoc comparison because the ANOVA and means tables can tell us all that we need to know. The ANOVA table will tell us if the two groups are reliably different and the means table will tell us which had higher SAT scores.
Scroll to the “Mauchly’s Test of Sphericity” table. We see that there is no violation of sphericity because the “Sig.” is greater than .05 (see Figure 17).
Figure 17
Mauchly’s Test of Sphericity
We can interpret the top line for each effect in the within-subjects effects table.
Figure 18 contains the “test of within-subjects effects” table. The table highlights the “Sphericity Assumed” rows and the “sig.” values.
Figure 18
Test of Within-Subjects Effects
With the sig. values being less .05, there is a significant main effect of “test” and a significant interaction effect of “test by music education background.”
We are going to save the write-up for these effects until after we review the between-subjects results.
We’ll want to esure that we haven’t violated the assumption of equal variance before we interpret our between-subjects effect.
Figure 19 contains an annotated version of “Levene’s Test of Equality of Error Variances” in which the relevant “Sig.” values are highlighted.
Figure 19
Levene’s Test of Equality of Variances
Luckily for us, our sig. values are > .05. This means that we have not violated our assumption of equal variance for any of our samples.
Figure 20 is an annotated verson of the “test of between-subjects effects” table. It highlights the row and sig. value for “MusicEd”. WIth a sig. value < .001, we can also claim a significant main effect of Music Education Background.
Figure 20
Test of Between-Subjects Effects
We need to do a little work to combine these effects into one ANOVA table. Although we typically either export or copy-and-paste the table from the SPSS output, I am going to recommend that you create a table from scratch in your favorite word processing software (e.g., Microsoft Word, Google Docs, Apple Pages, etc.).
The ANOVA table contains the following column headers “Source, Sum of Squares, df, Mean Square, F, and p”. That is a total of 6 columns.
Next, we’ll need to how many rows our combined table will need. In general, we need one row for each effect, and a row for error. When you have within-subjects factors, you’ll have separate error terms for each within-subjects factor. In our example, we’ll need a total of 5 rows (i.e., Music Ed, Between-Subjects Error, Test, Test*MusicEd, Error within Test).
Create a new table with 6 (5 for effects and errors, 1 for the column headers) rows and 6 columns. Figure 21 shows how to do this in Microsoft Word using the “Insert” menu.
Figure 21
Creating a 6 x 6 table in Microsoft Word
With the the structure of the table set, we can now fill in the cells.
Type in the column headers in the top row (i.e. “Source, Sum of Squares, df, Mean Square, F, and p”).
Type the effect and error term labes in the first row, below the “Source” header. That is, type “Test, Test * MusicEd, Error(Test), MusicEd, Error” in the first cells of the rows.
Figure 22 contains the basic table with the column and row headers.
Figure 22
Column and Row Headers
Now you can copy-and-paste or type in the values for each cell.
Table 2 is the APA-stylized version of our combined table
Table 2
Combined ANOVA table for Between- and Within-Subjects Effects
| Source | Sum of Squares | df | Mean Square | F | p |
|---|---|---|---|---|---|
| Test | 129341.217 | 2 | 64670.608 | 18942.573 | <.001 |
| Test * MusicEd | 110509.317 | 2 | 55254.658 | 16184.561 | <.001 |
| Error (Test) | 259.467 | 76 | 3.414 | – | – |
| MusicEd | 552706.133 | 1 | 552706.133 | 249.365 | .000 |
| Error | 84225.333 | 38 | 2216.456 | – | – |
Table 2 indicates that we have a significant main effect for test attempt, a significant main effect for music education background, and a significant test * music education interaction effect as all effects have p-values less than .05.
Remember, we will want to focus our post hoc analyses on the interaction effect because the interaction of the two IVs gives a more complete story of how the IVs simultaneously impact the DV.
To better direct our approach of performing post hoc analyses, I invite you to examine the interaction plot we produced in the GLM.
Figure 20 is the interaction plot of test attempt and music education background on SAT score.
Figure 20
Interacton Plot
This looks to be a fairly straight-forward ordinal interaction. Whereas SAT score does not increase across test attempt for those with no music education background, those with a music education background have an increase in scores from one test to the next.
We have two options for breaking down this interaction. The first is the simple effects test. This involves separating the samples and testing for main effects of one IV within the levels of the other IV. With the 95% confidence interval error bars, our figure indicates that an ANOVA for the effect of test in the no music education background sample will yield null results but will yield a significant effect for the music education background sample.
We would then need to follow up on that main effect of test attempt for those with a music education background using Bonferroni-corrected pairwise comparisons. Again, our figure indicates that we would see a significant difference between tests 1 and 2, between tests 2 and 3, and thus between 1 and 3.
I suggest that we cut to the chase and make some pointed comparisons using the means and confidence intervals produced through the “Estimated Marginal Means” portion of the GLM.
Figure 21 contains the means and confidence intervals of SAT scores produced for the interaction of music education background and test attempt.
Figure 21
Means and Confidence Intervals
As the figure suggested, the 95% CI for SAT scores have a lot of overlap for the no music education background samples. Furthermore, there is no overlap in 95% CI for SAT scores in the music education bakground samples.
The last detail to note is the lowest mean SAT score for the music education background samples (i.e., first test) is reliably higher than the all the no music education background samples.
We now have all that we need to complete our post hoc write up.
We’ve done a lot of the work along the way this time so we’ll just need to piece things together.
Let’s start with the interaction plot. From SPSS, you’ll want to:
Here is a reproduction of Figure 20 for which I have complete the above steps.
Figure 20
Interacton Plot
Note.Error bars represent 95% CI.
Now let’s present the results of the mixed factorial ANOVA by combining the statement of the test performed, the interpretation of the results, and the summarizing statistical information.
Here is our combined ANOVA table for convenience.
Table 2
Combined ANOVA table for Between- and Within-Subjects Effects
| Source | Sum of Squares | df | Mean Square | F | p |
|---|---|---|---|---|---|
| Test | 129341.217 | 2 | 64670.608 | 18942.573 | <.001 |
| Test * MusicEd | 110509.317 | 2 | 55254.658 | 16184.561 | <.001 |
| Error (Test) | 259.467 | 76 | 3.414 | – | – |
| MusicEd | 552706.133 | 1 | 552706.133 | 249.365 | .000 |
| Error | 84225.333 | 38 | 2216.456 | – | – |
“A mixed factorial ANOVA was implemented through the repeated measures general linear model. The results indicated significant effects for both test attempt (F[2,76] = 18942.573, p < .001) and music education background (F[1,38] = 249.365, p < .001) on SAT scores. There was also a significant interaction effect of the two IVs on SAT scores (F[2,76] = 16184.561, p < .001).”
As we explain the interaction effect, we should focus on patterns and deviation of patterns. We can also point out the key comparisons that help establish the principle findings within the interaction.
Rather than refering to a table, I am going to incorportate the confidence intervals directly into my write up. Notably, I am going to offer ranges for the lower bound and upper bound of the confidence intervals.
“Bonferroni-corrected 95% confidence intervals reveal that SAT scores remained very similar across test attempts for those with no music education background (95% CI lower bounds [1182.879, 1188.750]; 95% CI upper bounds [1207.521,1213.550]). A pattern of improvement in each test attempt, in contrast, was found for those with a music education background. Individuals scored the lowest in the first attempt (M = 1254.650, 95% CI [1242.329, 1266.971]), scored intermediately in the second attempt (M = 1340.050, 95% CI [1327.802, 1252.298]), and highest in the third attempt (M = 1409.100, 95% CI [1396.700, 1421.500]). Those with a music education background scored higher in the first attempt than those whithout a music education background (M = 1195.200, 95% CI [1182.879, 1207.521]).”
In this lesson, we’ve: