Statistical Analysis for More than Two Levels

It is not straight-forward to run permutation (M)ANOVAs. This would of course be the best way to treat data with more than two levels per variable. We are working on it with a statistician, but it will take a while for us to get there. As a valid alternative it is possible to work with differences. Consider this example for a 2x2 ANOVA: You have two groups, patients and controls. Each have two measurements, time 1 and time 2. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group X time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply:

The same principle holds true if one has more than two levels per factor. Let us assume you have 3 factor levels. The difficulty here is that the variance needs to be comparable in all factor levels. If it is not, the sphericity assumption is violated and the results are not valid (one might achieve significant results, although in truth they are not, and more rarely the other way round). So, in order to use the difference approach for more than two factor levels, you would need to compare the variance between the levels. It is not so easy to do this in time-series EEG data, as it is not clear, which time-window and sensor group to choose. So you would have to first calculate the differences (e.g. level 1 minus level 2 and level 1 minus level 3), compare the differences with a permutation test, and then use the cluster results for the sphericity test (you can run Levene’s test for homogeneous variances). The same would need to be repeated for the differences level 2 minus level one and level 2 minus level 3.