Statistical Analysis for More than Two Levels - Revision history

Jaehyun at 13:32, 5 May 2021

2021-05-05T13:32:43Z

Jaehyun at 12:57, 9 March 2020

2020-03-09T12:57:25Z

Harald at 09:15, 11 April 2016

2016-04-11T09:15:44Z

Todor at 16:10, 7 March 2016

2016-03-07T16:10:50Z

Todor at 13:54, 7 March 2016

2016-03-07T13:54:53Z

Todor: Created page with "It is not straight-forward to run permutation (M)ANOVAs. This would of course be the best way to treat data with more than two levels per variable. We are working on it with a..."

2016-03-07T13:50:34Z

Created page with "It is not straight-forward to run permutation (M)ANOVAs. This would of course be the best way to treat data with more than two levels per variable. We are working on it with a..."

New page

It is not straight-forward to run permutation (M)ANOVAs. This would of course be the best way to treat data with more than two levels per variable. We are working on it with a statistician, but it will take a while for us to get there. As a valid alternative it is possible to work with differences. Consider this example for a 2x2 ANOVA: You have two groups, patients and controls. Each have two measurements, time 1 and time 2. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group X time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply:

The same principle holds true if one has more than two levels per factor. Let us assume you have 3 factor levels. The difficulty here is that the variance needs to be comparable in all factor levels. If it is not, the sphericity assumption is violated and the results are not valid (one might achieve significant results, although in truth they are not, and more rarely the other way round). So, in order to use the difference approach for more than two factor levels, you would need to compare the variance between the levels. It is not so easy to do this in time-series EEG data, as it is not clear, which time-window and sensor group to choose. So you would have to first calculate the differences (e.g. level 1 minus level 2 and level 1 minus level 3), compare the differences with a permutation test, and then use the cluster results for the sphericity test (you can run Levene’s test for homogeneous variances). The same would need to be repeated for the differences level 2 minus level one and level 2 minus level 3.

@@ Line 2: / Line 2: @@
 |title = Module information
 |module = BESA Statistics
-|version = 1.0 or higher
+|version = BESA Statistics 1.0 or higher
 }}

@@ Line 5: / Line 5: @@
 }}
-The easiest way to treat data with more than two levels per variable is to work with differences. Consider this example for a 2&times;2 ANOVA: You have two groups, ''"Patients"'' and ''"Controls"''. Each have two measurements, ''"Time 1"'' and ''"Time 2"''. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group &times; time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply: [[https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method Holm–Bonferroni method]]
+The easiest way to treat data with more than two levels per variable is to work with differences.
-The same principle holds true if one has more than two levels per factor. Let us assume you have 3 factor levels. The difficulty here is that the variance needs to be comparable in all factor levels. If it is not, the sphericity assumption is violated and the results are not valid (one might achieve significant results, although in truth they are not, and more rarely the other way round). So, in order to use the difference approach for more than two factor levels, you would need to compare the variance between the levels. It is not so easy to do this in time-series EEG data, as it is not clear, which time-window and sensor group to choose. So you would have to first calculate the differences (e.g. level 1 minus level 2 and level 1 minus level 3), compare the differences with a permutation test, and then use the cluster results for the sphericity test (you can run Levene’s test for homogeneous variances). The same would need to be repeated for the differences level 2 minus level one and level 2 minus level 3.
+Consider this example for a 2&times;2 ANOVA: You have two groups, ''"Patients"'' and ''"Controls"''. Each have two measurements, ''"Time 1"'' and ''"Time 2"''. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition 2 per group and compare the differences. If this comparison becomes significant, it is identical to the interaction group &times; time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time 2 and the other group does not, etc. This could be done by planned comparisons in any statistical program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply: [https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method Holm–Bonferroni method]
 For additional information visit the FAQ page of FieldTrip: [http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_interaction_effect_using_cluster-based_permutation_tests How to test an interaction effect using cluster-based permutation tests?]
 [[Category:Statistics]]

← Older revision		Revision as of 09:15, 11 April 2016
Line 1:		Line 1:
		+	{{BESAInfobox
		+	\|title = Module information
		+	\|module = BESA Statistics
		+	\|version = 1.0 or higher
		+	}}
		+
	The easiest way to treat data with more than two levels per variable is to work with differences. Consider this example for a 2×2 ANOVA: You have two groups, ''"Patients"'' and ''"Controls"''. Each have two measurements, ''"Time 1"'' and ''"Time 2"''. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group × time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply: [[https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method Holm–Bonferroni method]]		The easiest way to treat data with more than two levels per variable is to work with differences. Consider this example for a 2×2 ANOVA: You have two groups, ''"Patients"'' and ''"Controls"''. Each have two measurements, ''"Time 1"'' and ''"Time 2"''. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group × time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply: [[https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method Holm–Bonferroni method]]

Line 4:		Line 10:

	For additional information visit the FAQ page of FieldTrip: [http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_interaction_effect_using_cluster-based_permutation_tests How to test an interaction effect using cluster-based permutation tests?]		For additional information visit the FAQ page of FieldTrip: [http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_interaction_effect_using_cluster-based_permutation_tests How to test an interaction effect using cluster-based permutation tests?]
		+
		+	[[Category:Statistics]]

← Older revision		Revision as of 16:10, 7 March 2016
Line 1:		Line 1:
−	The easiest way to treat data with more than two levels per variable is to work with differences. Consider this example for a 2×2 ANOVA: You have two groups, ''"Patients"'' and ''"Controls"''. Each have two measurements, ''"Time 1"'' and ''"Time 2"''. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group × time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply:	+	The easiest way to treat data with more than two levels per variable is to work with differences. Consider this example for a 2×2 ANOVA: You have two groups, ''"Patients"'' and ''"Controls"''. Each have two measurements, ''"Time 1"'' and ''"Time 2"''. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group × time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply: [[https://en.wikipedia.org/wiki/Holm%E2%80%93Bonferroni_method Holm–Bonferroni method]]

	The same principle holds true if one has more than two levels per factor. Let us assume you have 3 factor levels. The difficulty here is that the variance needs to be comparable in all factor levels. If it is not, the sphericity assumption is violated and the results are not valid (one might achieve significant results, although in truth they are not, and more rarely the other way round). So, in order to use the difference approach for more than two factor levels, you would need to compare the variance between the levels. It is not so easy to do this in time-series EEG data, as it is not clear, which time-window and sensor group to choose. So you would have to first calculate the differences (e.g. level 1 minus level 2 and level 1 minus level 3), compare the differences with a permutation test, and then use the cluster results for the sphericity test (you can run Levene’s test for homogeneous variances). The same would need to be repeated for the differences level 2 minus level one and level 2 minus level 3.		The same principle holds true if one has more than two levels per factor. Let us assume you have 3 factor levels. The difficulty here is that the variance needs to be comparable in all factor levels. If it is not, the sphericity assumption is violated and the results are not valid (one might achieve significant results, although in truth they are not, and more rarely the other way round). So, in order to use the difference approach for more than two factor levels, you would need to compare the variance between the levels. It is not so easy to do this in time-series EEG data, as it is not clear, which time-window and sensor group to choose. So you would have to first calculate the differences (e.g. level 1 minus level 2 and level 1 minus level 3), compare the differences with a permutation test, and then use the cluster results for the sphericity test (you can run Levene’s test for homogeneous variances). The same would need to be repeated for the differences level 2 minus level one and level 2 minus level 3.
		+
		+	For additional information visit the FAQ page of FieldTrip: [http://www.fieldtriptoolbox.org/faq/how_can_i_test_an_interaction_effect_using_cluster-based_permutation_tests How to test an interaction effect using cluster-based permutation tests?]

← Older revision		Revision as of 13:54, 7 March 2016
Line 1:		Line 1:
−	~~It is not straight-forward to run permutation (M)ANOVAs. This would of course be the best~~ way to treat data with more than two levels per variable~~. We are working on it with a statistician, but it will take a while for us to get there. As a valid alternative it~~ is ~~possible~~ to work with differences. Consider this example for a ~~2x2~~ ANOVA: You have two groups, ~~patients~~ and ~~controls~~. Each have two measurements, ~~time~~ 1 and ~~time~~ 2. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group X time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply:	+	The easiest way to treat data with more than two levels per variable is to work with differences. Consider this example for a 2×2 ANOVA: You have two groups, ''"Patients"'' and ''"Controls"''. Each have two measurements, ''"Time 1"'' and ''"Time 2"''. The question is, whether patients and controls change differently from time 1 to time 2. In this case one would subtract condition 1 from condition two per group and compare the differences. If this comparison becomes significant it is identical to the interaction group × time. Now it is necessary to investigate the source of the interaction, as it is not clear from the differing differences (sorry!), if the groups differ in time 1 and not time 2, or if one group changes from time 1 to time two and the other group does not, etc. This could be done by planned comparisons in any stats program (usually simple paired or unpaired t-tests), i.e. making only meaningful comparisons (e.g. controls time 1 vs. patients time 1). To do so, you should use the data (i.e. the mean) of the clusters that became significant in the difference comparison. If you want to be 100% strict, you would need to adjust the p-value of the planned comparisons, as this again means running multiple tests. A good way to do this is the Bonferroni-Holm correction, which is conservative, but not as conservative as the original Bonferroni correction. It is very easy to apply:

	The same principle holds true if one has more than two levels per factor. Let us assume you have 3 factor levels. The difficulty here is that the variance needs to be comparable in all factor levels. If it is not, the sphericity assumption is violated and the results are not valid (one might achieve significant results, although in truth they are not, and more rarely the other way round). So, in order to use the difference approach for more than two factor levels, you would need to compare the variance between the levels. It is not so easy to do this in time-series EEG data, as it is not clear, which time-window and sensor group to choose. So you would have to first calculate the differences (e.g. level 1 minus level 2 and level 1 minus level 3), compare the differences with a permutation test, and then use the cluster results for the sphericity test (you can run Levene’s test for homogeneous variances). The same would need to be repeated for the differences level 2 minus level one and level 2 minus level 3.		The same principle holds true if one has more than two levels per factor. Let us assume you have 3 factor levels. The difficulty here is that the variance needs to be comparable in all factor levels. If it is not, the sphericity assumption is violated and the results are not valid (one might achieve significant results, although in truth they are not, and more rarely the other way round). So, in order to use the difference approach for more than two factor levels, you would need to compare the variance between the levels. It is not so easy to do this in time-series EEG data, as it is not clear, which time-window and sensor group to choose. So you would have to first calculate the differences (e.g. level 1 minus level 2 and level 1 minus level 3), compare the differences with a permutation test, and then use the cluster results for the sphericity test (you can run Levene’s test for homogeneous variances). The same would need to be repeated for the differences level 2 minus level one and level 2 minus level 3.