Statistical Significance Tests
Using statistical tests, it is possible to make a statement about the significance of a set of measurements by calculating a test statistic. If it is unlikely to obtain a test statistic at least as extreme as the observed value, then the result is significant. For example, at a significance level of 5%, the probability of a false positive test result would be bounded by roughly 5%.
Parametric vs non-parametric tests
There is a multitude of tests for determining statistical significance. These tests can be differentiated into two categories: parametric and non-parametric tests. While parametric tests make assumptions on the distribution of the data, non-parametric tests do not rely on such assumptions. For example, the parametric t-test compares the means of two groups because it assumes that the data have a normal distribution.
The non-parametric Wilcoxon rank sum test (Mann-Whitney U test), on the other hand, considers the medians of the groups instead. If the assumptions of parametric tests are met, they are generally more capable of detecting an effect than non-parametric tests. If this is not the case, however, non-parametric tests should be preferred.
Choosing an appropriate significance test
To find an appropriate statistical test, the structure of the data should be considered. Before starting an analysis, one should ask the following questions:
- How many dependent/independent variables are there?
- What are the types of the variables?
- Are the measurements in some way associated (i.e. matched)?
What is there besides significance?
Once you have found an appropriate test, you may want to look into topics that go beyond mere significance, such as:
- How can I use effect sizes to describe the extent of an effect?
- How can I use power analysis to identify the likelihood that a test detects an effect if it exists?
- How can I interpret measurements using other quantities such as confidence intervals?
Posts on statistical testing
You can find answers to these questions (and more) in the following posts on statistical testing.
