First, this section discusses whether rejection of the null hypothesis should be an all-or-none proposition. Then, it discusses how to interpret non-significant results; for example, it explains why the null hypothesis should not be accepted or should be accepted with caution. It also describes how a non-significant result can increase confidence that the null hypothesis is false.
Interpreting Significant Results
- Discuss whether rejection of the null hypothesis should be an all-or-none proposition
- State the usefulness of a significance test when it is extremely likely that the null hypothesis of no difference is false even before doing the experiment
When a probability value is below the level, the effect is statistically significant and the null hypothesis is rejected. However, not all statistically significant effects should be treated the same way. For example, you should have less confidence that the null hypothesis is false if than . Thus, rejecting the null hypothesis is not an all-or-none proposition.
If the null hypothesis is rejected, then the alternative to the null hypothesis (called the alternative hypothesis) is accepted. Consider the one-tailed test in the James Bond case study: Mr. Bond was given 16 trials on which he judged whether a martini had been shaken or stirred and the question is whether he is better than chance on this task. The null hypothesis for this one-tailed test is that , where is the probability of being correct on any given trial. If this null hypothesis is rejected, then the alternative hypothesis that is accepted. If is greater than , then Mr. Bond is better than chance on this task.
Now consider the two-tailed test used in the Physicians' Reactions case study. The null hypothesis is:
If this null hypothesis is rejected, then there are two alternatives:
Naturally, the direction of the sample means determines which alternative is adopted. If the sample mean for the obese patients is significantly lower than the sample mean for the average-weight patients, then one should conclude that the population mean for the obese patients is lower than the population mean for the average-weight patients.
There are many situations in which it is very unlikely two conditions will have exactly the same population means. For example, it is practically impossible that aspirin and acetaminophen provide exactly the same degree of pain relief. Therefore, even before an experiment comparing their effectiveness is conducted, the researcher knows that the null hypothesis of exactly no difference is false. However, the researcher does not know which drug offers more relief. If a test of the difference is significant, then the direction of the difference is established. This point is also made in the section on the relationship between confidence intervals and significance tests.
Source: David M. Lane, https://onlinestatbook.com/2/logic_of_hypothesis_testing/significant.html
This work is in the Public Domain.