. Q.1. Where can I find a simple summary on how to distinguish the notion of ‘p-value’ from those of ‘significance level’ and ‘power’?

A. You are recommended to work your way through the Key Learning Tutorial Fundamentals of Hypothesis Testing. This tutorial will help you see by means of a table how these notions are built up in terms of the types of error that can arise from hypothesis testing. Of course, we wish to minimize the probability of these types of error; read on!

Note. It should be implicit from the above resource that a p-value is a probability. As such, a p-value has:

a minimum value of 0         (1.1)

and

ii) a maximum valued of 1.   (1.2)

It should be clear from the content of the above resource that a p-value is a conditional probability, not simply the probability that the null hypothesis is true.  Have a look at the blog How to Correctly Interpret p-values for further insight on the interpretation of  p-values.

Cautionary note

It is extremely important to note that it is implicit from the content in the above resources that p-values are not stand-a-lone entities, but the product of prior construction of carefully posed hypotheses. It is your responsibility as a researcher to formulate these hypotheses. It is not your sole mission to find a p-value but to formulate evidence to refute or not to refute a null hypothesis, and the p-value is only a minor part of that evidence. Confidence intervals matter a great deal as complimentary evidence.

Please enter the term confidence interval into the StatsforMedics search box to obtain specific examples of confidence intervals and how to calculate them using a statistical package.

Confidence intervals are also often part of the output from running hypotheses tests and you will see, for example, from choosing the FAQ page HYPOTHESIS TESTS FOR COMPARING TWO GROUPS OF MEASUREMENT OR ORDINAL DATA that clear instructions are available to help you extract these from your output and indeed make sense of them when writing up a report!

Furthermore, the entertaining video p-values will give you an insight into how repeated sampling to obtain study samples from a parent population can lead to a wide range of p-values, thus leading to the need for confidence intervals!  While it is unlikely to be possible for you to create your own range of p-values, you ought to gain better appreciation from this video of the limitations of a single p-value.  In turn, you will hopefully be redeemed from the temptation to pursue a statistician with the sole intention of getting them to show you how to generate a p-value, worse still, a p-value in Excel. It is better to take on board their considerable efforts to inform you on suitable resources to support your learning and indeed the explanations they have painstakingly provided on the rationale behind their use.

.Q. 2.  I have obtained a narrow confidence interval and I recognize that my result is statistically significant. However, I am unclear about the relevance of this finding to my field. How does clincial significance fit in?

A. Provided you have sufficient data, you may obtain strong  statistical sigificance for a very small effect size (such as a very small difference in means). Clinical signficance relates to the clinical importance of the effect size. For example, we may ask, is the improvement in quality of life likely to be recognizable by the patient? A variety of scenarios can arise whereby statistical and clinical significance may or may not be consistent with one another. To gain helpful guidance on how to respond in these scenarios, you can refer to the blog page Statistical significance vs. clinical significance.

.Q.3. I have generated lots of SPSS output but where do I find the p-value?

A. Please take a careful look at the presentation Identifying the p-value from your SPSS output.

.Q.4. The p-value in my output file reads

a) ‘0.000’

or

b)  ‘1.000’.

Have I made an error?

A. Please note that your computer value has provided the p-value as a figure rounded up to 3 decimal places.  The p-value is unlikely to be precisely 0 or 1.  If you are working with SPSS and are curious to read your value more accurately, double-click on the relevant table within your output file. Then double-click on the number itself and scroll along with your keyboard arrow keys.  The number is usually so small that this more accurate version is provided in scientific notation!

In the case of a), above for your purposes, it would usually suffice to report ‘p < 0.0005’.  As you are normally trying to verify that p is less than or equal to 0.05, you can safely conclude that the effect you have identified by means of your test is highly significant!

In the case of b), given property (1.2) in the solution to Q. 1, above, I think that you will agree there is a lack of evidence at the 5% significance level for the difference or association you are testing for!

 . Q. 5. When testing for statistical significance, what can I say if I find that
p < 0.001?

A. A result like this is well worth noting as it indicates that the probability of obtaining an effect as large as or larger than the one obtained for your sample is less than 1%. In such circumstances it is legitimate and fairly conventional to report that you have obtained a highly significant effect.

What if p = 0.000 or p = 1.000?

Statistical Significance, Statistical Power and Some Facts about p-values by Margaret MacDougall is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.