· Q 1. A pathologist and I are independently recording our estimates of the disease process for 196 patients by means of an ICD 10 classification. I would like to obtain a measure of the level of agreement between our estimates which did not occur by chance. Which measure do I need and can I test the significance of the value obtained? Also, can I perform the necessary calculations using SPSS and if so, how should I lay out my data?

A. If your response data have no natural ordering and there are no repeated measures for each respondent, the statistic that you need is Cohen’s simple kappa, details of which are available on pp. 140 – 143 of Chapter 5 of the Statistical Analysis Quick Reference Guidebook. You can find these pages easily by typing ’28’ in the page selection option using the task bar at the top of your screen. Note that worked example 5.6 in this resource assumes that your data are in summary form. However, this need not be the case. If you have two separate columns with one row per individual, just follow the instructions but miss out the requirement to weight cases.  You are not asked to provide the counts because when you provide the data on a per individual basis, SPSS provides these in the output. Therefore, you don’t need to weight the cases.

Confidence interval for Cohen’s simple kappa
Please take careful note of the method described in the above reference for calculating the confidence interval for your kappa value. Typically, you will be calculating a sample kappa and using this to make an inference about the level of inter-rater agreement within a general population. The confidence interval will give the reader a better idea of how good your sample kappa value is as an estimate of the real (or, population) kappa value. The above reference provides a formula for obtaining the 95% confidence interval for your kappa value. If you are in doubt as to how to find the standard error (SE) from your SPSS output to perform this calculation,  just scroll up to Table 5.15, where you will find a column header ‘Asymp. Std. Error’. The value of 0.088 under this header was the one used for the SE in the worked example to obtain the 95% confidence interval. You need to extract the corresponding value from your output.

Notice that Table 5.15 also contains the header ‘Approx. Sig.’ . The value under this header is the p-value relating to a test of the hypothesis that your kappa value is equal to zero (equivalently that there is no agreement). If the p-value exceeds 0.05, there is a lack of statistical evidence at the 5% significance level to reject this null hypothesis. Usually, you will be hoping to reject the null hypothesis, as was possible in the worked example. Incidentally, if you are bamboozled by the value 0.000 for the p-value under ‘Approx. Sig.’ in the worked example, don’t stay that way! Simply refer to the the solution to Q. 3 on the StatsforMedics WordPress page STATISTICAL SIGNIFICANCE, STATISTICAL POWER AND SOME FACTS ABOUT P-VALUES and take things from there!

On a practical level, first prepare your data in Excel and then use the File menu within SPSS to open up your Excel spreadsheet from within SPSS.

Thinking outside of the box

Cohen’s kappa can be used in a variety of scenarios which may not immediately seem obvious. A couple include assessing agreement between:

a) clinician views in specific contexts concerning the right procedure to follow and what they consider to be expected of them

and

b) patient self-reporting and corresponding content in medical records. The example provided at https://www.karger.com/Article/Fulltext/337798 under the header ‘Cohen’s K’ covers this case. Table 3 in the same reference appears to provide a handy guide for judging the strength of the agreement. However, the ranges are not expressed very accurately. On scrolling down to Q. 3 and the corresponding solution on the current StatsforMedics page, you will find advice on a better reference for obtaining ranges of values for judging the strength of agreement of individual values of Cohen’s kappa.

Likert scale data

N.B. When dealing with Likert scale data from a questionnaire (see Questionnaire Design index) or ordinal data, the distance between levels of agreement matters as there is a natural ordering in the data. In particular, it makes sense to take into consideration how far apart the respondents are in their assessments rather than simply ask whether or not they agree.

Alternatively, your data may be continuous and you may wish to take differences in scores into consideration (as with assessment marks expressed as % s).

In each of these two cases, the intra-class correlation coefficient (ICC) for absolute agreement would be a more appropriate choice.  ICCs may be used to measure consistency between raters (inter-rater reliability) and across repeated ratings for a single rater (intra-rater reliability).  For the  case where there are two or more different raters, the appropriate choice is the ICC for a two-way analysis and absolute agreement. From the point of view of your results, it will make no difference whether you opt for a two-way mixed or a two-way random model for this case when using the drop-down menu for the model choice within SPSS. However, conceptually speaking, the choice between the two models depends on whether you have good reason to suppose that the raters in your study have been randomly selected from a population of raters and therefore that your results are generalizable across that population of raters (two-way random model: ratee and rater effects both random). If you consider it more appropriate to interpret your findings specifically in terms of the raters involved in your study, your rater effect is assumed to be fixed, whilst the ratee effect is assumed to be random (two-way mixed model).

Please note that when estimating inter-rater agreement for Likert scale data from a questionnaire or ordinal data, the weighted kappa statistic with quadratic weighting is equivalent to that of the ICC for a two-way model involving absolute agreement, so if you are looking for a suitable kappa statistic you can use the above ICC and the corresponding p-value and confidence interval instead.

To learn more about ICCs and how to calculate them, please refer to the sub-header Intraclass correlation coefficients in the solution to Q. 7, below.

If your data represent the ranking of an entire group from the best to the worst, Kendall’s tau (two raters) or Kendall’s Coefficient of Concordance (more than two raters) are more appropriate choices. Kendall’s tau and Kendall’s Coefficient of Concordance may be used to measure agreement between raters (inter-rater agreement) and across repeated raters for a single rater (intra-rater agreement).

. Q 2. I would like to asses the reliability of my performance in using a frailty tool for diagnosing frailty in patients over 65 years of age. The tool ultimately generates a dichotomous outcome and therefore, I would like to use Cohen’s simple kappa. For practical reasons, I am interested in carrying out the reliability check for a random sample of patients. However, I am not sure how many would be enough. Is there an appropriate sample size calculation which I could hope to perform to estimate the required minimum sample size for my reliability check? 

A. Firstly, note that on the StatsforMedics WordPress page WORKING WITH SUBSETS OF THE ORIGINAL DATASET – FILTERING DATA, you will find resources on how to select a random sample of data from within an existing SPSS spreadsheet. The question of minimum sample size is a tricky one, as the task of obtaining the required ingredients for any suitable sample size calculation is quite exacting. An example of a relatively transparent sample size calculation for Cohens’s kappa is provided under Sample Size Determination. Even here, two of the required parameters – the overall probability of two persons agreeing and the probability of two persons agreeing purely due to chance – rely on a considerable degree of insight into what occurred in similar scenarios previously. If you do not have such information at your finger-tips and it is not forthcoming from a literature search, then you shouldn’t attempt the impossible; rather, you should recognize and report your work as novel and foresee the opportunity to use the findings to inform future sample size calculations.  For your current study, you would therefore need to maximize your sample size within the boundaries of what is possible, given the time constraints of the project.

· Q 3. I am surprised that for such a low value of the kappa statistic, I have found a significant agreement between my own choices of ICD and those of the pathologist. Can I obtain a handle on kappa statistic cut-off points for varying levels of strengths of agreement?

A. Some useful information can be obtained by studying the topic Assessing agreement (Topic 39 of 4th edition) of the book Medical Statistics at a Glance.

If you are registered with the University of Edinburgh, you can consult the electronic version of this book via the University’s library discovery system, DiscoverEd.

Here are some reference details:

• Title: Medical statistics at a glance
• Author: Aviva Petrie
• Caroline Sabin
• Publisher: Hoboken : Wiley
• Publication Date: 2020
• Edition:  4th edition

Q 4. The value of my kappa statistic is negative. Does this represent an error?

A. Not at all. Please have a look at The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements.

· Q 5. I would like to use Cohen’s kappa to measure agreement between standard scales used to determine the presence or absence neuropathic pain. Can I calculate the value of Cohen’s kappa for my data without using SPSS?

A. Yes, you can. Please make good use of the content under the header Kappa on the page at the link  Statistical calculators. The relevant calculator allows you, not only to calculate the value of Cohen’s kappa for your data but also, to understand a little more about the underlying theory and interpretation of your data. Additionally, the calculator provides the standard error for Cohen’s kappa, from which you may in turn derive a confidence interval using the procedure available in the reference for the solution to Q. 1., above.

Q 6. What if I wish to calculate individual kappa statistics – one for each possible response?

A. In that case, you should really consider using Fleiss’s kappa statistics and for this you will need to use Minitab rather than SPSS, if you wish to rely on drop down menus rather than use of syntax. There are resources available which include instructions on how to calculate values for Fleiss’s overall Kappa statistic and Fleiss’s individual Kappa statistics (one for each category) using Minitab. These resources cover how to follow the appropriate commands in Minitab, how to lay out your data in your spreadhseet and how to handle the Minitab dialogue box in terms of which columns go where. To find the resources, you should go to the Mintab menu Stat and select the sequence ‘Quality Tools –>Attribute Agreement Analysis’. If you click on the button Help within the resultant dialogue box, you will see that you are referred to a web page with a menu of the left with recommendations on how to lay out your data and with a worked example performed using the data in spreadsheet, TextilePrintQuality.MTW.

·Q 7. Can I use Cohen’s Kappa for the case where there are more than 2 observers?

A. No. Here, you can use Fleiss’s overall kappa statistic or (if your data represent the ranking of a group from best to worst), both Fleiss’s overall kappa statistic and Kendall’s coefficient of concordance. Please refer to the instructions in the solution to Q. 6. above. Using the recommended resources  there you will have the opportunity to explore calculating:

a) Fleiss’s Kappa, including how to compare the ratings of different raters with those of a Gold Standard.

and

b) Kendall’s Coefficient of Concordance (also known as Kendall’s correlation coefficient) using Minitab.

These instructions  include how to follow the appropriate commands on Minitab, how to lay out your data in your spreadhseet and how to handle the Minitab dialogue box in terms of which columns go where.

Note that there is likely to be more information and data in these examples than apply to your own case. For example, you possibly don’t have a gold standard and you may not have ranked data or repeated measures for each rater. This is for you to check. You can skip those steps that are not relevant to yourself and adapt the layout of your data so that for example, you are not recording the same same more than once for the same rater if they did not take repeated measurements. The  spreadsheet TextilePrintQuality.MTW is designed for calculating Fleiss’s kappa for the case where there are multiple raters and repeated measures for any one rater. You may wish to simply record the sample number only once rather than twice in the column ‘sample’ for your spreadsheet.

Multiple ratings (repeated measures) for multiple raters

For cases where there are, not only multiple raters but also, multiple ratings for each rater, you should also consider the worked example within the resource

together with the accompanying spreadhseet manufacturing.MTW, which offers you a useful illustration of how to lay out your data. Please note that the column ‘standard/attribute’ is optional. This column is for storing data against which all scorer ratings can be compared (e.g. if one were comparing ratings from registrars with those of a consultant – the gold standard). If you don’t have one, just proceed without it, missing out the unnecessary step in the Minitab procedure and extra detail on interpreting the output for this step.  The procedure is designed to be flexible! As such, it also covers the case of more than two raters.

If your data are on a Likert scale or are ordinal or  continuous (such as in the case of % marks in an assignment) and you wish to assign weights to assessor responses according as to how far apart they are from one another (see the last paragraph of the solution to Q. 1, above), you may wish to consider using an intra-class correlation coefficient (ICC). Please read on!

Intraclass correlation coefficients

Instructions on how to calculate ICCs using SPSS together with some accompanying exposition on ICCs are provided in the PowerPoint presentation

Intraclass correlation coefficients: reliability and more

(Thanks are due to Dr Alexa Romberg,  Indiana Univeristy Bloomington for very kindly permitting use of this, her presentation, here.)

N.B. When reading the advice on picking up the ICC from a table of SPSS output, also look out for the 95% CI. This, together with the sample size, ought to be included whenever you provide an ICC. Therefore, for the example in the above resource, one would write “0.290 (95%CI: (0.019, 0.7611), n = 10)”  when presenting the sample estimate, 0.290, for the ICC.  Notice how wide the CI is and correspondingly, how small the sample size is. Such observations should facilitate discussion in terms of critically reviewing current work and suggesting how a more accurate ICC could be obtained in the future.  This is worth noting in the writing up of your own work.

The article Choosing an intraclass correlation coefficient is referenced in the above resource and should serve as a  helpful guide.

For more comprehensive information, please refer to the guide,

Enhancement of Reliability Analysis: Application of Intraclass Correlations with SPSS/Windows v.8.

Do not worry that the above resource refers to Version 8.0 of SPSS. The instructions are still up-to-date.

The resource Intraclass correlation coefficients provides a concise summary of how the terminology in the literature naming different ICCs maps to the corresponding  procedures available for calculating ICCs in SPSS.  As with the previous resource, the relevant steps for calculating ICCs in SPSS are provided in detail.

Please also refer to Q. 8, below.

· Q 8.  I have been using the intraclass correlation coefficient (ICC) to measure consistency between medical student ratings and clinician ratings and have opted for a two-way mixed model.  However, I have a further choice to make.  In particular, the table that is produced in SPSS offers output pertaining to the ICC for both single measures and average measures. I have been working on the assumption that I should use the single measures value but I wondered if you could confirm if this is correct?

A. The best way to approach any such query is to ask yourself what might happen in clinical practice. Would we normally rely on the judgement of an individual clinician or would we take an average over the judgements of a team of clinicians? Similarly, if you were given the responsibility of making judgements, would you make multiple judgements yourself and then average them or in reality just rely on one of your own judgements per case?

This ought to help you see more clearly what makes best sense for your research.

· Q 9. Where can I view the formulae for Fleiss’s overall and individual kappa statistics?

A. Have a look under calculations for Fleiss’s kappa.

Statistical indices for measuring agreement and consistency between groups of categorical and measurement data by Margaret MacDougall is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.