Note. In considering the FAQs and corresponding solutions below, please be aware of the complementary StatsforMedics WordPress page REPORTING GUIDELINES, which you may also wish to consult in order to access appropriate guidelines for interpreting findings for the particular type of study design you have in mind.
.Q 1. Where can I find some general guidance on essential steps for critical appraisal of published research?
A. The paper Critical appraisal of published research: introductory guidelines should get you started. However, please note that you should also be aware of statistical criteria for appraising published research (see Q. 2 and the corresponding solution).
.Q 2. I don’t have much background in statistics but have been asked to do a critical appraisal of a published research paper from a statistical perspective. Can you advise me on what I should be looking out for in the paper?
A. The following article will get you initiated into the review process:
You should then progress to the site
Critical appraisal of Environmental and Occupational Health Literature
The BMJ paper by T Greenhalgh entitled How to read a paper: papers that summarise other papers (systematic reviews and meta-analyses) may also be of use.
It is also useful to have a checklist which is suitable for the type of study design which you are assessing. In order to evaluate research under any of the banners systematic reviews, randomised controlled trials, qualitative research studies, economic evaluation studies, cohort studies, case-control studies or diagnostic test studies, you may wish to consider using the checklists provided under
Critical Appraisal Skills Programme Checklists
and
With regards to randomized controlled trials in particular, the CONSORT (Consolidated Standards of Reporting Trials) checklist is actually your best choice, not least because it is endorsed by members of the CONSORT Group, who in turn act as journal referees.
Please refer to the The CONSORT Statement Website.
For a relevant paper of interest, see:
You would also be well advised to dip into the StatsforMedics WordPress page
MISUSE OF STATISTICS: SOME STATISTICAL BLUNDERS DISCOVERED WITHIN REFEREED PUBLICATIONS AND SOME STATISTICAL PHENOMENA TO LOOK OUT FOR
This is quite an education in itself!
Last but certainly not least, the reference How to read a medical journal article contains a wealth of information on how to design trials appropriately and well-explained real-case clinical examples of where published articles have led to spurious conclusions through poor research design. This reference is also highly educational.
. Q 3. On reviewing the literature, I discovered a procedure called Principal Components Analysis. How can I get a feel for what this procedure should be used for without delving too much into the fine detail of carrying out the procedure?
A. You might like to have a look through the introductory text provided at Principal Components Analysis, while noting in particular the advice on sample size and that any one component obtained by the analysis may be defined in terms of several of your original variables. In a study involving classification of tumours, if several variables are found to be strongly inter-correlated and thus identified as representing a given component, it may be the case that this is true because these variables (e.g. variables for proteins levels) are associated with a particular type of brain tumour. However, it would be advisable to consider other approaches to analysing the data, such as cluster analysis, the weighted flexible compound covariate method or symbolic discriminant analysis so as to ensure that the components from your analysis are not representative of entities other than tumour classes. It is better to have a body of evidence rather than jumping to conclusions!
. Q 4. Where can I find a presentation that introduces me to the fundamentals of systematic reviews?
A. Here:
Loading...
Reload document 
Open in new tab . Q 5. Where can I find some general advice on how to find studies for systematic reviews?
A. You are advised to refer to
Finding studies for systematic reviews: a resource list for researchers.
. Q 6. Where can I find appropriate training resources to assist me in conducting, editing or reading systematic reviews?
A. You should find useful and relevant information within the following resources:
1) The Centre for Evidence-Based Medicine guide
5 things to consider before you do a systematic review
2) The Cochrane resource
Learn how to conduct, edit, and read systematic reviews
and
3) Systematic reviews and meta-analyses: a step-by-step guide by Dr Susan D Shenkin, Senior Clinical Lecturer and Honorary Consultant.
. Q 7. I understand that it is standard reporting practice to summarise the process for selection of papers for a systematic review using a flow diagram. How can I find out more about this?
A. The type of diagram to which you are referring is a PRISMA flowchart.
Click here to learn more.
. Q 8. Can you recommend tools for assessing bias in randomized and non-randomized studies?
A. For randomized studies, you should find the Cochrane RoB 2 tool useful. This tool allows you to assess a study according to different domains of bias. With the support of algorithms, these domains are each classified according to risk of bias using the categories ‘low risk’, ‘some concerns’ and ‘high risk’.
For non-randomized studies, please consult the site Cochrane Risk Of Bias Tool For Non-Randomized Studies. The information at this site includes guidance on how to classify risk of bias into meaningful severity categories.
. Q 9. In addition to using the above tool, I would like to assess methodological quality more generally for individual studies. Can you recommend a suitable resource?
A. Yes; the Downs and Black scale and Newcastle-Ottawa scale are both relevant in this respect, the first of which is possibly the easier to apply, and hence more realistic to use in for a short project.
Note, however, that I can confirm through correspondence with the principal author for the paper at the link provided above for this scale that on December 2015 they could not vouch for any valid categorization of this scale for designating studies according to overall degrees of methodological quality (e.g. “low”, “medium” and “high”). The user is therefore left with a numerical indicator which lacks a descriptor. By contrast, the Newcastle-Ottawa scale offers a starring system. You are therefore advised to explore both scales to see which is better for your needs.
.Q 10. Can you recommend an introductory resource on meta-analysis?
A. Yes; you can gain a good overview, together with an explanation of the main distinction between a fixed effects and random effects model, by reading the Wikipedia article Meta-analysis.
. Q. 11. Can trials other than randomized controlled trials be included in a meta-analysis?
A. Yes, but strong words of caution and care are needed. Please refer to the Cochrane chapter Including non-randomized studies.
. Q 12. I am interested in carrying out a systematic review on the effectiveness of a particular type of clinical intervention and would like to obtain access to a database of controlled trials for this intervention. Can you offer any suggestions?
A. You should consult the Cochrane Database of Systematic Reviews (Cochrane Reviews) in the first instance. Even if your own specific type of review is not listed, you will probably find related reviews which can point you to a list of references which you can adapt to your own needs.
An obvious next step would be to consult the Cochrane Central Register of Controlled Trials (CENTRAL) and the Database of Abstracts for Reviews for Effects (DARE) for further references peculiar to your own interests.
Each of these databases may be selected on conducting a literature search at The Cochrane Library. Be sensitive, however, to the fact that even Cochrane Reviews are subject to error! Please see the StatsforMedics WordPress page Some Statistical Blunders Discovered Within Medical Publications for a topical example illustrating this very point.
A more extensive database of clinical reviews is accessible by means of the option Review Articles within an Ovid Medline search engine. The relevant link can be found by consulting the section Key Databases at the bottom of the NHS site The Knowledge Network.
. Q 13. I am looking for a suitable checklist for assessing the methodological quality of a systematic review. Are there any which you would recommend?
A. Yes; whether you wish to critically appraise your own systematic review or at the design or a previously conducted systematic review you should familiarise yourself with AMSTAR by referring to the resources itemized below.
1) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews (refer to Additional file 1 at this site within the section Supplementary material )
and
2) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
You should also take time to consider the OQAQ (Overview Quality Assessment Questionnaire), by Oxman and Guyatt, which is a more long-standing quality appraisal tool for systematic reviews to which questions were added in order to form AMSTAR. This tool is discussed in the following articles:
4) Validation of an index of the quality of review articles
and
5) Agreement among reviewers of review articles.
Thanks are due to Dr Andy D Oxman, Norwegian Knowledge Centre for the Health Services, Oslo, Norway for granting permission for both
6) Instructions on the use of the OQAQ
and
7) The OQAQ itself
Loading...
to be made accessible here. Since AMSTAR contains the OQAQ, users of AMSTAR should also find the instructions on use for the OQAQ beneficial.
However, for a systematic review, evaluation of the reporting is distinct from evaluation of the conduct and good quality in one of these two respects does not guarantee good quality in the other. You should therefore also take time to consider the PRISMA checklist referred to in the solution to Q. 12, below.
. Q 14. How should I critically appraise a meta-analysis?
A. Please consider the BMJ paper by T Greenhalgh entitled Papers that summarise other papers (systematic reviews and meta-analyses). The section “Meta-analysis for the non-statistician” contained therein may prove to be of particular relevance. Also, under the header Statistical Primer for Cardiovascular Research, there is a paper Meta-Analysis which is brief and to-the-point regarding good practice in preparing for and carrying out a meta-analysis. Recently, the Equator Network (a resource centre for good research reporting) have introduced a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement which includes a 27-item checklist and four-phase flow diagram aimed at improving the quality of reports of systematic reviews and meta-analyses.
Conducting a meta-analysis
Please be aware that it is unrealistic to attempt a meta-analysis within the time-frame of a short research project, particularly where time on this project is limited through curricular activities which must occur during the period allocated to that project. Time could be better spent conducting a high quality review of studies which are to be considered for a possible future meta-analysis, not least because this would require to be done anyway before considering a meta-analysis. This in itself is a highly demanding exercise, involving use of appropriate checklists and guidelines for quality evaluation and assessment of study bias and identification of possible needs for separate sub-group analyses. All of this should involve a team effort and should not be attempted by a single individual, particularly as judgements on studies need to be compared across reviewers. Please note also that a statistician should be involved a the early planning stages of a systematic review. The role of the statistician is not simply to perform the tricky calculations at the point at which you think you are stuck. They should want to understand the clinical nuances of study outcomes and study design and advise on the appropriate choice of effect size measures (e.g. sensitivity, specificity odds ratio or differences in proportion) well before you have reached that point.
In terms of preparing the way for a future meta-analysis, comprehensive advice is provided below. In terms of separating out studies into groups, this advice also applies to aspects of systematic reviews not involving meta-analysis at this stage.
- It is important to differentiate between the different study designs (e.g. randomized controlled trial or cohort study) assumed for each of the studies you have included in your systematic review. This may lead you to recognize scope for separate subgroup analyses according to study design and indeed, the need for more studies of a particular type in future work in order to make a meta-analysis feasible. If the authors have not specified their study design, you may wish to consult the StatsforMedics WordPress page UNDERSTANDING DIFFERENT STUDY DESIGNS in order to find it. This is assuming that the authors have provided sufficient information for you to do so.
- Be aware that while you may have heard of a random effects model as a means of accommodating heterogeneity across studies, for this model to be used, such heterogeneity ought to arise from relatively modest differences. For example, if the study outcome is mortality or radiological union and different studies have considerably different follow-up times (e.g. 3 months, 30 days, at discharge or not recorded at all) for assessing this outcome across patients or across studies, it does not make sense to lump the studies together. The same applies where, for comparison of outcomes of surgical and non-surgical interventions, the types of surgical or non-surgical interventions vary considerably across studies. In such circumstances, it is far better to report the shortcomings of current research in the field for the purpose of arriving at meaningful conclusions in terms of the effect of the intervention being studied.
- It is possible to estimate between-study heterogeneity in advance of any intention to progress from a literature review to a meta-analysis and to report the corresponding results as a meaningful component of a good quality systematic review which does not include a meta-analysis. Some very useful assistance with assessing between study heterogeneity can be gleaned from the resource Assessing heterogeneity in meta-analysis: Q statistic or I2 index. Please refer to p. 194 of this resource for advice on interpreting the magnitude of this index. Note, however, that, statistical heterogeneity, as obtained by the I2 index, applies to the study effect under consideration in any one case. A range of factors relating to research conduct and design can make the studies themselves heterogeneous and this in turn can lead to statistical heterogeneity. Correspondingly, different effect size estimates may be associated with different values of the I2 index.
Bearing in mind the different calculations involved for your different effect size estimates, it may prove helpful to review the studies included in a given meta-analysis to see if you can identify any clues to explain your statistical heterogeneity results at a speculative level. From a mathematical point of view, using small group sizes in a meta-analysis can lead to spurious and hence irreplicable findings, leading to the desire for larger studies to improve the evidence base. - Be aware that a rule of thumb for use of as funnel plot to detect bias in conducting a meta-analysis is that there should be at least 10 studies for the corresponding meta-analysis.
. Q 15. I am conducting a systematic review and need some training on how to summarize each study in preparation for a narrative synthesis of my findings. Can you recommend a suitable resource?
A. You should find Chapter 7 of the Cochrane Handbook for Systematic Reviews of Interventions useful in terms of ideas and expectations. The resources at the University of South Australia Data extraction site should in turn prove helpful in putting theory into practice. Please note that there is no data extraction form set in stone and you can adapt the instructions and any exemplar form to suit your needs.
. Q 16. Where can I read about how a meta-analysis is performed for a regression analysis?
The source
is recommended for your interest.
. Q 17. Under what circumstances should I check for lead time, length or overdiagnosis bias and what do these terms really mean anyway?
A. Lead time, length or overdiagnosis bias can occur when the results of an intervention for a screened population are compared with those for a population for which diagnosis has been achieved according to signs and symptoms. The results under consideration may include patient survival time, although other possible outcome measures may be of relevance. The effect of each kind of bias is to inflate the overall benefit of the intervention for the screened group. You can learn how these forms of bias differ at: Primer on Lead Time, Length and Overdiagnosis Bias but you should also carefully check the literature you are reviewing to decide whether it is possible that particular types of bias have been accounted for, either by means of the study design, or by means of a specified post-hoc correction.
Critical Appraisals and Systematic Reviews, including meta-analyses by Margaret MacDougall is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
