thesis for systematic review

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT)

How to write the methods section of a systematic review

Home | Blog | How To | How to write the methods section of a systematic review

Covidence breaks down how to write a methods section

The methods section of your systematic review describes what you did, how you did it, and why. Readers need this information to interpret the results and conclusions of the review. Often, a lot of information needs to be distilled into just a few paragraphs. This can be a challenging task, but good preparation and the right tools will help you to set off in the right direction 🗺️🧭.

Systematic reviews are so-called because they are conducted in a way that is rigorous and replicable. So it’s important that these methods are reported in a way that is thorough, clear, and easy to navigate for the reader – whether that’s a patient, a healthcare worker, or a researcher.

Like most things in a systematic review, the methods should be planned upfront and ideally described in detail in a project plan or protocol. Reviews of healthcare interventions follow the PRISMA guidelines for the minimum set of items to report in the methods section. But what else should be included? It’s a good idea to consider what readers will want to know about the review methods and whether the journal you’re planning to submit the work to has expectations on the reporting of methods. Finding out in advance will help you to plan what to include.

Describe what happened

While the research plan sets out what you intend to do, the methods section is a write-up of what actually happened. It’s not a simple case of rewriting the plan in the past tense – you will also need to discuss and justify deviations from the plan and describe the handling of issues that were unforeseen at the time the plan was written. For this reason, it is useful to make detailed notes before, during, and after the review is completed. Relying on memory alone risks losing valuable information and trawling through emails when the deadline is looming can be frustrating and time consuming!

Keep it brief

The methods section should be succinct but include all the noteworthy information. This can be a difficult balance to achieve. A useful strategy is to aim for a brief description that signposts the reader to a separate section or sections of supporting information. This could include datasets, a flowchart to show what happened to the excluded studies, a collection of search strategies, and tables containing detailed information about the studies.This separation keeps the review short and simple while enabling the reader to drill down to the detail as needed. And if the methods follow a well-known or standard process, it might suffice to say so and give a reference, rather than describe the process at length.

Follow a structure

A clear structure provides focus. Use of descriptive headings keeps the writing on track and helps the reader get to key information quickly. What should the structure of the methods section look like? As always, a lot depends on the type of review but it will certainly contain information relating to the following areas:

Selection criteria ⭕
Data collection and analysis 👩‍💻
Study quality and risk of bias ⚖️

Let’s look at each of these in turn.

1. Selection criteria ⭕

The criteria for including and excluding studies are listed here. This includes detail about the types of studies, the types of participants, the types of interventions and the types of outcomes and how they were measured.

2. Search 🕵🏾‍♀️

Comprehensive reporting of the search is important because this means it can be evaluated and replicated. The search strategies are included in the review, along with details of the databases searched. It’s also important to list any restrictions on the search (for example, language), describe how resources other than electronic databases were searched (for example, non-indexed journals), and give the date that the searches were run. The PRISMA-S extension provides guidance on reporting literature searches.

Systematic reviewer pro-tip:

Copy and paste the search strategy to avoid introducing typos

3. Data collection and analysis 👩‍💻

This section describes:

how studies were selected for inclusion in the review
how study data were extracted from the study reports
how study data were combined for analysis and synthesis

To describe how studies were selected for inclusion , review teams outline the screening process. Covidence uses reviewers’ decision data to automatically populate a PRISMA flow diagram for this purpose. Covidence can also calculate Cohen’s kappa to enable review teams to report the level of agreement among individual reviewers during screening.

To describe how study data were extracted from the study reports , reviewers outline the form that was used, any pilot-testing that was done, and the items that were extracted from the included studies. An important piece of information to include here is the process used to resolve conflict among the reviewers. Covidence’s data extraction tool saves reviewers’ comments and notes in the system as they work. This keeps the information in one place for easy retrieval ⚡.

To describe how study data were combined for analysis and synthesis, reviewers outline the type of synthesis (narrative or quantitative, for example), the methods for grouping data, the challenges that came up, and how these were dealt with. If the review includes a meta-analysis, it will detail how this was performed and how the treatment effects were measured.

4. Study quality and risk of bias ⚖️

Because the results of systematic reviews can be affected by many types of bias, reviewers make every effort to minimise it and to show the reader that the methods they used were appropriate. This section describes the methods used to assess study quality and an assessment of the risk of bias across a range of domains.

Steps to assess the risk of bias in studies include looking at how study participants were assigned to treatment groups and whether patients and/or study assessors were blinded to the treatment given. Reviewers also report their assessment of the risk of bias due to missing outcome data, whether that is due to participant drop-out or non-reporting of the outcomes by the study authors.

Covidence’s default template for assessing study quality is Cochrane’s risk of bias tool but it is also possible to start from scratch and build a tool with a set of custom domains if you prefer.

Careful planning, clear writing, and a structured approach are key to a good methods section. A methodologist will be able to refer review teams to examples of good methods reporting in the literature. Covidence helps reviewers to screen references, extract data and complete risk of bias tables quickly and efficiently. Sign up for a free trial today!

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers:

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

Formulate a research question
Develop a protocol
Search for all relevant studies
Apply the selection criteria
Extract the data
Synthesize the data
Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

Academic style
Vague sentences
Style consistency

See an example

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Prevent plagiarism. Run a free check.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
Access to databases and journal archives. Often, your educational institution provides you with access.
Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

They minimize research bias by considering all available evidence and evaluating each study for bias.
Their methods are transparent , so they can be scrutinized by others.
They’re thorough : they summarize all available evidence.
They can be replicated and updated by others.

Systematic reviews also have a few cons .

They’re time-consuming .
They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

Allow you to more effectively communicate your research to other researchers and practitioners
Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

Population(s) or problem(s)
Intervention(s)
Comparison(s)

You can rearrange these four components to write your research question:

What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

Type of study design(s)
The population of patients with eczema
The intervention of probiotics
In comparison to no treatment, placebo , or non-probiotic treatment
The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
Randomized control trials, a type of study design

Their research question was:

What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

Background information : Provide the context of the research question, including why it’s important.
Research objective (s) : Rephrase your research question as an objective.
Selection criteria: State how you’ll decide which studies to include or exclude from your review.
Search strategy: Discuss your plan for finding studies.
Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
Handsearch: Conference proceedings and reference lists of articles
Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

Abstract : A summary of the review
Introduction : Including the rationale and objectives
Methods : Including the selection criteria, search method, data extraction method, and synthesis method
Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
Discussion : Including interpretation of the results and limitations of the review
Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

Student’s t -distribution
Normal distribution
Null and Alternative Hypotheses
Chi square tests
Confidence interval
Quartiles & Quantiles
Cluster sampling
Stratified sampling
Data cleansing
Reproducibility vs Replicability
Peer review
Prospective cohort study

Research bias

Implicit bias
Cognitive bias
Placebo effect
Hawthorne effect
Hindsight bias
Affect heuristic
Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .

An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 3, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Request new password
Create a new account

Doing a Systematic Review: A Student's Guide

Student resources, chapter 1. carrying out a systematic review as a master's thesis.

Here are some other helpful materials for you to read through.

Centre for Review and Dissemination’s guidance for undertaking systematic reviews in health care

www.york.ac.uk/crd/guidance

A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
First published as a Review in Advance on August 08, 2018
Copyright © 2019 by Annual Reviews. All rights reserved

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Article metrics loading...

Full text loading...

Literature Cited

APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Affiliations.

1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected].
2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom.
3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected].
PMID: 30089228
DOI: 10.1146/annurev-psych-010418-102803

Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.

PubMed Disclaimer

LinkOut - more resources

Full text sources.

Ingenta plc
Ovid Technologies, Inc.

Other Literature Sources

scite Smart Citations

Miscellaneous

NCI CPTAC Assay Portal
Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

the definition of a systematic literature review
the purpose of a systematic literature review
the different types of systematic reviews
how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
Prognostic reviews examine the course of a disease and its potential outcomes.
Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

Decide on your team.
Formulate your question.
Plan your research protocol.
Search for the literature.
Screen the literature.
Assess the quality of the studies.
Extract the data.
Analyze the results.
Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
Intervention : which main intervention or treatment are you considering for assessment?
Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

the objectives of your project
the specific methods and processes that you’ll use
the eligibility criteria of the individual studies
how you plan to extract data from individual studies
which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

non-peer-reviewed journals
pharmaceutical industry files
conference proceedings
pharmaceutical company websites
internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

The databases you search and which years they cover
The dates you first run the searches, and when they’re updated
Which strategies you use, including search terms
The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

Pre-screening of all titles and abstracts, and selecting those appropriate
Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

Were the participants really randomly allocated to their groups?
Were the groups similar in terms of prognostic factors?
Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

Knowledge Base
Methodology
Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 18 July 2024.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

Formulate a research question
Develop a protocol
Search for all relevant studies
Apply the selection criteria
Extract the data
Synthesise the data
Write and publish a report

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
Access to databases and journal archives. Often, your educational institution provides you with access.
Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

They minimise research b ias by considering all available evidence and evaluating each study for bias.
Their methods are transparent , so they can be scrutinised by others.
They’re thorough : they summarise all available evidence.
They can be replicated and updated by others.

Systematic reviews also have a few cons .

They’re time-consuming .
They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

Allow you to more effectively communicate your research to other researchers and practitioners
Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

Population(s) or problem(s)
Intervention(s)
Comparison(s)

You can rearrange these four components to write your research question:

What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

Type of study design(s)
The population of patients with eczema
The intervention of probiotics
In comparison to no treatment, placebo , or non-probiotic treatment
The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
Randomised control trials, a type of study design

Their research question was:

What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

Background information : Provide the context of the research question, including why it’s important.
Research objective(s) : Rephrase your research question as an objective.
Selection criteria: State how you’ll decide which studies to include or exclude from your review.
Search strategy: Discuss your plan for finding studies.
Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
Handsearch: Conference proceedings and reference lists of articles
Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

You should apply the selection criteria in two phases:

Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
Your judgement of the quality of the evidence, including risk of bias .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

Abstract : A summary of the review
Introduction : Including the rationale and objectives
Methods : Including the selection criteria, search method, data extraction method, and synthesis method
Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
Discussion : Including interpretation of the results and limitations of the review
Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

To familiarise yourself with the current state of knowledge on your topic
To ensure that you’re not just repeating what others have already done
To identify gaps in knowledge and unresolved problems that your research can address
To develop your theoretical framework and methodology
To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2024, July 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 3 September 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Wiley Open Access Collection

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure 1 . A list of excluded studies with reasons can be found in Appendix B .

An external file that holds a picture, illustration, etc.
Object name is JEBM-15-39-g001.jpg

Study selection flowchart

3.1. Characteristics of included reviews

Table 1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

Author, year	Search strategy (year last searched; no. databases; supplementary searches)	SR design (type of review; no. of studies included)	Topic; subject area	SR objectives	SR authors’ comments on study quality
Crumley, 2005	2004; Seven databases; four journals handsearched, reference lists and contacting authors	SR; = 64	RCTs and CCTs; not specified	To identify and quantitatively review studies comparing two or more different resources (e.g., databases, Internet, handsearching) used to identify RCTs and CCTs for systematic reviews.	Most of the studies adequately described reproducible search methods, expected search yield. Poor quality in studies was mainly due to lack of rigor in reporting selection methodology. Majority of the studies did not indicate the number of people involved in independently screening the searches or applying eligibility criteria to identify potentially relevant studies.
Hopewell, 2007	2002; eight databases; selected journals and published abstracts handsearched, and contacting authors	SR and MA; = 34 (34 in quantitative analysis)	RCTs; health care	To review systematically empirical studies, which have compared the results of handsearching with the results of searching one or more electronic databases to identify reports of randomized trials.	The electronic search was designed and carried out appropriately in majority of the studies, while the appropriateness of handsearching was unclear in half the studies because of limited information. The screening studies methods used in both groups were comparable in most of the studies.
Hopewell, 2007	2005; two databases; selected journals and published abstracts handsearched, reference lists, citations and contacting authors	SR and MA; = 5 (5 in quantitative analysis)	RCTs; health care	To review systematically research studies, which have investigated the impact of gray literature in meta‐analyses of randomized trials of health care interventions.	In majority of the studies, electronic searches were designed and conducted appropriately, and the selection of studies for eligibility was similar for handsearching and database searching. Insufficient data for most studies to assess the appropriateness of handsearching and investigator agreeability on the eligibility of the trial reports.
Horsley, 2011	2008; three databases; reference lists, citations and contacting authors	SR; = 12	Any topic or study area	To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists.	Interpretability and generalizability of included studies was difficult. Extensive heterogeneity among the studies in the number and type of databases used. Lack of control in majority of the studies related to the quality and comprehensiveness of searching.
Morrison, 2012	2011; six databases and gray literature	SR; = 5	RCTs; conventional medicine	To examine the impact of English language restriction on systematic review‐based meta‐analyses	The included studies were assessed to have good reporting quality and validity of results. Methodological issues were mainly noted in the areas of sample power calculation and distribution of confounders.
Robson, 2019	2016; three databases; reference lists and contacting authors	SR; = 37	N/R	To identify and summarize studies assessing methodologies for study selection, data abstraction, or quality appraisal in systematic reviews.	The quality of the included studies was generally low. Only one study was assessed as having low RoB across all four domains. Majority of the studies were assessed to having unclear RoB across one or more domains.
Schmucker, 2017	2016; four databases; reference lists	SR; = 10	Study data; medicine	To assess whether the inclusion of data that were not published at all and/or published only in the gray literature influences pooled effect estimates in meta‐analyses and leads to different interpretation.	Majority of the included studies could not be judged on the adequacy of matching or adjusting for confounders of the gray/unpublished data in comparison to published data.
					Also, generalizability of results was low or unclear in four research projects
Morissette, 2011	2009; five databases; reference lists and contacting authors	SR and MA; = 6 (5 included in quantitative analysis)	N/R	To determine whether blinded versus unblinded assessments of risk of bias result in similar or systematically different assessments in studies included in a systematic review.	Four studies had unclear risk of bias, while two studies had high risk of bias.
O'Mara‐Eves, 2015	2013; 14 databases and gray literature	SR; = 44	N/R	To gather and present the available research evidence on existing methods for text mining related to the title and abstract screening stage in a systematic review, including the performance metrics used to evaluate these technologies.	Quality appraised based on two criteria‐sampling of test cases and adequacy of methods description for replication. No study was excluded based on the quality (author contact).

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n = 3), literature search ( n = 3), study selection ( n = 2), data extraction ( n = 1), and RoB assessment ( n = 2) (Table 2 ).

Summary of findings from review evaluating systematic review methods

Key elements	Author, year	Method assessed	Evaluations/outcomes (P—primary; S—secondary)	Summary of SR authors’ conclusions	Quality of review

Excluding study data based on publication status	Hopewell, 2007	Gray vs. published literature	Pooled effect estimate	Published trials are usually larger and show an overall greater treatment effect than gray trials. Excluding trials reported in gray literature from SRs and MAs may exaggerate the results.	Moderate
	Schmucker, 2017	Gray and/or unpublished vs. published literature	P: Pooled effect estimate	Excluding unpublished trials had no or only a small effect on the pooled estimates of treatment effects. Insufficient evidence to conclude the impact of including unpublished or gray study data on MA conclusions.	Moderate
			S: Impact on interpretation of MA
Excluding study data based on language of publication	Morrison, 2012	English language vs. non‐English language publications	P: Bias in summary treatment effects	No evidence of a systematic bias from the use of English language restrictions in systematic review‐based meta‐analyses in conventional medicine. Conflicting results on the methodological and reporting quality of English and non‐English language RCTs. Further research required.	Low
			S: number of included studies and patients, methodological quality and statistical heterogeneity

Resources searching	Crumley, 2005	Two or more resources searching vs. resource‐specific searching	Recall and precision	Multiple‐source comprehensive searches are necessary to identify all RCTs for a systematic review. For electronic databases, using the Cochrane HSS or complex search strategy in consultation with a librarian is recommended.	Critically low
Supplementary searching	Hopewell, 2007	Handsearching only vs. one or more electronic database(s) searching	Number of identified randomized trials	Handsearching is important for identifying trial reports for inclusion in systematic reviews of health care interventions published in nonindexed journals. Where time and resources are limited, majority of the full English‐language trial reports can be identified using a complex search or the Cochrane HSS.	Moderate
	Horsley, 2011	Checking reference list (no comparison)	P: additional yield of checking reference lists	There is some evidence to support the use of checking reference lists to complement literature search in systematic reviews.	Low
			S: additional yield by publication type, study design or both and data pertaining to costs

Reviewer characteristics	Robson, 2019	Single vs. double reviewer screening	P: Accuracy, reliability, or efficiency of a method	Using two reviewers for screening is recommended. If resources are limited, one reviewer can screen, and other reviewer can verify the list of excluded studies.	Low
			S: factors affecting accuracy or reliability of a method
		Experienced vs. inexperienced reviewers for screening		Screening must be performed by experienced reviewers
		Screening by blinded vs. unblinded reviewers		Authors do not recommend blinding of reviewers during screening as the blinding process was time‐consuming and had little impact on the results of MA
Use of technology for study selection	Robson, 2019	Use of dual computer monitors vs. nonuse of dual monitors for screening	P: Accuracy, reliability, or efficiency of a method	There are no significant differences in the time spent on abstract or full‐text screening with the use and nonuse of dual monitors	Low
			S: factors affecting accuracy or reliability of a method
		Use of Google translate to translate non‐English citations to facilitate screening		Use of Google translate to screen German language citations
	O'Mara‐Eves, 2015	Use of text mining for title and abstract screening	Any evaluation concerning workload reduction	Text mining approaches can be used to reduce the number of studies to be screened, increase the rate of screening, improve the workflow with screening prioritization, and replace the second reviewer. The evaluated approaches reported saving a workload of between 30% and 70%	Critically low
Order of screening	Robson, 2019	Title‐first screening vs. title‐and‐abstract simultaneous screening	P: Accuracy, reliability, or efficiency of a method	Title‐first screening showed no substantial gain in time when compared to simultaneous title and abstract screening.	Low
			S: factors affecting accuracy or reliability of a method

Reviewer characteristics	Robson, 2019	Single vs. double reviewer data extraction	P: Accuracy, reliability, or efficiency of a method	Use two reviewers for data extraction. Single reviewer data extraction followed by the verification of outcome data by a second reviewer (where statistical analysis is planned), if resources preclude	Low
			S: factors affecting accuracy or reliability of a method
		Experienced vs. inexperienced reviewers for data extraction		Experienced reviewers must be used for extracting continuous outcomes data
		Data extraction by blinded vs. unblinded reviewers		Authors do not recommend blinding of reviewers during data extraction as it had no impact on the results of MA
Use of technology for data extraction		Use of dual computer monitors vs. nonuse of dual monitors for data extraction		Using two computer monitors may improve the efficiency of data extraction
		Data extraction by two English reviewers using Google translate vs. data extraction by two reviewers fluent in respective languages		Google translate provides limited accuracy for data extraction
		Computer‐assisted vs. double reviewer extraction of graphical data		Use of computer‐assisted programs to extract graphical data
Obtaining additional data		Contacting study authors for additional data		Recommend contacting authors for obtaining additional relevant data

Reviewer characteristics	Robson, 2019	Quality appraisal by blinded vs. unblinded reviewers	P: Accuracy, reliability, or efficiency of a method	Inconsistent results on RoB assessments performed by blinded and unblinded reviewers. Blinding reviewers for quality appraisal not recommended	Low
			S: factors affecting accuracy or reliability of a method
	Morissette, 2011	Risk of bias (RoB) assessment by blinded vs. unblinded reviewers	P: Mean difference and 95% confidence interval between RoB assessment scores	Findings related to the difference between blinded and unblinded RoB assessments are inconsistent from the studies. Pooled effects show no differences in RoB assessments for assessments completed in a blinded or unblinded manner.	Moderate
			S: qualitative level of agreement, mean RoB scores and measures of variance for the results of the RoB assessments, and inter‐rater reliability between blinded and unblinded reviewers
	Robson, 2019	Experienced vs. inexperienced reviewers for quality appraisal	P: Accuracy, reliability, or efficiency of a method	Reviewers performing quality appraisal must be trained. Quality assessment tool must be pilot tested.	Low
			S: factors affecting accuracy or reliability of a method
		Use of additional guidance vs. nonuse of additional guidance for quality appraisal		Providing guidance and decision rules for quality appraisal improved the inter‐rater reliability in RoB assessments.
Obtaining additional data		Contacting study authors for obtaining additional information/use of supplementary information available in the published trials vs. no additional information for quality appraisal		Additional data related to study quality obtained by contacting study authors improved the quality assessment.
RoB assessment of qualitative studies		Structured vs. unstructured appraisal of qualitative research studies		Use of structured tool if qualitative and quantitative studies designs are included in the review. For qualitative reviews, either structured or unstructured quality appraisal tool can be used.

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n = 4), Hopewell et al. 20 and Crumley et al. 19 ( n = 30), and Robson et al. 14 and Morissette et al. 23 ( n = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table 2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

UNC Libraries
HSL Academic Process
Systematic Reviews

Systematic Reviews: Home

Created by health science librarians.

Systematic review resources

What is a Systematic Review?

A simplified process map, how can the library help, publications by hsl librarians, systematic reviews in non-health disciplines, resources for performing systematic reviews.

Step 1: Complete Pre-Review Tasks
Step 2: Develop a Protocol
Step 3: Conduct Literature Searches
Step 4: Manage Citations
Step 5: Screen Citations
Step 6: Assess Quality of Included Studies
Step 7: Extract Data from Included Studies
Step 8: Write the Review

Check our FAQ's

Email us

Call (919) 962-0800

Make an appointment with a librarian

Request a systematic or scoping review consultation

A systematic review is a literature review that gathers all of the available evidence matching pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods, documented in a protocol, to minimize bias , provide reliable findings , and inform decision-making. ¹

There are many types of literature reviews.

Before beginning a systematic review, consider whether it is the best type of review for your question, goals, and resources. The table below compares a few different types of reviews to help you decide which is best for you.

Comparing Systematic, Scoping, and Systematized Reviews
Systematic Review	Scoping Review	Systematized Review
Conducted for Publication	Conducted for Publication	Conducted for Assignment, Thesis, or (Possibly) Publication
Protocol Required	Protocol Required	No Protocol Required
Focused Research Question	Broad Research Question	Either
Focused Inclusion & Exclusion Criteria	Broad Inclusion & Exclusion Criteria	Either
Requires Large Team	Requires Small Team	Usually 1-2 People

Scoping Review Guide For more information about scoping reviews, refer to the UNC HSL Scoping Review Guide.

Systematic Reviews: A Simplified, Step-by-Step Process Map

UNC HSL's Simplified, Step-by-Step Process Map A PDF file of the HSL's Systematic Review Process Map.
Text-Only: UNC HSL's Systematic Reviews - A Simplified, Step-by-Step Process A text-only PDF file of HSL's Systematic Review Process Map.

Creative commons license applied to systematic reviews image requires that reusers give credit to the creator. It allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, for noncommercial purposes only.

The average systematic review takes 1,168 hours to complete. ¹ A librarian can help you speed up the process.

Systematic reviews follow established guidelines and best practices to produce high-quality research. Librarian involvement in systematic reviews is based on two levels. In Tier 1, your research team can consult with the librarian as needed. The librarian will answer questions and give you recommendations for tools to use. In Tier 2, the librarian will be an active member of your research team and co-author on your review. Roles and expectations of librarians vary based on the level of involvement desired. Examples of these differences are outlined in the table below.

Roles and expectations of librarians based on level of involvement desired.
Tasks	Tier 1: Consultative	Tier 2: Research Partner / Co-author

Guidance on process and steps	Yes	Yes
Background searching for past and upcoming reviews	Yes	Yes

Development and/or refinement of review topic	Yes	Yes
Assistance with refinement of PICO (population, intervention(s), comparator(s), and key questions	Yes	Yes
Guidance on study types to include	Yes	Yes

Guidance on protocol registration	Yes	Yes

Identification of databases for searches	Yes	Yes
Instruction in search techniques and methods	Yes	Yes
Training in citation management software use for managing and sharing results	Yes	Yes
Development and execution of searches	No	Yes
Downloading search results to citation management software and removing duplicates	No	Yes
Documentation of search strategies	No	Yes
Management of search results	No	Yes

Guidance on methods	Yes	Yes
Guidance on data extraction, and management techniques and software	Yes	Yes

Suggestions of journals to target for publication	Yes	Yes
Drafting of literature search description in "Methods" section	No	Yes
Creation of PRISMA diagram	No	Yes
Drafting of literature search appendix	No	Yes
Review other manuscript sections and final draft	No	Yes
Librarian contributions warrant co-authorship	No	Yes

Request a systematic or scoping review consultation

The following are systematic and scoping reviews co-authored by HSL librarians.

Only the most recent 15 results are listed. Click the website link at the bottom of the list to see all reviews co-authored by HSL librarians in PubMed

Researchers conduct systematic reviews in a variety of disciplines. If your focus is on a topic outside of the health sciences, you may want to also consult the resources below to learn how systematic reviews may vary in your field. You can also contact a librarian for your discipline with questions.

EPPI-Centre methods for conducting systematic reviews The EPPI-Centre develops methods and tools for conducting systematic reviews, including reviews for education, public and social policy.

Environmental Topics

Collaboration for Environmental Evidence (CEE) CEE seeks to promote and deliver evidence syntheses on issues of greatest concern to environmental policy and practice as a public service

Social Sciences

Siddaway AP, Wood AM, Hedges LV. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annu Rev Psychol. 2019 Jan 4;70:747-770. doi: 10.1146/annurev-psych-010418-102803. A resource for psychology systematic reviews, which also covers qualitative meta-syntheses or meta-ethnographies
The Campbell Collaboration

Social Work

Software engineering

Guidelines for Performing Systematic Literature Reviews in Software Engineering The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students.

Sport, Exercise, & Nutrition

Application of systematic review methodology to the field of nutrition by Tufts Evidence-based Practice Center Publication Date: 2009
Systematic Reviews and Meta-Analysis — Open & Free (Open Learning Initiative) The course follows guidelines and standards developed by the Campbell Collaboration, based on empirical evidence about how to produce the most comprehensive and accurate reviews of research

Systematic Reviews by David Gough, Sandy Oliver & James Thomas Publication Date: 2020

Updating reviews

Updating systematic reviews by University of Ottawa Evidence-based Practice Center Publication Date: 2007
Next: Step 1: Complete Pre-Review Tasks >>
Last Updated: Jul 15, 2024 4:55 PM
URL: https://guides.lib.unc.edu/systematic-reviews

Systematic Reviews and Meta Analysis

Getting Started
Guides and Standards
Review Protocols
Databases and Sources
Randomized Controlled Trials
Controlled Clinical Trials
Observational Designs
Tests of Diagnostic Accuracy
Software and Tools
Where do I get all those articles?
Collaborations
EPI 233/528
Countway Mediated Search
Risk of Bias (RoB)

Systematic review Q & A

What is a systematic review.

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

For details about carrying out systematic reviews, see the Guides and Standards section of this guide.

Is my research topic appropriate for systematic review methods?

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of the findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

If not a systematic review, then what?

You might consider performing a scoping review . This framework allows iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. The framework includes built-in mechanisms to adjust the analysis as the work progresses and more is learned about the topic. A scoping review won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

Can my student complete a systematic review during her summer project?

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion criteria and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

How can I know if my topic has been been reviewed already?

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be captured by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist. Cochrane systematic reviews do not include 'systematic' in the title, however. It's worth checking the Cochrane Database of Systematic Reviews independently.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO , a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library .

Next: Guides and Standards >>
Last Updated: Sep 4, 2024 4:04 PM
URL: https://guides.library.harvard.edu/meta-analysis

10 Shattuck St, Boston MA 02115 | (617) 432-2136

| |
Copyright © 2020 President and Fellows of Harvard College. All rights reserved.

Open access
Published: 12 December 2017

Acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in Europe

Livia Puljak ORCID: orcid.org/0000-0002-8467-6061 1 , 2 , 3 &
Damir Sapunar 3

Systematic Reviews volume 6 , Article number: 253 ( 2017 ) Cite this article

16k Accesses

25 Citations

63 Altmetric

Metrics details

Systematic reviews (SRs) have been proposed as a type of research methodology that should be acceptable for a graduate research thesis. The aim of this study was to analyse whether PhD theses in European biomedical graduate programs can be partly or entirely based on SRs.

In 2016, we surveyed individuals in charge of European PhD programs from 105 institutions. The survey asked about acceptance of SRs as the partial or entire basis for a PhD thesis, their attitude towards such a model for PhD theses, and their knowledge about SR methodology.

We received responses from 86 individuals running PhD programs in 68 institutions (institutional response rate of 65%). In 47% of the programs, SRs were an acceptable study design for a PhD thesis. However, only 20% of participants expressed a personal opinion that SRs meet the criteria for a PhD thesis. The most common reasons for not accepting SRs as the basis for PhD theses were that SRs are ‘not a result of a PhD candidate’s independent work, but more of a team effort’ and that SRs ‘do not produce enough new knowledge for a dissertation’. The majority of participants were not familiar with basic concepts related to SRs; questions about meta-analyses and the type of plots frequently used in SRs were correctly answered by only one third of the participants.

Conclusions

Raising awareness about the importance of SRs and their methodology could contribute to higher acceptance of SRs as a type of research that forms the basis of a PhD thesis.

Peer Review reports

Systematic reviews (SRs) are a type of secondary research, which refers to the analysis of data that have already been collected through primary research [ 1 ]. Even though SRs are a secondary type of research, a SR needs to start with a clearly defined research question and must follow rigorous research methodology, including definition of the study design a priori, data collection, appraisal of study quality, numerical analyses in the form of meta-analyses and other analyses when relevant and formulation of results and conclusions. Aveyard and Sharp defined SRs as ‘original empirical research’ because they ‘review, evaluate and synthesise all the available primary data, which can be either quantitative or qualitative’ [ 2 ]. Therefore, a SR represents a new research contribution to society and is considered the highest level in the hierarchy of evidence in medicine [ 3 ].

SRs have been proposed as a type of research methodology that should be acceptable as the basis for a graduate research thesis [ 4 , 5 ]. To the best of our knowledge, there are no reports on the acceptance of SRs as the basis for PhD theses. A recent review addressed potential advantages and disadvantages of such a thesis type and presented opposing arguments about the issue [ 5 ]. However, there were no actual data that would indicate how prevalent one opinion is over another with regard to the acceptance of a SR as the primary research methodology for a PhD thesis. The aim of this cross-sectional study was to assess whether a PhD thesis in European biomedical graduate programs can be partly or entirely based on a SR, as well as to explore the attitudes and knowledge of individuals in charge of PhD programs with regard to a thesis of this type.

Participants

The Organization of PhD Education in Biomedicine and Health Sciences in the European System (ORPHEUS) includes 105 institutional members from 40 countries and six associate members from Canada, Georgia, Iran, Kyrgyzstan, Kazakhstan and the USA [ 6 ]. The ORPHEUS encompasses a network of higher education institutions committed to developing and disseminating best practice within PhD training programs in biomedicine, health sciences and public health. ORPHEUS approved the use of their mailing list for the purpose of this study. The mailing list had 1049 contacts. The study authors were not given the mailing list due to data protection and privacy. Instead, it was agreed that ORPHEUS officials would send the survey via email to the mailing list. The General Secretary of the ORPHEUS contacted individuals responsible for PhD programs (directors or deputy directors) among the institutional members, via e-mail, on 5th of July 2016. These individuals were sent an invitation to complete an online survey about SRs as the basis for PhD theses. We invited only individuals responsible for PhD programs (e.g., directors, deputy directors, head of graduate school, vice deans for graduate school or similar). We also asked them to communicate with other individuals in charge of their program to make sure that only one person per PhD program filled out the survey. If there were several PhD programs within one institution, we asked for participation of one senior person per program.

The survey was administered via Survey Monkey (Portland, OR, USA). The survey took 5–10 min to complete. One reminder was sent to the targeted participants 1 month after the first mail.

The ethics committee of the University of Split School of Medicine approved this study, which formed part of the Croatian Science Foundation grant no. IP-2014-09-7672 ‘Professionalism in Health Care’.

Questionnaire

The 20-item questionnaire, designed specifically for this study by both authors (LP and DS), was first tested for face validity and clarity among five individuals in charge of PhD programs. The questionnaire was then modified according to their feedback. The questionnaire included questions about their PhD program; whether PhD candidates are required to publish manuscript(s) before thesis defence; the minimum number of required manuscripts for defending a PhD thesis; the authorship requirements for a PhD candidate with regard to published manuscript(s); whether there is a requirement for a PhD candidate to publish manuscript(s) in journals indexed in certain databases or journals of certain quality, and how the quality is defined; the description about other requirements for defending a PhD thesis; whether a SR partly or fully meets requirements for approval of a PhD thesis in their graduate program; what are the rules related to the use of a SR as the basis for a PhD thesis; and the number of PhD theses based on SRs relative to other types of research methods.

Participants were also asked about their opinion with regard to the main reasons that SRs are not recognised in some institutions as the basis for a doctoral dissertation, and their opinion about literature reviews, using a four-item Likert scale, ranging from ‘agree’ to ‘disagree’, including an option for ‘don’t know’. In the last question, the participants’ knowledge about SR methodology was examined using nine statements; participants had to rate each statement as either ‘correct’, ‘incorrect’, ‘unsure’ or ‘I don’t know’. Finally, participants were invited to leave their email address if they wanted to receive survey results. The survey sent to the study participants can be found in an additional file (Additional file 1 ).

Data analysis

Survey responses were entered into a spreadsheet, checked by both authors and analysed using Microsoft Excel (Microsoft Inc., Redmond, WA, USA). Descriptive data are presented as frequencies and percentages. All raw data and analysed data sets used in the manuscript are available from authors on request. A point-biserial correlation (SPSS, IBM, Chicago, IL, USA) was used to measure the strength of the association between results on the knowledge test (continuous variable) and the attitude towards SRs as the basis for dissertations (dichotomous variable; we used the answer to the following question as this measure: ‘Do you agree that a systematic review, in whole or in part, meets the criteria for a publication on which a doctoral dissertation can be based?’).

Study participants

There are 105 institutions included in the ORPHEUS network. We received a response from 86 individuals representing 68 institutions from 37 countries (65% institutional response rate). There were more respondents than institutions because some institutions have several PhD programs and thus several program directors. Those responders were used as a unit of analysis in the analysis of attitudes and knowledge; institutions were the unit of analysis when analysing criteria for theses. Some of the questionnaires ( n = 15) were only partly completed. In most cases, the missing data were related to knowledge about SR methodology.

Overview of requirements for a dissertation

Based on the information provided by the graduate program directors, in the majority of the included PhD programs, students were required to publish a research manuscript prepared within their PhD thesis prior to their thesis defence (83%; n = 64). Among 13 programs (17%) that did not have this requirement, five respondents (38%) indicated that in their opinion their school’s rules related to a PhD thesis should be changed such as to specify that each thesis should be based on work that is already published in a journal.

The minimum number of published manuscripts necessary for the PhD thesis defence was prespecified in 94% ( n = 60) of the programs that required publication of research manuscripts prior to the thesis defence. In most of the programs (37%; n = 22), the number of required manuscripts was three or more. Two manuscripts were required in 30% ( n = 18) and one was required in 33% ( n = 20) of the programs. In four programs, there was no formal policy on this matter, but there was a strong expectation that the student will have contributed substantially to several manuscripts in peer-reviewed journals.

In most cases, the PhD candidates’ contribution to published manuscripts within the PhD thesis was determined through first authorship. A requirement that a PhD candidate should be the first author on a manuscript(s) that constitutes a PhD thesis was reported in 82% ( n = 64) of the graduate programs.

In 60% ( n = 52) of the graduate programs, the quality of the journals where a PhD candidate has to publish research manuscripts as a part of a PhD thesis was defined by the database in which these journals are indexed. The most commonly specified databases were Web of Science (41%; n = 35) and MEDLINE/PubMed (13%; n = 11), followed by Science Citation Index, Scopus, Current Contents, a combination of several databases or, in two cases, a combination of journals from a list defined by some governing body.

Systematic reviews as a PhD thesis

SRs, in whole or in part, met the criteria for acceptable research methodology for a PhD thesis in 47% ( n = 40) of programs, whereas 53% ( n = 46) of programs specifically stated that they did not accept SRs in this context (Fig. 1 a, b). Among the programs that accepted SRs, theses could be exclusively based on a SR in 42% ( n = 17) of programs, while in the remaining programs, SRs were acceptable as one publication among others in a dissertation.

a European PhD programs that recognise a systematic review as a PhD thesis (green dot) and those that do not (red dot). Half red and half green dots indicate the five universities with institutions that have opposite rules regarding recognition of a systematic review as a PhD thesis. The pie chart presents b the percentage of the programs in which systematic reviews, in whole or in part, meet the criteria for a dissertation and c the opinion of participants about whether systematic reviews should form the basis of a publication within a PhD dissertation

The majority of participants (80%; n = 69) indicated that SRs did not meet criteria for a publication on which a PhD dissertation should be based (Fig. 1 c). The main arguments for not recognising a SR as the basis for a PhD thesis are listed in Table 1 . The majority of respondents were neutral regarding the idea that scoping reviews or SRs should replace traditional narrative reviews preceding the results of clinical and basic studies in doctoral theses. Most of the respondents agreed that narrative or critical/discursive literature reviews preceding clinical studies planned as part of a dissertation should be replaced with systematic reviews (Table 2 ).

Most of the programs that accepted SRs as a research methodology acceptable for PhD theses had defined rules related to the use of an SR as part of a PhD thesis (Fig. 2 ). The most common rule was that a SR can be one publication among others within a PhD thesis. Some of the respondents indicated that empty (reviews that did not find a single study that should be included after literature search) or updated reviews could also be used for a PhD thesis (Fig. 2 ).

Frequency of different rules that define the use of systematic reviews as a part of a PhD thesis in European biomedical graduate programs

The results of the survey regarding knowledge about SR methodology indicated that the majority of respondents were not familiar with this methodology. Only three out of nine questions were correctly answered by more than 80% of the participants, and questions about meta-analyses and the type of plots frequently used in a SR were correctly answered by only one third of the participants (Table 3 ). The association between participants’ results on the knowledge test and attitudes towards SRs was tested using a point-biserial correlation; this revealed that lack of knowledge was not correlated with negative attitudes towards SRs ( r pb = 0.011; P = 0.94).

In this study conducted among individuals in charge of biomedical graduate programs in Europe, we found that 47% of programs accepted SRs as research methodology that can partly or fully fulfil the criteria for a PhD thesis. However, most of the participants had negative attitudes about such a model for a PhD thesis, and most had insufficient knowledge about the basic aspects of SR methodology. These negative attitudes and lack of knowledge likely contribute to low acceptance of SRs as an acceptable study design to include in a PhD thesis.

A limitation of this study was that we relied on participants’ responses and not on assessments of formal rules of PhD programs. Due to a lack of familiarity with SRs, it is possible that the respondents gave incorrect answers. We believe that this might be the case since we received answers from different programs in the same university, where one person claimed that SRs were accepted in their program, and the other person claimed that they were not accepted in the other program. We had five such cases, so it is possible that institutions within the same university have different rules related to accepted research methodology in graduate PhD programs. This study may not be generalisable to different PhD programs worldwide that were not surveyed. The study is also not generalisable to Europe, as there are no universal criteria or expectations for PhD theses in Europe. Even in the same country, there may be different models and expectations for a PhD in different higher education institutions.

A recent study indicated a number of opposing views and disadvantages related to SRs as research methodology for graduate theses, including lack of knowledge and understanding by potential supervisors, which may prevent them from being mentors and assisting students to complete such a study [ 5 ]. This same manuscript emphasised that there may be constraints if the study is conducted in a resource-limited environment without access to electronic databases, that there may be a very high or very low number of relevant studies that can impact the review process, that methods may not be well developed for certain types of research syntheses and that it may be difficult to publish SRs [ 5 ].

Some individuals believe that a SR is not original research. Indeed, it has been suggested that SRs as ‘secondary research’ are different than ‘primary or original research’, implying that they are inferior and lacking in novelty and methodological rigour as compared to studies that are considered primary research. In 1995, Feinstein suggested that such studies are ‘statistical alchemy for the 21st century’ and that a meta-analysis removes or destructs ‘scientific requirements that have been so carefully developed and established during the 19th and 20th centuries’ [ 7 ]. There is little research about this methodological issue. Meerpohl et al. surveyed journal editors and asked whether they consider SRs to be original studies. The majority of the editors indicated that they do think that SRs are original scientific contributions (71%) and almost all journals (93%) published SRs. That study also highlighted that the definition of original research may be a grey area [ 8 ]. They argued that, in an ideal situation, ‘the research community would accept systematic reviews as a research category of its own, which is defined by methodological criteria, as is the case for other types of research’ [ 8 ]. Biondi-Zoccai et al. pointed out that the main criteria to judge a SR should be its novelty and usefulness, and not whether it is original/primary or secondary research [ 9 ].

In our study, 80% of the participants reported negative attitudes, and more than half of the respondents agreed with a statement that SRs are ‘not a result of the candidate’s independent work since systematic reviews tend to be conducted by a team’. This opinion is surprising since other types of research are also conducted within a team, and single authorship is very rare in publications that are published within a PhD thesis. On the contrary, the mean number of authors of research manuscripts is continuously increasing [ 10 ]. At the very least, the authors of manuscripts within a PhD will include the PhD candidate and a mentor, which is a team in and of itself. Therefore, it is unclear why somebody would consider it a problem that a SR is conducted within a team.

The second most commonly chosen argument against such a thesis was that SRs ‘do not produce enough new knowledge for a dissertation’. The volume of a SR largely depends on the number of included studies and the available data for numerical analyses. Therefore, it is unfair to label a SR as a priori lacking in new knowledge. There are SRs with tens or hundreds of included studies, and some of them not only include meta-analyses, but also network meta-analyses, which are highly sophisticated statistical methods. However, limiting SRs within a thesis only to those with meta-analysis would be unfair because sometimes meta-analysis is not justified due to clinical or statistical heterogeneity [ 11 ] and the presence or absence of a meta-analysis is not an indicator of the quality of a SR. Instead, there are relevant checklists for appraising methodological and reporting quality of a SR [ 12 , 13 ].

The third most commonly chosen argument against SRs within PhD theses was ‘lack of adequate training of candidates in methodology of systematic reviews’. This could refer to either insufficient formal training or insufficient mentoring. The graduate program and the mentor need to ensure that a PhD candidate receives sufficient knowledge to complete the proposed thesis topic. Successful mentoring in academic medicine requires not only commitment and interpersonal skills from both the mentor and mentee, but also a facilitating institutional environment [ 14 ]. This finding could be a result of a lack of capacity and knowledge for conducting SRs in the particular institutions where the survey was conducted, and not general opinion related to learning a research method when conducting a PhD study. Formal training in skills related to SRs and research synthesis methods [ 15 , 16 ], as well as establishing research collaborations with researchers experienced in this methodology, could alleviate this concern.

One third of the participants indicated a ‘lack of appreciation of systematic review methodology among faculty members’ as a reason against such a thesis model. This argument, as well as the prevalent negative attitude towards SRs as PhD theses, perhaps can be traced to a lack of knowledge about SR methodology; however, although the level of knowledge was quite low in our study, there was no statistically significant correlation between knowledge and negative attitudes. Of the nine questions about SR research methodology, only three questions were correctly answered by more than half of the participants. This could be a cause for concern because it has been argued that any health research should begin with a SR of the literature [ 17 ]. It has also been argued that the absence of SRs in the context of research training might severely hamper research trainees and may negatively impact the research conducted [ 18 ]. Thus, it has been recommended that SRs should be included ‘whenever appropriate, as a mandatory part of any PhD program or candidature’ [ 18 ].

It has recently been suggested that the overwhelming majority of investment in research represents an ‘avoidable waste’ [ 19 ]. Research that is not necessary harms both the public and patients, because funds are not invested where they are really necessary, and necessary research may not be conducted [ 17 ]. This is valid not only for clinical trials, but also for other types of animal and human experiments [ 20 ]. SRs can help improve the design of new experiments by relying on current evidence in the field and by helping to clarify which questions still need to be addressed. SRs can be instrumental in improving methodological quality of new experiments, providing evidence-based recommendations for research models, reducing avoidable waste, and enabling evidence-based translational research [ 20 ].

Four respondents from three institutions indicated that empty SRs are accepted as a PhD thesis. While it makes sense to include such a SR as a part of the thesis to indicate lack of evidence in a certain field, it is highly unlikely that an entire thesis can be based on an empty SR, without a single included study.

There are many advantages of a SR as a graduate thesis [ 4 , 5 ], especially as a research methodology suitable for low-resource settings. A PhD candidate can prepare a Cochrane SR as a part of the PhD thesis, yielding a high-impact publication [ 4 ]. Non-Cochrane SRs can also be published in high-impact journals. A PhD candidate involved in producing a SR within a PhD thesis goes through the same research process as those conducting primary research, from setting up a hypothesis and a research question, to development of a protocol, data collection, data analysis and appraisal, and formulation of conclusions. Graduate programs can set limits, such as the prevention of empty reviews and the recognition of updated reviews as valid for a PhD thesis, and engage experienced researchers as advisors and within thesis evaluation committees, to ensure that a candidate will conduct a high-quality SR [ 4 ]. Conducting a SR should not be mandatory, but candidates and mentors willing to produce such research within a graduate program should be allowed to do so.

Further studies in this field could provide better insight into attitudes related to SRs as graduate theses and explore interventions that can be used to change negative attitudes and improve knowledge of SRs among decision-makers in graduate education.

Raising awareness about the importance of SRs in biomedicine, the basic aspects of SR methodology and the status of SRs as original secondary research could contribute to greater acceptance of SRs as potential PhD theses. Our results can be used to create strategies that will enhance acceptance of SRs among graduate education program directors.

Gopalakrishnan S, Ganeshkumar P. Systematic reviews and meta-analysis: understanding the best evidence in primary healthcare. J. Fam. Med Prim Care. 2013;2(1):9–14.

Article CAS Google Scholar

Aveyard H, Sharp P. A beginner’s guide to evidence-based practice in health and social care. Glasgow: McGraw Open Press University; 2011.

Google Scholar

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article CAS PubMed Google Scholar

Puljak L, Sambunjak D. Cochrane systematic review as a PhD thesis: an alternative with numerous advantages. Biochemia Medica. 2010;20(3):319–2.

ten Ham-Baloyi W, Jordan P. Systematic review as a research method in post-graduate nursing education. Health SA Gesondheid. 2016;21:120–8.

Article Google Scholar

Organisation for PhD Education in Biomedicine and Health Sciences in the European System (ORPHEUS). Available at: http://www.orpheus-med.org/ .

Feinstein AR. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol. 1995;48(1):71–9.

Meerpohl JJ, Herrle F, Reinders S, Antes G, von Elm E. Scientific value of systematic reviews: survey of editors of core clinical journals. PLoS One. 2012;7(5):e35732.

Article CAS PubMed PubMed Central Google Scholar

Biondi-Zoccai G, Lotrionte M, Landoni G, Modena MG. The rough guide to systematic reviews and meta-analyses. HSR proc intensive care cardiovascular anesth. 2011;3(3):161–73.

CAS Google Scholar

Baethge C. Publish together or perish: the increasing number of authors per article in academic journals is the consequence of a changing scientific culture. Some researchers define authorship quite loosely. Dtsch. Arztebl. Int. 2008;105(20):380–3.

PubMed PubMed Central Google Scholar

Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statist Med. 2002;21:1539–58.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Article PubMed PubMed Central Google Scholar

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, Porter AC, Tugwell P, Moher D, Bouter LM. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.

Sambunjak D, Straus SE, Marusic A. A systematic review of qualitative research on the meaning and characteristics of mentoring in academic medicine. J Gen Intern Med. 2010;25(1):72–8.

Article PubMed Google Scholar

Balajic K, Barac-Latas V, Drenjancevic I, Ostojic M, Fabijanic D, Puljak L. Influence of a vertical subject on research in biomedicine and activities of the Cochrane collaboration branch on medical students’ knowledge and attitudes toward evidence-based medicine. Croat Med J. 2012;53(4):367–73.

Marusic A, Sambunjak D, Jeroncic A, Malicki M, Marusic M. No health research without education for research—experience from an integrated course in undergraduate medical curriculum. Med Teach. 2013;35(7):609.

Mahtani KR. All health researchers should begin their training by preparing at least one systematic review. J R Soc Med. 2016;109(7):264–8.

Olsson C, Ringner A, Borglin G. Including systematic reviews in PhD programmes and candidatures in nursing - ‘Hobson’s choice’? Nurse Educ Pract. 2014;14(2):102–5.

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

de Vries RB, Wever KE, Avey MT, Stephens ML, Sena ES, Leenaars M. The usefulness of systematic reviews of animal experiments for the design of preclinical and clinical studies. ILAR J. 2014;55(3):427–37.

Download references

Acknowledgements

The authors thank the ORPHEUS secretariat for administering the survey and the study participants for taking time to participate in the survey. We are grateful to Prof. Ana Marušić for the critical reading of the manuscript.

This research was funded by the Croatian Science Foundation, grant no. IP-2014-09-7672 ‘Professionalism in Health Care’. The funder had no role in the design of this study or its execution and data interpretation.

Availability of data and materials

The datasets used and/or analysed for the current study are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Cochrane Croatia, University of Split School of Medicine, Šoltanska 2, 21000, Split, Croatia

Livia Puljak

Department for Development, Research and Health Technology Assessment, Agency for Quality and Accreditation in Health Care and Social Welfare, Planinska 13, 10000, Zagreb, Croatia

Laboratory for Pain Research, University of Split School of Medicine, Šoltanska 2, 21000, Split, Croatia

Livia Puljak & Damir Sapunar

You can also search for this author in PubMed Google Scholar

Contributions

Both authors participated in the study design, data collection and analysis and writing of the manuscript, and both read and approved the final manuscript.

Corresponding author

Correspondence to Livia Puljak .

Ethics declarations

Ethics approval and consent to participate.

The Ethics Committee of the University of Split School of Medicine approved the study. All respondents consented to participate in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional file

Additional file 1:.

Online survey used in the study. Full online survey that was sent to the study participants. (PDF 293 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Puljak, L., Sapunar, D. Acceptance of a systematic review as a thesis: survey of biomedical doctoral programs in Europe. Syst Rev 6 , 253 (2017). https://doi.org/10.1186/s13643-017-0653-x

Download citation

Received : 29 August 2017

Accepted : 30 November 2017

Published : 12 December 2017

DOI : https://doi.org/10.1186/s13643-017-0653-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Systematic review
PhD program
Biomedicine
Study design

Systematic Reviews

ISSN: 2046-4053

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

Search Menu
Sign in through your institution
Advance articles
Editor's Choice
100 years of the AJE
Collections
Author Guidelines
Submission Site
Open Access Options
About American Journal of Epidemiology
About the Johns Hopkins Bloomberg School of Public Health
Journals Career Network
Editorial Board
Advertising and Corporate Services
Self-Archiving Policy
Dispatch Dates
Journals on Oxford Academic
Books on Oxford Academic

Article Contents

Assessing the certainty of the evidence in systematic reviews: importance, process, and use.

Article contents
Figures & tables
Supplementary Data

Romina Brignardello-Petersen, Gordon H Guyatt, Assessing the Certainty of the Evidence in Systematic Reviews: Importance, Process, and Use, American Journal of Epidemiology , 2024;, kwae332, https://doi.org/10.1093/aje/kwae332

Permissions Icon Permissions

When interpreting results and drawing conclusions, authors of systematic reviews should consider the limitations of the evidence included in their review. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach provides a framework for the explicit consideration of the limitations of the evidence included in a systematic review, and for incorporating this assessment into the conclusions. Assessments of certainty of evidence are a methodological expectation of systematic reviews. The certainty of the evidence is specific to each outcome in a systematic review, and can be rated as high, moderate, low, or very low. Because it will have an important impact, before conducting certainty of evidence, reviewers must clarify the intent of their question: are they interested in causation or association. Serious concerns regarding limitations in the study design, inconsistency, imprecision, indirectness, and publication bias can decrease the certainty of the evidence. Using an example, this article describes and illustrates the importance and the steps for assessing the certainty of evidence and drawing accurate conclusions in a systematic review.

publication bias
grade approach

Email alerts

Citing articles via, looking for your next opportunity.

Recommend to your Library

Affiliations

Online ISSN 1476-6256
Print ISSN 0002-9262
Copyright © 2024 Johns Hopkins Bloomberg School of Public Health
About Oxford Academic
Publish journals with us
University press partners
What we publish
New features
Open access
Institutional account management
Rights and permissions
Get help with access
Accessibility
Advertising
Media enquiries
Oxford University Press
Oxford Languages
University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

Copyright © 2024 Oxford University Press
Cookie settings
Cookie policy
Privacy policy
Legal notice

This Feature Is Available To Subscribers Only

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Log in using your username and password

Search More Search for this keyword Advanced search
Latest content
For authors
Browse by collection
BMJ Journals

Jiulin Liu 1 , 2 ,
Ran Xiao 3 ,
Huijia Yin 1 ,
Ying Hu 1 ,
Siyu Zhen 1 ,
Shihao Zhou 1 , 2 ,
http://orcid.org/0000-0001-8516-1396 Dongming Han 1
1 Department of Magnetic Resonance Imaging (MRI) , The First Affiliated Hospital of Xinxiang Medical University , Weihui , Henan , China
2 Department of Radiology , Luoyang Orthopedic-Traumatological Hospital of Henan Province (Henan Provincial Orthopedic Hospital) , Zhengzhou , Henan , China
3 Department of Respiratory Medicine , The First Affiliated Hospital of Xinxiang Medical University , Weihui , Henan , China
Correspondence to Dr Dongming Han; 625492590{at}qq.com

Objective The objective is to evaluate the diagnostic effectiveness of contrast-enhanced spectral mammography (CESM) in the diagnosis of breast cancer.

Data sources PubMed, Embase and Cochrane libraries up to 18 June 2022.

Eligibility criteria for selecting studies We included trials studies, compared the results of different researchers on CESM in the diagnosis of breast cancer, and calculated the diagnostic value of CESM for breast cancer.

Data extraction and synthesis Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) evaluated the methodological quality of all the included studies. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses specification. In addition to sensitivity and specificity, other important parameters were explored in an analysis of CESM accuracy for breast cancer diagnosis. For overall accuracy estimation, summary receiver operating characteristic curves were calculated. STATA V.14.0 was used for all analyses.

Results This meta-analysis included a total of 12 studies. According to the summary estimates for CESM in the diagnosis of breast cancer, the pooled sensitivity and specificity were 0.97 (95% CI 0.92 to 0.98) and 0.76 (95% CI 0.64 to 0.85), respectively. Positive likelihood ratio was 4.03 (95% CI 2.65 to 6.11), negative likelihood ratio was 0.05 (95% CI 0.02 to 0.09) and the diagnostic odds ratio was 89.49 (95% CI 45.78 to 174.92). Moreover, there was a 0.95 area under the curve.

Conclusions The CESM has high sensitivity and good specificity when it comes to evaluating breast cancer, particularly in women with dense breasts. Thus, provide more information for clinical diagnosis and treatment.

breast imaging
breast tumours
diagnostic radiology

Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjopen-2022-069788

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

This systematic review was a comprehensive search of experimental and observational studies on contrast-enhanced spectral mammography (CESM) in the diagnosis of breast cancer.

We included only prospective studies. Prospective studies were of higher quality with less bias, and our study screening criteria were developed prior to the meta-analysis.

The study was conducted by two people and was strictly based on inclusion criteria.

The data in this study were summarised using sound statistical methods.

A recent literature was added, and a literature from the same institution included only the most recent or the largest sample size.

We summarised the sensitivity and specificity of CESM in the diagnosis of breast cancer.

Introduction

Globally, female breast cancer has overtaken lung cancer as the leading cause of cancer death, making it the fifth most common cause of death. 1 From the mid-20th century, the incidence of breast cancer in women has been increasing slowly by about 0.5% per year. 2 At present, the diagnostic methods of breast cancer include MRI, full field digital mammography (FFDM) and ultrasound (US). MRI is the most sensitive examination in the diagnosis of breast cancer at present. 3 However, it has some disadvantages such as no claustrophobic and high price. In addition, although FFDM is an effective diagnostic method for breast cancer, it also has the hazard of recall and needs further testing. 4 Ultrasonography has good diagnostic efficacy for breast cancer, especially in women with dense breasts; however, it has a relatively low positive predictive value. 5 Contrast-enhanced spectral mammography (CESM), which visualises breast neovascularisation in a manner similar to MRI, is an emerging technology that uses iodine contrast agent. 6 CESM has the advantages of patient friendliness and low cost. Previous studies have shown that CESM has obvious advantages in displaying lesions compared with US. The advantage of CESM is that it can show changes in anatomy and local blood perfusion, which may be caused by tumour angiogenesis. 7 Moreover, CESM is useful in detecting the suspicious findings in routine breast imaging 7 and the sensitivity and specificity of CESM are different in different studies.

It has been reported that several meta-analyses have been conducted regarding the diagnostic performance of CESM for breast cancer; however, their pooled results were different and had several limitations. 8–11 On the one hand, the sensitivity and specificity differed across the above-mentioned meta-analyses. 8 10 11 On the other hand, the numbers of included studies were limited. In addition, partial meta-analyses included none-English studies and overlapped studies, which might affect their pooled results. In the past few years, several studies evaluating the diagnostic value of CESM in breast cancer have been published. Therefore, we conducted this meta-analysis using available evidence to comprehensively determine whether CESM is effective in detecting breast cancer in women.

Material and methods

As recommended by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), we conducted our study followed the PRISMA specification, 12 which met the requirements of diagnostic systematic review.

Search strategy

To evaluate the accuracy of CESM in diagnosing breast cancer, we retrieved the following databases: PubMed, Embase and Cochrane library. Two reviewers, JL and RX, independently searched the above databases up to the date of 18 June 2022. Our searching terms included ‘contrast-enhanced spectral mammography’, ‘Dual-Energy Contrast-Enhanced Spectral Mammography’, ‘CESM’, ‘contrast-enhanced digital mammography’, ‘CEDM’, ‘Breast Neoplasms’, ‘Breast Neoplasm’, ‘Breast Tumor’, ‘Breast Tumors’, ‘Breast Cancer’, ‘Malignant Neoplasm of Breast’, ‘Breast Malignant Neoplasm’, ‘Breast Carcinomas’, ‘Breast Carcinoma’, ‘breast mass’, ‘breast lesion’, ‘breast lesions’, ‘breast diseases’. In addition, the references of all the included studies were also reviewed.

Inclusion and exclusion criteria

Following is the list of inclusion criteria: (1) studies diagnosing breast cancer, (2) studies provided data on the sensitivity and specificity, (3) studies involving ≥10 patients or case, (4) English language and(5) prospective studies. Following is the list of exclusion criteria: (1) overlapped research, (2) commentaries, letters, editorials or abstracts or (3) studies referencing artificial intelligence and radiomics.

Study screening

The titles and abstracts of the literature in the electronic databases were initially screened by two authors, following the above criteria for inclusion and exclusion. Each of the two researchers screened two times to avoid omission. If there is any disagreement, the third author was consulted to decide. Eligibly downloaded full texts and further screened. First, if the authors and institutions of the study are the same, we will include the most recently published studies with the largest sample size. If the research institutions are the same, but the authors are different, we will send an email to the corresponding authors to ask. If we do not receive a reply, we will include the most recently published studies having the largest sample size.

Data abstraction

Two reviewers extracted data. If necessary, the difference shall be solved by the third reviewer. Each study was analysed for the following information: first author name, publication year, country, the numbers of patients and lesions, median age, the results of true positive (TP), false positive (FP), false negative (FN) and true negative (TN).

Quality assessment

The quality of the methodology included in the publication was assessed by the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2). 13 QUADAS-2 were mainly focused on the following four domains: patient selection, index test, reference standard and flow and timing, with minimal overlapping, which present the main quality of the diagnostic study. Each domain is assessed according to risk of bias, with the three domains assessed according to applicability. The risk of bias was considered low if the study met the above criteria and high otherwise. Disagreements between the two reviewers on quality assessment were resolved by consensus.

Statistical analysis

STATA V.14.0 was used for all analyses. I 2 measure was used to quantify the heterogeneity between studies. If there is no statistical heterogeneity, the fixed effect model is used to consolidate the data. On the contrary, the random effect model is used to summarise the data. The sensitivity was shown in the form TP/(TP+FN), where TP represents the number of true-positive results and FN represent the number of FN results. The specificity was shown in the form TN/(TN+FP), where TN represent the number of TN results and FP represent the number of FN results. 14 We also computed other significant measures on the evaluation of diagnostic experiments such as positive likelihood ratio (PLR) and negative likelihood ratio (NLR) and diagnostic OR (DOR). The summary receiver operating characteristic curve ROC (SROC) curve and the area under the curve (AUC) of the SROC curve were also computed.

Study characteristics

After a systematic search, we included 12 studies. 15–26 The complete selection process is in detail in PRISMA flowchart ( figure 1 ). From 544 screened studies, 85 studies were subjected to full text reading. The characteristics of all the 12 included studies are shown in table 1 . These 12 studies are all prospective studies published between 2014 and 2022. Most patients had US, mammography and related examinations before CESM examination. The dense breast we collected account for approximately two-thirds. In addition, the methodological quality assessment of all included studies was shown in online supplemental table 1 .

Supplemental material

View inline

Study characteristics of each included study

Download figure
Open in new tab
Download powerpoint

The figure shows the workflow for study screening and selection. CESM, contrast-enhanced spectral mammography.

Diagnostic accuracy of CESM

The sensitivity and specificity values were shown in Forest plots ( figure 2 ). A very high pooled test sensitivity of 0.97 (95% CI 0.92 to 0.98) was estimated. The pooled specificity was 0.76 (95% CI 0.64 to 0.85). The PLR was 4.03 (95% CI 2.65 to 6.11), NLR was 0.05 (95% CI 0.02 to 0.09) ( figure 3 ) and DOR was 89.49 (95% CI 45.78 to 174.92) ( online supplemental figure 1 ). I 2 values of sensitivity, specificity, PLR, NLR and DOR were 76.60%, 87.95%, 86.25%, 65.73% and 99.78%, respectively.

Forest plot of estimates of sensitivity and specificity for contrast-enhanced spectral mammography in the diagnosis of breast cancer.

Forest plot of estimates of positive likelihood ratio and negative likelihood ratio for contrast-enhanced spectral mammography in the diagnosis of breast cancer.

As shown in figure 4 , the SROC curve shows an AUC of 0.95 (0.93 to 0.97). CI is an interval estimation based on the average point estimation. The prediction interval is the interval estimation based on the individual value point estimation.

The plot shows the summary bivariate ROC curve for CESM diagnostic accuracy. AUC, area under the curve; CESM, contrast-enhanced spectral mammography; ROC, receiver operating characteristic curve; SENS, sensitivity; SPEC, specificity; SROC, summary receiver operating characteristic curve.

A confidence contour and a prediction contour were shown in the figure.

Fagan plots were drawn to understand the prior probability (current incidence) and the posterior probability (incidence estimated from this diagnostic experiment). In our sample, the pretest probability of malignancy was 50%, with a positive finding at CESM a post-test probability of 80% while a negative finding a post-test probability of 4% ( online supplemental figure 2 ).

Regression analysis

We analysed some covariates (number of lesions, number of patients, being dense breast or not, year of publication) possible influence on the diagnostic accuracy of CESM. The regression analysis showed that the sensitivity of the studies that only included dense breast was different from that of other studies, but both were high ( online supplemental figure 3 ). In addition, a limited number of studies were included, which reduced the reliability of the regression analysis.

Publication bias

A funnel plot drawn with Stata V.14.0 software was used to analyse the publication bias of the included studies ( online supplemental figure 4 ). The included studies were evenly distributed on both sides of the regression line, showing that the included literatures had no obvious publication bias (p=0.78).

CESM is emerging as a valuable tool for the diagnosis and staging of breast cancer. CESM combines the contrast enhancement effect caused by tumour neovascularisation with the information of anatomical changes. The lesions were highlighted by reciprocal subtraction of the images, which further increased the sensitivity of CESM for the diagnosis of breast cancer. It improves the accuracy in diagnosing breast cancer, providing more accurate tumour size and identification of multifocal disease, especially in patients with the dense type of breast. 27

Results showed that the pooled sensitivity (0.97, 95% CI 0.92 to 0.98) was higher and the pooled specificity (0.76, 95% CI 0.64 to 0.85) was slightly lower than a previous meta-analysis 9 which indicated a pooled sensitivity of 0.89 (95% CI 0.88 to 0.91) and a pooled specificity of 0.84 (95% CI 0.82 to 0.85). The reason for the high sensitivity may be that our study went through more rigorous study screening, included the latest literature, and CESM has been increasingly used in clinical practice in recent years. Another point is that all the studies we included are prospective studies, which are less susceptible to bias than retrospective studies. Another previous meta-analysis 8 has obtained that CESM has high sensitivity for the diagnosis of breast cancer, but it has low specificity. This may be due to the following reasons: three studies included by the meta-analysis were similar and written by the same first author; the meta-analysis only included eight studies and the pooled specificity were obtained by six literatures. All the reasons may result in some bias. However, during our screening, there are five studies from the same authors 15 28–31 and with similar results, we only included one in which the study type was prospective and with large sample size and longest time span.

In addition, compared with other studies, this study included the latest studies in recent years, and conducted a more rigorous article screening, with each of the two researchers screening two times.

The DOR is a common statistic in epidemiology that expresses the strength of the association between exposure and disease. 32 The diagnostic DOR for a test is the ratio of the odds of being positive in the disease to the odds of being positive in the non-disease. In our meta-analysis, the DOR was 89.49 (95% CI45.78 to 174.92), which was high. It indicated that if CESM showed a positive result, the probability of a true breast cancer being correctly diagnosed was 89.49 to 1. DOR offers considerable advantages in a meta-analysis of diagnostic studies by combining results from different studies into a more precise pooled estimate. The I 2 statistic, also known as the inconsistency index, is a measure of heterogeneity or variability across studies in a meta-analysis. It quantifies the proportion of total variation in effect estimates that is due to heterogeneity rather than chance. Differences in study populations: the studies included in the meta-analysis may have varied in terms of patient characteristics, such as age, mammary gland type, disease severity or comorbidities. These differences can contribute to heterogeneity in the estimated DOR. Clinical and contextual factors: heterogeneity in DOR can also arise from differences in the clinical context, such as variations in disease prevalence, healthcare settings or geographic locations.

The SROC curve method takes into account the possible heterogeneity of thresholds. 33 The SROC indicates the relationship between the TP rate and FP rate at different diagnostic thresholds. 34 In general, the AUC of a diagnostic method between 0.5 and 0.7 means low accuracy, 0.7 and 0.9 means good accuracy, above 0.9 high accuracy. The SROC curve shows an AUC of 0.95, indicating high accuracy.

The study of Hobbs et al 35 reminds of that patients’ preferences for CESM will provide further evidence supporting the adoption of CESM as an alternative to ce-MRI in selected clinical indications, if diagnostic non-inferiority of CESM is confirmed. Ferranti et al 25 suggested that CESM may provide compensation for MRI through a slight FN tendency. Furthermore, Clauser et al 36 thought the specificity of CESM is higher than that of MRI. CEM determines breast cancer based on tumour angiogenesis assessment. 24 Growth factors secreted by cancer cells promote the formation of new blood vessels during division and proliferate to tumour cells. It is because of the increased vascular endothelial cell gap and permeability that the contrast in the tumour area is enhanced. CESM may combine the high sensitivity of MRI with the low cost and availability of FFDM. 37

However, there are some limitations in the study. First, primary source participants were all patients with lesions diagnosed by breast US or mammography. This may induce a selection bias. Second, the majority of the main participants were with dense breast. This point, while highlighting the superiority of CESM over dense breast examination, may still be subject to some bias. Third, due to the excessive number of retrieved literatures, we only included prospective studies and studies writing in English. In this way, some reliable studies and results may be missed.

The CESM has high sensitivity and good specificity when it comes to evaluating breast cancer, particularly in women with dense breasts. Thus, provide more information for clinical diagnosis and treatment.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

Siegel RL , et al
Siegel RL ,
Miller KD ,
Fuchs HE , et al
Poortmans P ,
Morrow M , et al
Lobbes MBI ,
Jochelson MS ,
Tagliafico AS ,
Bignotti B ,
Rossi F , et al
Huang J-M ,
Zhang K , et al
Pesapane F ,
Agazzi GM , et al
McInnes MDF ,
Thombs BD , et al
Whiting PF ,
Rutjes AWS ,
Westwood ME , et al
Psoter KJ ,
Roudsari BS ,
Dighe MK , et al
Luczyńska E ,
Heinze-Paluchowska S ,
Dyczek S , et al
Mokhtar O ,
Wang L , et al
Akashi-Tanaka S ,
Suzuki S , et al
Sun B , et al
El Ghany EA
Pan Y , et al
Petrillo A ,
Vallone P , et al
Mohamed SAS ,
Moftah SG ,
Chalabi NAEM , et al
Sannapareddy K ,
Potlapalli A , et al
Ferranti FR ,
Vasselli F ,
Barba M , et al
Abu Samra MF ,
Ibraheem MA , et al
Adamczyk A , et al
Łuczyńska E ,
Hendrick E , et al
Luczynska E ,
Niemiec J ,
Ambicka A , et al
Lijmer JG ,
Prins MH , et al
Takwoingi Y ,
Riley RD , et al
Taylor DB ,
Buzynski S , et al
Clauser P ,
Baltzer PAT ,
Kapetas P , et al
Bicchierai G ,
Tonelli P ,
Piacenti A , et al

Contributors JL and RX designed the study. SZou and YH gathered data. JL and SZhen performed the analysis. HY and DH revised it critically for important intellectual content. DH acted as guarantor. All authors contributed to the article and approved the submitted version.

Funding This work has received funding by the Henan Medical Science and Technology Research Program (LHGJ20210498,LHGJ20230528).

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

Ohio State nav bar

The Ohio State University

BuckeyeLink
Find People
Search Ohio State

How Covidence Helps Streamline Systematic Reviews

Extended Library Access is available at the HSL!
Dr. Teresa Long Papers: Discover Recent History of Public Health in the Archives
Hidden Gems of the Health Sciences Library
Art by HSL Medical Illustrator Featured on Medical Journal Cover
HSL Medical Illustrators’ Art on Display in IHSC

Author: Kerry Dhakal, MAA, MLS

Covidence is a systematic review screening tool used to streamline the process of screening, reviewing, analyzing, synthesizing and disseminating data you collect from reviewed articles. The main point of conducting systematic reviews is to analyze the findings across studies to answer a clinical question. This process is a time-consuming team process, so having access to and the use of a tool that can simplify some of the steps of the process is extremely helpful.

Covidence has a few features that people really find helpful: deduplication of imported citations, automatic population of the PRISMA flowsheet and the ability to export lists of articles and data into different formats for analysis.

De-duplication citations

When conducting a systematic review, team members will be searching for articles across multiple databases, as the goal of a systematic review is to find all possible studies that potentially answer the clinical question of the review. When importing citation information from each database (i.e., PubMed, CINAHL and EMBASE), Covidence will combine all the individual databases’ citation information to make a full list of all citations imported. Some citations imported will be duplicates, as they are included in possibly more than one database and Covidence, when it recognizes a duplicate citation, will automatically remove it, that is, de-duplicate the list of citations. This is an important step because of the need to show in a PRISMA flowsheet how the literature search was conducted in a systematic review.

Automatic population of the PRISMA flowsheet

One tool used for the documentation of the literature search steps is the PRISMA flowsheet. Luckily for Covidence users, the PRISMA flowsheet is a widget in Covidence that automatically gets populated with information when the team conducts the citation importing, screening and full-text review steps. This saves teams time and effort to create the flowsheet manually.

Exporting article and data information

The PRISMA flowsheet, lists of articles included in screening and full-text review steps and data extracted through Covidence templates can all be exported into other document types for analysis and publication. One example of this is the ability to export data extracted in either Covidence-created or -customized extraction templates into CSV or Excel files. This makes all data available in one spreadsheet to review and find themes and patterns in the data versus having to acquire and document this information manually.

Diving deeper with Covidence

Would you like to learn more about using Covidence? Covidence has its own YouTube channel with videos about using this tool: Covidence - YouTube . Covidence answers some of the most frequently asked questions (FAQs) in its help section found here: Covidence Knowledge Base .

You can also reach out to your liaison librarian at the Health Sciences Library with questions about using Covidence for systematic review assignments and projects: Research & Education | Health Sciences Library (osu.edu) .

Your Contact

Kerry Dhakal, MAA, MLS

Prior Hall 208

Download PDF
CME & MOC
Share X Facebook Email LinkedIn
Permissions

Causes and Clinical Presentation of Drug-Induced Dermatomyositis : A Systematic Review

1 Department of Dermatology, Stanford University School of Medicine, Redwood City, California

Question What are the most common medication causes, clinical presentations, and laboratory findings for drug-induced dermatomyositis (DM)?

Findings In this systematic review of 134 studies reporting data on 165 cases of drug-induced DM, drug-induced DM was found to occur after treatment with hydroxyurea, immune checkpoint inhibitors, statins and lipid-lowering agents, penicillamine, and tumor necrosis factor inhibitors.

Meaning These results can help inform disease-specific guidelines for patients with drug-induced DM.

Importance While several medications are known to induce dermatomyositis (DM), most existing studies are case reports or small case series from a single institution. There is also limited information on DM induced by immune checkpoint inhibitors, which are increasingly used in oncologic therapy.

Objective To characterize causes and clinical presentation of drug-induced DM based on the current literature.

Evidence Review A systematic review was performed in PubMed according to Preferred Reporting Items for Systematic Reviews and Meta-analyses ( PRISMA ) reporting guidelines, from inception to August 22, 2022. Articles meeting preestablished inclusion criteria (written in English and classified as original articles, case reports, literature reviews, and observation letters) were selected and data abstracted. Articles that met the scope of the review were also added from reference lists. When possible, study results were quantitatively combined.

Findings In 134 studies (114 from the literature search and 20 additional studies pulled from reference lists) describing 165 cases, 88 patients (53.3%) were female, and the median (IQR) age was 61 (49-69) years. Among the cases of drug-induced DM, the most common associated medications were hydroxyurea (50 [30.3%]), immune checkpoint inhibitors (27 [16.4%]), statins (22 [13.3%]), penicillamine (10 [6.1%]), and tumor necrosis factor inhibitors (10 [6.1%]). Histopathologic testing, when undertaken, helped establish the diagnosis. There was a median (IQR) of 60 (21-288) days between drug initiation and drug-induced DM onset. History of cancer was reported in 85 cases (51.6%).

Conclusions and Relevance In this systematic review, drug-induced DM was associated with multiple types of medications, including chemotherapies and immunotherapies. It is essential that dermatologists promptly recognize and diagnose drug-induced DM so that they can guide management to minimize interruption of therapy when possible.

Manage citations:

Artificial Intelligence Resource Center

Dermatology in JAMA : Read the Latest

Browse and subscribe to JAMA Network podcasts!

Others Also Liked

Select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

Academic Medicine
Acid Base, Electrolytes, Fluids
Allergy and Clinical Immunology
American Indian or Alaska Natives
Anesthesiology
Anticoagulation
Art and Images in Psychiatry
Artificial Intelligence
Assisted Reproduction
Bleeding and Transfusion
Caring for the Critically Ill Patient
Challenges in Clinical Electrocardiography
Climate and Health
Climate Change
Clinical Challenge
Clinical Decision Support
Clinical Implications of Basic Neuroscience
Clinical Pharmacy and Pharmacology
Complementary and Alternative Medicine
Consensus Statements
Coronavirus (COVID-19)
Critical Care Medicine
Cultural Competency
Dental Medicine
Dermatology
Diabetes and Endocrinology
Diagnostic Test Interpretation
Drug Development
Electronic Health Records
Emergency Medicine
End of Life, Hospice, Palliative Care
Environmental Health
Equity, Diversity, and Inclusion
Facial Plastic Surgery
Gastroenterology and Hepatology
Genetics and Genomics
Genomics and Precision Health
Global Health
Guide to Statistics and Methods
Hair Disorders
Health Care Delivery Models
Health Care Economics, Insurance, Payment
Health Care Quality
Health Care Reform
Health Care Safety
Health Care Workforce
Health Disparities
Health Inequities
Health Policy
Health Systems Science
History of Medicine
Hypertension
Images in Neurology
Implementation Science
Infectious Diseases
Innovations in Health Care Delivery
JAMA Infographic
Law and Medicine
Leading Change
Less is More
LGBTQIA Medicine
Lifestyle Behaviors
Medical Coding
Medical Devices and Equipment
Medical Education
Medical Education and Training
Medical Journals and Publishing
Mobile Health and Telemedicine
Narrative Medicine
Neuroscience and Psychiatry
Notable Notes
Nutrition, Obesity, Exercise
Obstetrics and Gynecology
Occupational Health
Ophthalmology
Orthopedics
Otolaryngology
Pain Medicine
Palliative Care
Pathology and Laboratory Medicine
Patient Care
Patient Information
Performance Improvement
Performance Measures
Perioperative Care and Consultation
Pharmacoeconomics
Pharmacoepidemiology
Pharmacogenetics
Pharmacy and Clinical Pharmacology
Physical Medicine and Rehabilitation
Physical Therapy
Physician Leadership
Population Health
Primary Care
Professional Well-being
Professionalism
Psychiatry and Behavioral Health
Public Health
Pulmonary Medicine
Regulatory Agencies
Reproductive Health
Research, Methods, Statistics
Resuscitation
Rheumatology
Risk Management
Scientific Discovery and the Future of Medicine
Shared Decision Making and Communication
Sleep Medicine
Sports Medicine
Stem Cell Transplantation
Substance Use and Addiction Medicine
Surgical Innovation
Surgical Pearls
Teachable Moment
Technology and Finance
The Art of JAMA
The Arts and Medicine
The Rational Clinical Examination
Tobacco and e-Cigarettes
Translational Medicine
Trauma and Injury
Treatment Adherence
Ultrasonography
Users' Guide to the Medical Literature
Vaccination
Venous Thromboembolism
Veterans Health
Women's Health
Workflow and Process
Wound Care, Infection, Healing
Register for email alerts with links to free full-text articles
Access PDFs of free articles
Manage your interests
Save searches and receive search alerts

Open access
Published: 02 September 2024

The effect of mode of delivery on postpartum comfort level and breastfeeding self-efficacy: a systematic review and meta-analysis

Hava Özkan ORCID: orcid.org/0000-0001-7314-0934 1 ,
Elif Odabaşi Aktaş ORCID: orcid.org/0000-0002-3435-7118 2 &
Hafsa Kübra Işik ORCID: orcid.org/0000-0002-2472-6987 2

Maternal Health, Neonatology and Perinatology volume 10 , Article number: 17 ( 2024 ) Cite this article

1 Altmetric

Metrics details

This study was conducted to determine the effect of the mode of delivery on maternal postpartum comfort level and breastfeeding self-efficacy.

The study was conducted as a systematic review and a meta-analysis. Searching was performed from March to July 2022, on PubMed, National Thesis Center, Dergi Park, Google Scholar, Web of Science, and EBSCO search engines and we included studies from the last 10 years. The Joanna Briggs Institute Critical Appraisal tools used in cross-sectional studies were employed to appraise the methodological quality and performed meta-analyses using a random-effects model for all outcomes. Study data consisted of continuous variables calculated by Mean Difference.

From 3732 records received, 21 cross-sectional studies involving 5266 participants were determined to be eligible. Meta-analysis results showed that cesarean section reduced postpartum comfort, albeit not statistically significant (MD: -0.87 95%: -1.98–0.24, Z = 1.53, p = 0.44), whereas the combined results of breastfeeding self-efficacy showed that delivery type did not affect breastfeeding self-efficacy.

The results of this review have clinical implications for postpartum caregivers, as the effects of mode of delivery on postpartum comfort and breastfeeding self-efficacy have been well documented in previous studies. The authors recommend caregivers plan maternal care to increase their comfort, taking into account the factors that may affect postpartum comfort in the light of evidence-based practices.

Introduction

Childbirth is one of the most critical stages in a woman's life and often marks the transition to a new life [ 1 ]. One of the essential factors affecting the healthy progress of this shift is delivery type, which can depend on maternal obstetric characteristics, individual preferences, and advice from family, friends, or the doctor [ 2 ]. Mode of delivery, i.e., vaginal or cesarean delivery, is an essential factor affecting the birth and postpartum period. With proper guidance during labor, the most physiologically appropriate mode of delivery for the female body is vaginal delivery. Nevertheless, cesarean delivery is performed in cases where work does not progress as it should, hence putting the mother and the fetus at risk [ 2 , 3 ]. Whether by Cesarean section or vaginal birth, postpartum is the beginning of a new period in the interaction between mother and infant. It comes with a series of psychosocial changes, adapting to which can create considerable anxiety and stress. Providing comfort to the mother is very important in facilitating her adaptation to this new period of life [ 4 ].

Kolcaba developed her comfort theory in 2003, which defined comfort as ‘the immediate experience of being strengthened through having the needs for relief, ease, or transcendence met in four contexts: environmental, social, psycho-spiritual and physical [ 5 ]. Postpartum comfort is essential for women to pinpoint and solve the problems they face during that period [ 6 ]. Midwives play a crucial role in solving the factors affecting postpartum comfort and helping mothers adapt to the postpartum period. Comfort-oriented care by midwives contributes to the individual’s quality of life and promotes quick mother-infant bonding, postpartum care, and satisfaction [ 7 ].

Quick mother-infant bonding can be facilitated through comfort and mode of delivery. A critical factor in mother-infant adhesion and the formation of a healthy society is breastfeeding [ 8 ]. According to UNICEF (UN International Children’s Emergency Fund), breastfeeding gives all children the most nutritious start. It leads to lower healthcare costs, healthier families, and a more competent workforce [ 9 ]. To promote breastfeeding, WHO (World Health Organization) notes that breastfeeding acts as a baby’s first vaccine after birth and recommends that babies be exclusively breastfed for the first six months of life and continue to be breastfed for up to 2 years and beyond [ 10 ]. Many factors such as the mother's age, education, employment and economic status, smoking, family structure, frequency of prenatal follow-up, breast milk and breastfeeding education, number of living children and births, skin-to-skin contact, postpartum depression, birth weight and birth type affects breastfeeding self-efficacy [ 6 ].

While, in general terms, the perception of self-efficacy plays a significant role in the activities that an individual will or will not do, the perception of breastfeeding self-efficacy is a mother’s confidence in her ability to breastfeed her new infant and, above all, her interest and desire for breastfeeding. High breastfeeding self-efficacy has been positively associated with the duration and exclusivity of breastfeeding [ 11 ]. Mode of delivery and perception of breastfeeding self-efficacy affect postpartum breastfeeding [ 11 , 12 ]. Some studies in the literature state those women who gave birth by cesarean delivery started breastfeeding later than those who had a vaginal delivery [ 13 , 14 ]. No systematic review or meta-analysis on the subject was found in the literature review. This study, which will be carried out due to the gap in the literature, will contribute to the field.

This systematic review and meta-analysis study aimed to determine the effect of the mode of delivery on maternal postpartum comfort level and breastfeeding self-efficacy based on primary studies. The questions that the researchers sought to answer were:

How does the mode of delivery affect maternal postpartum comfort level?

How does the mode of delivery affect maternal breastfeeding self-efficacy level?

Protocol and registration

This study is a systematic review and meta-analysis, which was performed according to the standard guideline of “Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)” [ 15 ]. This systematic review and meta-analysis protocol was registered in the PROSPERO database (Trial Registration Number: CRD42020191106; 26 May 2022). PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analysis) protocols were followed for review and reporting [ 15 ].

Search strategy

Data base ( www.crd.york.ac.uk/prospero ) was explored to confirm whether systematic review or meta-analysis existed before. Two researchers, independently between March and July 2022 on PubMed, National Thesis Center, Dergi Park, Google Scholar, Medline, Web of Science, and EBSCO search engines, conducted searching within the scope of this systematic review and meta-analysis. The search was carried out using the following keywords or medical subject headings: “mode of delivery” AND “cesarean section” OR “cesarean delivery” AND “vaginal birth” OR “vaginal delivery” AND “postpartum comfort” AND “breastfeeding self-efficacy”. In addition, additional studies were independently checked against included articles by the other two authors. Using the snowball method, reference lists of included studies and previous systematic reviews were checked for additional studies. To eliminate the risk of bias in the study, two researchers searched, screened, extracted data, and appraised the articles included independently. The researchers discussed and resolved any differences of opinion arising on any aspect of the study. For discrepancies, we screened with the help of a third reviewer. Before starting the study, a pilot study was conducted with the participation of all authors agreeing on a joint roadmap, including all stages of research. We used Cohen kappa (K = 0.9) to assess inter–rater agreement.

Inclusion and exclusion criteria

The criteria for the studies to be included in this review were as follows:

Participant groups included mothers' mode of delivery, postpartum comfort levels, and breastfeeding self-efficacy levels.

Attention was paid to the fact that the studies were full-text articles published in Turkish and English between May 2012 and July 2022.

A clinical trial design was used, including cross-sectional, analytical cross-sectional, and comparative cross-sectional studies.

PECOS criteria were used to screen the eligible studies for this systematic review and meta-analysis (Table 1 ). Exclusion criteria;

Delivery type, postpartum comfort levels, and breastfeeding self-efficacy levels.

The full text was not available.

The authors could not be reached and reported unextractable or irrelevant raw data.

Published in English or other than Turkish.

Reviews, editorials, books, news, etc. were.

The number of studies searched for systematic review, the number of studies found eligible and included in the review, and the number of studies excluded and the reasons for exclusion are shown in Fig. 1 , in PRISMA flow diagram format.

PRISMA flow diagram

Data extraction

All articles were imported to a citation manager (Mendeley), and duplicates were removed. Two trained investigators (the first and second authors of this paper) searched the databases and independently screened the titles and abstracts. After searching, filtering by title and abstract, and removing duplicates, the authors gathered and decided on eligible studies suitable for analysis on a full-text basis. Later on, some of the studies had to be excluded from the study during the analysis, as the data they contained was unsuitable. The number of studies searched for systematic review, the number of studies found eligible and included in the review, and the number of studies excluded and the reasons for exclusion are shown in Fig. 1 , in PRISMA flow diagram format.

The researchers designed a data extraction tool (Table 2 ) to acquire study data (Table 2 ). The data extraction tool made it possible to collect data about the studies included in systematic review and meta-analysis (i.e., author details, place and year of publication, data collection dates, design, scale used, sample size, mean maternal age, and other main findings reported) (see Table 2 ).

Assessment of methodological quality

The Joanna Briggs Institute Appraisal tools for cross-sectional studies were used for the studies’ quality appraisal. The checklist consisted of eight items [ 30 ]. Each item was rated as Yes, No, Unclear or Not Applicable. Appraisal results are given in Table 2 , with the total number of items (number of Yes responses) accepted as the studies’ “Quality Score”. Quality appraisal scores for cross-sectional studies were found to be 6/8 yes in 21 studies (Table 2 ). The vast majority of the studies reviewed met the quality appraisal criteria, which represents a low risk of bias.

Data analysis

The data in this systematic review were synthesized by meta-analysis. Study data in the postpartum comfort section ( n = 10) were synthesized by meta-analysis. In contrast, the data on breastfeeding self-efficacy ( n = 11) were explained in tabulated form as the studies examined as part of this study found that the mode of delivery did not affect maternal breastfeeding self-efficacy scores (Table 2 ). Data obtained from cross-sectional, cross-sectional-analytical, and cross-sectional-comparative studies were synthesized by meta-analysis. There is no missing data in the study. Review Manager Version 5.4 was used for meta-analysis, and the CMA program was used for regression testing. The extent of heterogeneity in the studies was measured by Cochran’s Q and Higgins’ I 2 tests, and it was accepted that I 2 above 50% was an important indicator of heterogeneity. Accordingly, “Random Effect” results were considered when I 2 was greater than 50%, and “Fixed Effect” results if it was below. Study data consisted of continuous variables calculated by “Mean Difference.” All tests were calculated as two-tailed, and p < 0.05 was considered statistically significant.

Study selection

After subject selection, a total of 3736 records were identified, therefrom 3732 records were found through database searching and four additional records through other sources. After removal of duplicates, the records were screened by title and abstract, and as a result, the full texts of 41 articles were accessed. These 41 full-text articles were assessed for eligibility, and as a result, 21 articles were included in the study in Fig. 1 .

Study and participant characteristics

Of the studies included in systematic review, 16 were cross-sectional, two was analytical-cross- sectional, and three were comparative cross-sectional studies [ 3 , 4 , 6 , 7 , 8 , 11 , 12 , 16 , 29 ]. The studies had been conducted in 12 different cities of Turkey: Konya (3), Ankara (1), Izmir (3), Erzurum (3), Antalya (1), Istanbul (3), Çankırı (1), Diyarbakir (1), Trabzon (1), Malatya (1), Muğla (1) and Aydın (1). Total sample size of the studies was 5266 (postpartum comfort sample group: 3847; breastfeeding self- efficacy sample group: 1429) (Table 1 ). Of the studies ( n = 21), 3 had been published in English and 19 in Turkish. Systematic review of the studies showed that, in general terms, the mean age of postpartum women was between 23 and 28 years, while nine studies specified no mean age. It was determined that the studies included in the review and analysis were published between 2014–2022. The data collection tools, sample size and quality score of the studies and the age and group characteristics of participants are presented in Table 2 .

Postpartum comfort

In this systematic review and meta-analysis, ten studies [ 3 , 4 , 6 , 7 , 8 , 16 , 17 , 18 , 19 , 20 ] were found to report results on the effect of mode of delivery on maternal postpartum comfort, as assessed by means of a Postpartum Comfort Questionnaire. Combined results of these studies showed that delivery by cesarean delivery reduced postpartum comfort, while in meta-analysis the outcome was found to be statistically insignificant (MD:-1.63, 95%: -5.76- 2.50, Z = 0.77, p = 0.44, Fig. 2 ). Estimated heterogeneity as measured by I 2 was 96% ( p < 0.00001), and asymmetric outliers were seen in funnel plot (Figs. 2 and 3 ). Egger's regression test showed that the effect of publication bias was small (t: 1, 92, p = 0,090).

Forest plot of the effect of mode of delivery on postpartum comfort level

Funnel plot of the effect of mode of delivery on postpartum comfort level

Postpartum comfort sub-dimensions

Physical comfort sub-dimension.

Of the studies examined in this systematic review and meta- analysis, ten studies [ 3 , 4 , 6 , 7 , 8 , 16 , 17 , 18 , 19 , 20 ] reported results on the effect of mode of delivery on the physical sub- dimension of maternal postpartum comfort. Combined results of these studies showed that cesarean delivery reduced physical postpartum comfort, while in meta-analysis the outcome was found to be statistically insignificant (MD: -0,97, 95%: -4,11- 2.17, Z = 0.61, p = 0.54, Fig. 4 ).

Forest plot of the effect of mode of delivery on postpartum comfort sub-dimensions level

Psycho-spiritual comfort sub-dimension

Of the studies examined in this systematic review and meta-analysis, ten studies [ 3 , 4 , 6 , 7 , 8 , 16 , 17 , 18 , 19 , 20 ] reported results on the effect of mode of delivery on the psycho- spiritual sub-dimension of maternal postpartum comfort. Combined results of these studies showed that vaginal delivery increased psycho-spiritual postpartum comfort, while in meta-analysis the outcome was found to be statistically insignificant (MD: 0.47 95%: -1.13–2.08, Z = 0.58, p = 0.57, Fig. 4 ).

Socio-cultural comfort sub-dimension

Of the studies examined in this systematic review and meta-analysis, ten studies [ 3 , 4 , 6 , 7 , 8 , 16 , 17 , 18 , 19 , 20 ] reported results on the effect of mode of delivery on the socio-cultural sub-dimension of maternal postpartum comfort. Combined results of these studies showed that cesarean delivery reduced sociocultural postpartum comfort, while in meta-analysis the outcome was found to be statistically insignificant (MD: -0.87 95%: -1.98–0.24, Z = 1.53, p = 0.13, Fig. 4 ).

Breastfeeding self-efficacy

In the 11 studies [ 11 , 12 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ] examined in this systematic review in terms of breastfeeding self- efficacy, one study found a significant difference between delivery mode and breastfeeding self- efficacy, but no significant difference was found in the remaining ten. When we look at the results of the said one study conducted by Işık et al. [ 29 ], in which they found that the mode of delivery affected breastfeeding self-efficacy, it can be seen that vaginal delivery was found to affect breastfeeding self- efficacy 24 h postpartum, but not by week 4. Since most of the studies included in this systematic review did not have any effect on breastfeeding self-efficacy and there were no homogeneous data, no meta-analysis was conducted on this subject.

This systematic review and meta-analysis present the combined results of 21 cross-sectional studies reporting on postpartum comfort and breastfeeding self-efficacy outcomes by mode of delivery [ 3 , 4 , 6 , 7 , 8 , 11 , 12 , 16 , 29 ]. The results of these studies are important as they can contribute to improving the follow-up and care services offered to postpartum women.

The combined results of the studies examined in the meta-analysis showed that cesarean delivery reduced postpartum comfort, and the outcome was found statistically insignificant. A survey by Şahin and Sinan [ 31 ] reported moderate maternal postpartum comfort, whereas postpartum comfort level after a cesarean section was low. The study shows parallelism with the meta-analysis findings. In this sense, factors affecting maternal postpartum comfort in the event of a cesarean section may include incision pain, nausea, vomiting, pain, breastfeeding problems, and excretory issues, suggesting that maternal comfort level is lower after a cesarean delivery. Midwives should observe the problems mothers face, regardless of their mode of delivery, and should provide adequate care to solve those problems and improve their comfort level.

The combined results of the studies examined in the meta-analysis showed that cesarean delivery reduced physical postpartum comfort, and the outcome was found statistically insignificant. Pınar et al. [ 32 ] also found that maternal physical comfort was higher in women who had given birth vaginally. Another study also determined that physical comfort was more elevated in mothers who had given birth vaginally [ 33 ]. Physical comfort is associated with bodily perceptions, including physiological factors like nutrition, excretion, rest, and hemodynamics that affect the individual's physical condition [ 34 ]. The worse a person's physical condition gets, the lower their level of physical comfort. Since mothers who gave birth by cesarean section were exposed to anesthesia at much higher rates, the side effects of anesthesia and the pain level they experienced suggest that physical comfort is lower in cesarean deliveries compared to mothers who gave vaginal birth.

The combined results of the studies examined in the meta-analysis showed that vaginal delivery increased psycho-spiritual postpartum comfort, and the outcome was found statistically insignificant. Karakaplan [ 33 ] stated in his research that the psycho-spiritual level in women with a vaginal delivery was higher than in those with a cesarean section. Psycho-spiritual comfort covers emotional concepts such as self-esteem, self-concept, and self-awareness, all of which give meaning to an individual's life. The factors affecting psycho-spiritual postpartum comfort include maternal role adaptation, emotional changes in the maternal role, mother-infant interaction, maternal stress, and anxiety [ 7 , 17 ]. Based on the results, it can be said that psycho-spiritual comfort levels are higher in mothers giving birth vaginally, which can be attributed to reasons like the higher level of pain, stress, and anxiety experienced by mothers having a cesarean delivery, along with prolonged maternal role adaptation and delayed mother-infant interaction.

The combined results of the studies examined in the meta-analysis showed that cesarean delivery reduced sociocultural postpartum comfort, and the outcome was found statistically insignificant. Factors that constitute sociocultural comfort include family traditions/customs, religious beliefs, benefiting from financial support, and interpersonal communication [ 5 , 34 ]. In this context, information and counseling should be given to the individuals to create sociocultural care, and it should be ensured that they receive care, social support, discharge, and education, taking into account their family traditions and habits [ 5 , 35 ]. Being hospitalized for delivery in an environment away from home, facing the complex structure of the hospital, being dependent on others in many activities during and after delivery, limitation of movement, and having to adapt to unfamiliar procedures in the hospital environment are all factors that disrupt sociocultural comfort [ 17 ]. Hence, the mode of delivery is one of the most important factors affecting comfort [ 7 ]. According to the results of the meta-analysis, it can be said that the sociocultural comfort of the mother after vaginal delivery is higher than after cesarean section because women who have vaginal delivery have freedom of movement, can take care of themselves, and have a shorter hospital stay.

The studies included in this systematic review, reporting findings on breastfeeding self-efficacy, found that mode of delivery did not affect breastfeeding self-efficacy. Gürol [ 36 ], Aydın, and Aba [ 37 ] said there was no significant difference in breastfeeding self-efficacy among vaginal or cesarean delivery women. In a systematic review, Işık et al. [ 29 ] reported that mothers who had given vaginal birth had a high and significant breastfeeding self-efficacy within 24 h.

According to Dennis, the mother's perception of breastfeeding self-efficacy; shows whether the mother will breastfeed, how much effort she will put into it, her thoughts about breastfeeding, and her ability to cope emotionally with the difficulties she will face during the breastfeeding process. The self-efficacy perceived by the mother regarding breastfeeding may be related to the problems experienced in different situations before [ 38 , 39 ]. Mothers with high self-efficacy encourage themselves in the face of difficulties and try to solve the events by thinking positively [ 40 ]. These mothers prefer to breastfeed more, are more courageous, and act positively when faced with difficulties. In parallel with this study, in a study conducted by Dennis, it was determined that mothers who gave birth by cesarean section had lower breastfeeding self-efficacy scores due to delayed mother-infant interaction. Regardless of the mode of delivery, starting breastfeeding within the first half hour after birth is very important for mother-infant interaction [ 41 ]. It is also concluded that mother-infant interaction is delayed due to postpartum pain, stress, anxiety, and problems adapting to a new life and that, therefore, mothers may tend to prioritize their self-care. Since midwives are direct observers of the breastfeeding behaviors of mothers, they have an active role in solving existing problems and providing breastfeeding counseling. Counseling by midwives is important for solving breastfeeding problems faced by mothers and raising their self-efficacy perceptions.

Strengths and limitations of the study

The strengths of the present study were the scores recorded in the quality appraisal of the studies reviewed in this systematic review and meta-analysis and the wide variety of additional sources available for screening. The large sample size available to review data within the scope of this study was another vital aspect that strengthened the outcomes. Analyzing and comparing the results with other studies supported the analysis even further. The weakness of this review was that the studies included in the systematic review and meta-analysis were conducted only in Türkiye and hence, cannot be generalized to the general population. Since the low homogeneity of the studies reviewed may weaken the strength of evidence, the Random Effect model was used to control this factor in studies with high heterogeneity.

Conclusion and suggestions

In this systematic review and meta-analysis, we combined the results of 21 studies to present comprehensive data on maternal postpartum comfort and breastfeeding self-efficacy by mode of delivery. Because of the review, it was seen that the mode of delivery did not affect breastfeeding self-efficacy. In addition, it was observed that the mode of delivery affected postpartum comfort and its sub-dimensions, albeit not significantly. It was seen that, regarding the impact of the mode of delivery on postpartum comfort, cesarean delivery reduced physical comfort and sociocultural comfort, while vaginal delivery increased psycho-spiritual comfort. Against the background of these results, it is recommended that:

healthcare professionals take into account the factors that may affect postpartum comfort in the light of evidence-based practices and plan maternal care to increase their comfort,

maternal postpartum comfort be improved by avoiding unnecessary procedures during normal labor, and

healthcare professionals ground maternal postpartum care on the theory of comfort,

healthcare professionals identify breastfeeding problems by observing the mother before, during, and after birth, giving counseling when necessary, and leveraging their perception of breastfeeding self-efficacy.

Availability of data and materials

No datasets were generated or analysed during the current study.

Karabulutlu Ö. Factors affecting women’s birth type preferences. Florence Nightingale Nurs J. 2012;20(3):210.

Google Scholar

Taşkın, L. Doğum ve kadın sağlığı hemşireliği. 17. (Taşkın L. ed.). Ankara: Baskı, Akademisyen Kitabevi; 2021. p. 121–32.

Büyükkal T. The effect of delivery type on optimality level and comfort. Istanbul: Istanbul University-Cerrahpaşa, Graduate School of Education, Midwifery Department. Master’s Thesis; 2019.

Taytan S. The effect of caesarean section on the postpartum comfort level of mothers. Aydın: Aydın Adnan Menderes University Institute of Health Sciences Department of Midwifery. Master’s Thesis; 2019.

Kolcaba K. Comfort theory and practice: a vision for holistic health care and research. New York: Springer Publishing Co.; 2003.

Aksoy Derya Y, İbici Akça E, Kamalak H, Gökbulut N. The effect of the number of births, the mode of delivery and the planned pregnancy status on posttraumatic development and comfort in puerperant women. Mersin Univ Health Sci J. 2021;14(3):366–77. https://doi.org/10.26559/mersinsbd.828603 .

Article Google Scholar

Çapık A, Özkan H, Apay ES. Determination of postpartum comfort levels and influencing factors in postpartum women. Dokuz Eylul Univ Fac Nurs Electr J. 2014;7(3):186–92.

Narin Balsak H. The relationship between postpartum comfort and breastfeeding self-efficacy in mothers. Turkey: Institute of Health Sciences, Department of Nursing (Doctoral dissertation, Master's Thesis, Gaziantep: Gaziantep University); 2018.

UNICEF. Breastfeeding: A smart investment. Available at https://www.globalbreastfeedingcollective.org/ . Accessed 02 Aug. 2022.

WHO. Health Topics: Breastfeeding. Available at https://www.who.int/health-topics/breastfeeding#tab=tab_2 . Accessed 02 Aug. 2022.

Cantürk D, Akgün Kostak M. First breastfeeding behaviors, breastfeeding self-efficacy levels and influencing factors of mothers who had vaginal delivery and cesarean section. Mersin Univ J Health Sci. 2020;13(3):382–94. https://doi.org/10.26559/mersinsbd.718662 .

Aslan Y, Ege E. Association of mothers with breastfeeding self-efficacy and risk of depression. J Hum Sci. 2016;13(2):3160–72.

Akyüz A, Kaya T, Şenel N. Determining the breastfeeding behavior of the mother and the conditions affecting breastfeeding. TAF Prev Med Bullet. 2007;6(5):331–5.

Şahin G, Ergüven M. Evaluation of knowledge, practice and behaviors of breastfeeding mothers. İstanbul: Master thesis. Advisor: Müferet Ergüven, Istanbul Göztepe Training and Research Hospital Family Medicine Coordinator; 2008.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: PRISMA statement. BMJ. 2009;339:b2535. https://doi.org/10.1136/bmj.b2535

Article PubMed PubMed Central Google Scholar

Birgili F. Postpartum comfort of women giving birth and affecting factors. Anatolian J Nurs Health Sci. 2020;23(3):351–60. https://doi.org/10.17049/ataunihem.484941 .

Sis Çelik A, Çelik EC. Do the type of birth and the type of anesthesia taken at birth affect postpartum comfort? Jaren. 2020;6(1):97–108. https://doi.org/10.5222/jaren.2020.46704 .

Kurt Can E, Ejder Apay S. Mode of delivery: does it affect postpartum comfort and birth satisfaction levels? J İnönü Univ Voc School Health Serv. 2020;8(3):547–65. https://doi.org/10.33715/inonusaglik.753497 .

Erkaya R, Türk R, Sakar T. Determining comfort levels of postpartum women after vaginal and caesarean birth. Procedia Soc Behav Sci. 2017;237:1526–32. https://doi.org/10.1016/j.sbspro.2017.02.240 .

Özöztürk S, Aluş Tokat M, Aypar Akbağ NN, Ekinci F. The relationship between birth satisfaction and postpartum comfort by delivery type and parity. TJFMPC. 2022;16(1):179–88. https://doi.org/10.21763/tjfmpc.952205 .

Akkoyun S, Taş AF. Breastfeeding self-efficacy of breastfeeding mothers during the first six months. J Pediatr Res. 2016;3(4):191–5. https://doi.org/10.4274/jpr.50469 .

Yol E. Evaluation of breastfeeding self-efficacy and breastfeeding success of mothers whose babies were diagnosed with indirect hyperbilirubinemia. Ankara: Nursing Master’s Thesis), Ankara University Institute of Health Sciences; 2017.

İnce T, Aktaş G, Aktepe N, Aydın A. Evaluation of mothers' breastfeeding self-efficacy and characteristics affecting breastfeeding success. Journal of Dr. Behcet Uz Children's Hospital. 2017; 7 (3). https://doi.org/10.5222/buchd.2017.183 .

Erkal Aksoy Y, Ünal F, İnçke G, Çil G, Oruç K, Yılmaz T, et al. The effect of postpartum depression on breastfeeding. Gumushane Univ J Health Sci. 2016;5(3):90–6.

Bölükbaşı H. The effect of postpartum depression on breastfeeding self-efficacy, breastfeeding success and nutritional status in the postpartum six-month period. Ankara: Gazi University Institute of Health Sciences, Department of Nutrition and Dietetics, Master’s Thesis; 2016.

Turan A, Bozkurt G. Breastfeeding self-efficacy levels of primiparous mothers according to the mode of delivery and breastfeeding characteristics. EUHFD. 2020;36(1):1–10.

Kırca N, Adıbelli D. The effect of delivery type on breastfeeding self-efficacy perception. Int J Mainten Sci. 2020;13(1):698.

Ergezen Y, Efe E, Çalışkan Özdöl F. The Relationship Between Breastfeeding Self-Efficacy Perceptions of Mothers and Breastfeeding Success in the Postpartum Period. Dokuz Eylul Univ Fac Nurs Electr J. 2021;14(3):217–23. https://doi.org/10.46483/deuhfed.675733 .

Işık G, Cetişli NE, Başkaya VA. Postpartum pain, age and seasonal self-efficacy of mothers by birthday. Dokuz Eylul Univ Fac Nurs Electr J. 2018;11(3):224–32.

Moola S, Munn Z, Tufanaru C, Stern C, Porritt K, Farrow J.Chapter 7: Systematic reviews of etiology and risk. In: Aromataris E, Munn Z, ed. JBI Guide to Evidence Synthesis. JBI. 2020. https://doi.org/10.46658/JBIMES-20-08 .

Sahin S, Sinan Ö. Investigation of mothers’ postpartum breastfeeding and comfort conditions. Health Care Women Int. 2021;42(4–6):913–24. https://doi.org/10.1080/07399332.2021.1883022 .

Article PubMed Google Scholar

Pınar G, Doğan N, Algıer L, Kaya N, Çakmak F. Factors affecting the postpartum comfort of mothers. Dicle Med J. 2009;36(3):184–90.

Karakaplan S. The effects of delivery type on postpartum comfort of mothers and newborns. İstanbul: Marmara University, Institute of Health Sciences, Department of Obstetrics and Gynecology Nursing. Master’s Thesis; 2007.

Kolcaba KY. A Theory of holistic comfort for nursing. J Adv Nurs. 1994;19(6):1178–84.

Article CAS PubMed Google Scholar

Kolcaba KY, Kolcaba RJ. An analysis of the concept of comfort. J Adv Nurs. 1991;16:1301–10.

Gürol ES. Determining the relationship between breastfeeding proficiency status of mothers and nipple pain/trauma. İzmir: Master’s Thesis, Ege University. Health Sciences Institute,; 2011.

Aydın D, Aba YA. The relationship between mothers’ health literacy levels and breastfeeding self-efficacy perceptions. DEUHFED. 2019;12(1):31–9.

Dennis CL, Faux S. Development and psychometric testing of breastfeeding self - efficacy scale. Res Nurs Health. 1999;22:399–409. https://doi.org/10.1002/(SICI)1098-240X(199910)22:5<399::AID-NUR6>3.0.CO;2-44 .

Wutke K, Dennis CL. The reliability and validity of the Polish version of the Breastfeeding Self-Efficacy Scale-Short Form: translation and psychometric assessment. Int J Nurs Stud. 2007;44(8):1439–46. https://doi.org/10.1016/j.ijnurstu.2006.08.001 .

Dennis CL. Theoretical underpinnings of breast-feeding confidence: a self-efficacy framework. J Hum Lact. 1999;15:195–201. https://doi.org/10.1177/089033449901500303 .

Dennis CL. Breastfeeding self-efficacy scale: Psychometric assessment of the short form. J Obstet Gynecol Neonatal Nurs. 2003;32(6):734–44.

Download references

No funding was received for this article.

Author information

Authors and affiliations.

Department of Midwifery, Faculty of Health Sciences, Atatürk University, Erzurum, Turkey

Department of Midwifery, Faculty of Health Sciences, Bayburt University, Bayburt, Turkey

Elif Odabaşi Aktaş & Hafsa Kübra Işik

You can also search for this author in PubMed Google Scholar

Contributions

HO Conceptualization; Investigation; Writing - original draft; Funding acquisition; Methodology; Validation; Writing - review & editing; Visualization; Software; Formal analysis; Project administration; Data curation; Supervision; Resources. EOA Supervision; Resources; Writing - review & editing; Writing - original draft; Data curation. HKI Writing - original draft; Writing - review & editing.

Corresponding author

Correspondence to Elif Odabaşi Aktaş .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Özkan, H., Aktaş, E.O. & Işik, H.K. The effect of mode of delivery on postpartum comfort level and breastfeeding self-efficacy: a systematic review and meta-analysis. matern health, neonatol and perinatol 10 , 17 (2024). https://doi.org/10.1186/s40748-024-00187-3

Download citation

Received : 21 April 2024

Accepted : 23 July 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s40748-024-00187-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Cesarean delivery
Meta-analysis
Vaginal delivery

Maternal Health, Neonatology and Perinatology

ISSN: 2054-958X

Submission enquiries: Access here and click Contact Us
General enquiries: [email protected]

IMAGES

(PDF) How to Write a Systematic Review
(PDF) How to do a systematic literature review
Systematic Literature Review Methodology
Systematic Literature Review Methodology
FREE 12+ Sample Literature Review Templates in PDF, Word
(PDF) Systematic Reviews as Undergraduate Dissertation Research

VIDEO

Literature Review Vs Systematic Review
Differences Between Systematic Review and Scoping Review
Ace the Systematic Literature Review!
Systematic Literature Review: An Introduction [Urdu/Hindi]
Complete Thesis Guidelines
Research introduction structure

COMMENTS

How to write the methods section of a systematic review
Keep it brief. The methods section should be succinct but include all the noteworthy information. This can be a difficult balance to achieve. A useful strategy is to aim for a brief description that signposts the reader to a separate section or sections of supporting information. This could include datasets, a flowchart to show what happened to ...
Chapter 1. Carrying Out a Systematic Review as a Master's Thesis
by Angela Boland, M. Gemma Cherry and Rumona Dickson. Chapter 1. Carrying Out a Systematic Review as a Master's Thesis. Explore the wealth of resources available across the web. Here are some good places to start. Link to the Campbell Collaboration, an organization that prepares, maintains and disseminates systematic reviews in education, crime ...
How to Write a Systematic Review: A Narrative Review
Background. A systematic review, as its name suggests, is a systematic way of collecting, evaluating, integrating, and presenting findings from several studies on a specific question or topic.[] A systematic review is a research that, by identifying and combining evidence, is tailored to and answers the research question, based on an assessment of all relevant studies.[2,3] To identify assess ...
How to do a systematic review
A systematic review aims to bring evidence together to answer a pre-defined research question. This involves the identification of all primary research relevant to the defined review question, the critical appraisal of this research, and the synthesis of the findings.13 Systematic reviews may combine data from different.
Chapter 1: Carrying Out a Systematic Review as a Master's Thesis
Doing a Systematic Review: A Student's Guide. by Angela Boland, M. Gemma Cherry and Rumona Dickson. Chapter 1: Carrying Out a Systematic Review as a Master's Thesis. What Is The Difference Between A Systematic Review And A Meta-Analysis?
Guidelines for writing a systematic review
Guidelines for writing a systematic review. 1. Introduction. A key feature of any academic activity is to have a sufficient understanding of the subject area under investigation and thus an awareness of previous research. Undertaking a literature review with an analysis of the results on a specific issue is required to demonstrate sufficient ...
Systematic Review
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr. Robert Boyle and his colleagues published a systematic review in ...
Chapter 1. Carrying Out a Systematic Review as a Master's Thesis
by Angela Boland, M. Gemma Cherry and Rumona Dickson. Chapter 1: Carrying Out a Systematic Review as a Master's Thesis. Chapter 3: Defining My Review Question and Identifying Inclusion and Exclusion Criteria. Chapter 5: Applying Inclusion and Exclusion Criteria. Chapter 8: Understanding and Synthesizing Numerical Data from Intervention Studies.
How to Do a Systematic Review: A Best Practice Guide ...
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to ...
How to Do a Systematic Review: A Best Practice Guide for Conducting and
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question.
Introduction to Systematic Reviews
A systematic review identifies and synthesizes all relevant studies that fit prespecified criteria to answer a research question (Lasserson et al. 2019; IOM 2011).What sets a systematic review apart from a narrative review is that it follows consistent, rigorous, and transparent methods established in a protocol in order to minimize bias and errors.
How to write a systematic literature review [9 steps]
Analyze the results. Interpret and present the results. 1. Decide on your team. When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.
PDF Conducting a Systematic Review: Methodology and Steps
METHODOLOGY AND STEPS. 9. MPLE DATA EXTRACTION FORM 201.INTRODUCTIONSystematic reviews have gained momentum as a key method of evidence syn. hesis in global development research in recent times. As defined in the Cochrane Handbook on Systematic reviews "Systematic reviews seek to collate evidence that fits pre-specified eligibility cri.
Systematic reviews: Structure, form and content
Topic selection and planning. In recent years, there has been an explosion in the number of systematic reviews conducted and published (Chalmers & Fox 2016, Fontelo & Liu 2018, Page et al 2015) - although a systematic review may be an inappropriate or unnecessary research methodology for answering many research questions.Systematic reviews can be inadvisable for a variety of reasons.
PDF DOING A SYSTEMATIC REVIEW
5 A Systematic Review as a Master's Thesis A systematic review as a Master's thesis: advantages and disadvantages Advantages • You are in control of your learning objectives and your project • You can focus on a topic that you're interested in • You don't have to gain formal ethical approval for your review before you begin
Systematic Review
A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr Robert Boyle and his colleagues published a systematic review in ...
An overview of methodological approaches in systematic reviews
1. INTRODUCTION. Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the "gold standard" of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search ...
Systematic Reviews: Home
Systematic reviews and meta-analysis by Julia H. Littell, Jacqueline Corcoran, ... Rumona Dickson (Editor) If you are a Masters or a PhD student conducting a systematic review for your dissertation or thesis, then this is the book for you! Written by an expert team of authors with years of experience in conducting systematic reviews and ...
Systematic Reviews and Meta Analysis
It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.
Guidance on Conducting a Systematic Literature Review
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
Acceptance of a systematic review as a thesis: survey of biomedical
Background Systematic reviews (SRs) have been proposed as a type of research methodology that should be acceptable for a graduate research thesis. The aim of this study was to analyse whether PhD theses in European biomedical graduate programs can be partly or entirely based on SRs. Methods In 2016, we surveyed individuals in charge of European PhD programs from 105 institutions. The survey ...
PDF Master'S Thesis a Systematic Literature Review on Agile Project ...
le project management (APM), the thesis uses a systematic literature review. Since managing projects in agile way is a relatively new concept compared to the traditional waterfall model, the resu. ts of the review provide an overview of the research conducted in this area. The results are expected t. he.
Assessing the Certainty of the Evidence in Systematic Reviews
When interpreting results and drawing conclusions, authors of systematic reviews should consider the limitations of the evidence included in their review. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach provides a framework for the explicit consideration of the limitations of the evidence included in a ...
Meta-analysis and systematic review of the diagnostic value of contrast
Data extraction and synthesis Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) evaluated the methodological quality of all the included studies. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses specification. In addition to sensitivity and specificity, other important parameters were explored in an analysis of CESM accuracy for breast ...
How Covidence Helps Streamline Systematic Reviews
Covidence is a systematic review screening tool used to streamline the process of screening, reviewing, analyzing, synthesizing and disseminating data you collect from reviewed articles. The main point of conducting systematic reviews is to analyze the findings across studies to answer a clinical question. This process is a time-consuming team ...
Something old, new, and borrowed . Rise of the systematic reviews
Systematic reviews and other types of literature reviews are more prevalent in clinical medicine than in other fields. The recurring need for improvement and updates in these disciplines has led to the Living Systematic Review (LSR) concept to enhance the effectiveness of scientific synthesis efforts. While LSR was introduced in 2014, its adoption outside clinical medicine has been limited ...
Causes and Clinical Presentation of Drug-Induced Dermatomyositis
Key Points. Question What are the most common medication causes, clinical presentations, and laboratory findings for drug-induced dermatomyositis (DM)?. Findings In this systematic review of 134 studies reporting data on 165 cases of drug-induced DM, drug-induced DM was found to occur after treatment with hydroxyurea, immune checkpoint inhibitors, statins and lipid-lowering agents ...
(PDF) A Systematic Review of E-Waste: Growing ...
A Systematic Review of E-Waste: Growing Environmental Risk, Its Impact and Management. August 2024; In book: Green India: Pathways to Environmental Sustainability (pp.115-127)
A systematic review of chlamydia, gonorrhoea, trichomoniasis, and
This systematic review aimed to identify and collate prevalence estimates for the European general population and populations of special interest for the four curable STIs: chlamydia (etiological agent Chlamydia trachomatis), gonorrhoea (etiological agent Neisseria gonorrhoeae ), trichomoniasis (etiological agent Trichomonas vaginalis ), and syphilis (etiological agent Treponema pallidum ...
The effect of mode of delivery on postpartum comfort level and
This study was conducted to determine the effect of the mode of delivery on maternal postpartum comfort level and breastfeeding self-efficacy. The study was conducted as a systematic review and a meta-analysis. Searching was performed from March to July 2022, on PubMed, National Thesis Center, Dergi Park, Google Scholar, Web of Science, and EBSCO search engines and we included studies from the ...

How to write the methods section of a systematic review

Covidence breaks down how to write a methods section

Describe what happened

Keep it brief

Follow a structure

1. Selection criteria ⭕

2. Search 🕵🏾‍♀️

3. Data collection and analysis 👩‍💻

4. Study quality and risk of bias ⚖️

Laura Mellor. Portsmouth, UK

Top 5 Tips for High-Quality Systematic Review Data Extraction

How to get through study quality assessment Systematic Review

How to extract study data for your systematic review

Better systematic review management

Have a language expert improve your writing

Systematic Review | Definition, Example & Guide

Table of contents

Receive feedback on language, structure, and formatting

Prevent plagiarism. Run a free check.

Step 1: Formulate a research question

Step 2: Develop a protocol

Step 3: Search for all relevant studies

Step 4: Apply the selection criteria

Step 5: Extract the data

Step 6: Synthesize the data

Step 7: Write and publish a report

Cite this Scribbr article

Is this article helpful?

Shaun Turney

Doing a Systematic Review: A Student's Guide

Annual Review of Psychology

Most Read This Month

Save citation to file

Add to My Bibliography

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses

Similar articles

LinkOut - more resources

Other Literature Sources

Miscellaneous

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Have a language expert improve your writing

Systematic Review | Definition, Examples & Guide

Table of contents

Prevent plagiarism, run a free check.

Step 1: Formulate a research question

Step 2: Develop a protocol

Step 3: Search for all relevant studies

Step 4: Apply the selection criteria

Step 5: Extract the data

Step 6: Synthesise the data

Step 7: Write and publish a report

Cite this Scribbr article

Is this article helpful?

Shaun Turney

An overview of methodological approaches in systematic reviews

Hanny Calache

Akshaya Pandian

Mohd Masood

1. INTRODUCTION

2.1. Eligibility criteria

2.2. Search

2.3. Study selection and data extraction

2.4. Quality assessment of included reviews

2.5. Data synthesis

3.1. Characteristics of included reviews

3.2. Methodological quality of included reviews

3.3. Evidence on systematic review methods

3.3.2. Methods for searching studies

3.3.3. Methods for selecting studies

3.3.4. Methods for data extraction

3.3.5. Methods for RoB assessment

4. DISCUSSION

4.2. Implications for future research

4.3. Strengths and limitations

5. CONCLUSIONS

CONFLICT OF INTEREST

Supporting information

ACKNOWLEDGMENTS

Systematic Reviews: Home