systematic literature review grid

We apologize for any inconvenience as we update our site to a new look.

• Walden University
• Faculty Portal

Common Assignments: Literature Review Matrix

Literature review matrix.

As you read and evaluate your literature there are several different ways to organize your research. Courtesy of Dr. Gary Burkholder in the School of Psychology, these sample matrices are one option to help organize your articles. These documents allow you to compile details about your sources, such as the foundational theories, methodologies, and conclusions; begin to note similarities among the authors; and retrieve citation information for easy insertion within a document.

You can review the sample matrixes to see a completed form or download the blank matrix for your own use.

• Literature Review Matrix 1 This PDF file provides a sample literature review matrix.
• Literature Review Matrix 2 This PDF file provides a sample literature review matrix.
• Literature Review Matrix Template (Word)
• Literature Review Matrix Template (Excel)

Related Resources

Didn't find what you need? Email us at [email protected] .

• Previous Page: Commentary Versus Opinion
• Next Page: Professional Development Plans (PDPs)
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

• Subscribe to journal Subscribe
• Get new issue alerts Get alerts

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Systematic Reviews

Constructing a search strategy and searching for evidence.

Aromataris, Edoardo PhD; Riitano, Dagmara BHSC, BA

Edoardo Aromataris is the director of synthesis science at the Joanna Briggs Institute in the School of Translational Health Science, University of Adelaide, South Australia, where Dagmara Riitano is a research officer. Contact author: Edoardo Aromataris, [email protected] . The authors have disclosed no potential conflicts of interest, financial or otherwise.

The Joanna Briggs Institute aims to inform health care decision making globally through the use of research evidence. It has developed innovative methods for appraising and synthesizing evidence; facilitating the transfer of evidence to health systems, health care professionals, and consumers; and creating tools to evaluate the impact of research on outcomes. For more on the institute's approach to weighing the evidence for practice, go to http://joannabriggs.org/jbi-approach.html .

Overview 

This article is the third in a new series on the systematic review from the Joanna Briggs Institute, an international collaborative supporting evidence-based practice in nursing, medicine, and allied health fields. The purpose of the series is to show nurses how to conduct a systematic review—one step at a time. This article details the major considerations surrounding search strategies and presents an example of a search using the PubMed platform (pubmed.gov).

The third article in a series from the Joanna Briggs Institute details how to develop a comprehensive search strategy for a systematic review.

The systematic literature review, widely regarded as the gold standard for determining evidence-based practice, is increasingly used to guide policy decisions and the direction of future research. The findings of systematic reviews have greater validity than those of other types of reviews because the systematic methods used seek to minimize bias and increase rigor in identifying and synthesizing the best available evidence on a particular question. It's therefore important that when you search for evidence, you attempt to find all eligible studies and consider them for inclusion in your review. 1

One rule of thumb we use when beginning a search for evidence to support a systematic review: if you don't find the evidence, it can't be reviewed! Unfortunately, there is no prescriptive approach to conducting a comprehensive search. But searching is an art that can be cultivated and practiced. It involves several standard processes, such as developing search strings, searching across bibliographic citation databases that index health care research, looking for “gray,” or unpublished, literature, and hand searching.

GETTING STARTED

Developing a search strategy is an iterative process—that is, it involves continual assessment and refinement. As keywords or key terms are used in a search, their usefulness will be determined by the search results. Consequently, searching for evidence is sometimes considered more of an art than a science. It's therefore unlikely that two people, whether they are clinicians or librarians, will develop an identical search strategy or yield identical results from a search on the same review question.

The time required to conduct a search for a systematic review will also vary. It's dependent on the review question, the breadth of the evidence base, and the scope of the proposed search as stated in the review protocol. Narrow searches will often be adequate when investigating a topic requiring a few specific keywords, such as when you're searching only for randomized controlled trials (RCTs) conducted in a single population with a rare disorder. A narrow search will be less resource intensive than a search conducted when the review question is broader or the search relies on general keywords (such as education , prevention , or experience ). And while it may seem important conceptually to use a general keyword (such as safety in a search for articles on medical errors, for example), in practice it will add few relevant studies beyond those identified using more specific terms (such as error or harm ).

When beginning the search for evidence, you should conduct a few small searches as a test of various search terms and combinations of terms. An ideal search strategy is both sensitive and specific: a sensitive search will recall relevant studies, while a specific search will exclude irrelevant studies. A search that is overly sensitive may capture all the necessary studies but may require a labor-intensive vetting of unnecessary studies at the stage of study selection. A search that is overly specific will yield fewer results but is always subject to the risk that important studies may have been omitted.

Finding help. Given the complexity of the many indexing languages and rules governing the various databases, we recommend that early in the process you make use of an experienced research librarian who can examine your search strategy and help you choose citation databases relevant to your review question. If you can't easily access the services of a research librarian, there are many online tutorials that can help. A Google search—for example, “How do I search using PubMed?”—will reveal sites containing helpful hints and training developed by the U.S. National Library of Medicine (NLM) and librarians from across the globe.

DEVELOPING THE SEARCH STRATEGY

A review protocol with a clearly defined review question and inclusion criteria will provide the foundation for your search strategy. Before embarking on the search, you will need to understand the review question and what information you'll need to address it. For example, it's important to consider the type of data being sought (quantitative, qualitative, economic), the types of studies that report the data (RCTs, cohort studies, ethnographic studies), and the limits or restrictions you'll apply (publication date or language). This will shorten the time required to search and help to ensure that the information retrieved is both relevant and valid.

Once you've determined the review question, you'll need to identify the key terms articulated in the question and the protocol and create a logic grid or concept map. In a logic grid for a review on the effectiveness of an intervention, for example, each column represents a discrete concept that is generally aligned with each element of the PICO mnemonic— P opulation, I ntervention, C omparison intervention, and O utcome measures.

Consider an example using the following review question: “Is animal-assisted therapy more effective than music therapy in managing aggressive behavior in elderly people with dementia?” Within this question are the four PICO concepts: elderly patients with dementia (population), animal-assisted therapy (intervention), music therapy (comparison intervention), and aggressive behavior (outcome measures) (see Table 1 for an example of a logic grid).

Keywords or free-text words. The first formal step in all searches is to determine any alternative terms or synonyms for the identified concepts in the logic grid. Normally, you'll identify these terms—often referred to as keywords or free-text words—within the literature itself. Perhaps you'll start with a simple search using the terms dementia and animal-assisted therapy or music therapy and aggressive behavior . By looking at the titles and abstracts of the retrieved articles, you can find key terms used in the literature, as well as key concepts that are important to your question. For instance, is the term animal-assisted therapy used synonymously with the term pet therapy ? Furthermore, retrieving and reading a few relevant studies of any design—such as an experimental study or a traditional literature review on the topic—will further aid in identifying any commonly used terms.

When developing your search strategy, note that most search platforms (such as Ovid or EBSCOhost) used to access databases (such as MEDLINE) search for the exact terms entered in the database, including any misspellings. This means that to conduct a comprehensive search, you should enter as many relevant key terms as possible. Important articles may be overlooked if all relevant synonyms for a concept aren't included, as some authors may refer to the same concept using a different term (such as heart attack instead of myocardial infarction ). Such differences notwithstanding, you may find that including a relevant but broad term may retrieve many irrelevant studies.

Expanding on the logic grid shown in Table 1 , Table 2 now contains the keywords chosen from scanning the titles and abstracts of retrieved articles in your initial search. Column one contains terms relating to dementia , the defining feature of the population of interest; columns two and three contain terms relating to animal-assisted therapy and music therapy , the intervention and comparator of interest; and column four contains terms relating to aggressive behavior , the outcome of interest. Placing the terms into a logic grid illustrates how the related concepts or synonyms will combine to construct the final search string.

Index terms or subject headings. Comprehensive search strategies should consist of both keywords or free-text words and index terms, which are used by some major bibliographic databases to describe the content of each published article using a “controlled vocabulary”—that is, a list of standard terms that categorize articles based on their content (such terms will vary from database to database). For example, PubMed uses medical subject heading (MeSH) terms, the controlled vocabulary of MEDLINE. 2 MeSH terms are categorized within 16 main “trees” (such as anatomy, organisms, diseases, drugs, and chemicals), each of which branches from the broadest to the most specific terms.

To determine whether index terms exist for the concepts you've identified in your review question, you can search for each term in the MeSH database (selected from the drop-down list on the PubMed home page). For example, by entering dementia , PubMed will identify relevant MeSH terms that include Dementia and Alzheimer Disease . By selecting Dementia , you'll see the term's tree, including the subcategories listed below it, such as Lewy Body Disease .

As was the case when identifying key terms to use in the search strategy, it is also recommended that an initial, simple search using a few key concepts ( dementia AND animal-assisted therapy or dementia AND music therapy AND aggressive behavior ) be performed in PubMed to identify index terms. The aim is to retrieve a few relevant articles to see how they were indexed using the controlled vocabulary. Once the results are displayed, you can scroll through the citations and click on the title of any eligible article to view its details. From here, follow the link to the article's MeSH terms and examine which ones were used to describe the article's content. Repeat this process with a number of different articles to determine whether similar indexing terms have been used.

The terms in the logic grid can now be updated with the MeSH terms you have chosen from those listed with each retrieved article (see Table 3 ). The [mh] that appears next to these terms in the grid is the search-field descriptor that stands for “MeSH headings.” It's worth noting that “Entry Terms” under each search term's MeSH listing (if one is available) can also be examined for suggestions of alternative terms that can be searched in titles and abstracts.

Because new articles in PubMed are not indexed immediately, and because indexing is a manual, subjective process susceptible to human variation, it's important to also search for the key terms in the titles and abstracts of articles—in other words, for free-text or keywords—to capture any articles that could be missed by using index terms (such as MeSH headings) alone. For example, if we did not search for free-text words and did not include the index term Bonding, Human Pet (a MeSH term), we might miss an important article that wasn't indexed under the MeSH term Animal-Assisted Therapy .

By adding the search-field descriptor [tiab] (meaning “title/abstract”) to a search term, you can direct PubMed to search the title and abstract field code for these terms. A number of other search-field descriptors can be used as well, such as [au] for “author” and [pt] for “publication type.” 2 Using a search-field descriptor such as [tw] (“text word”) is often preferred over [tiab] for systematic reviews because the former searches in the title and abstract of articles as well as across a greater number of fields and will return a greater number of results for the same search query. Shortcuts or “wildcard” characters can also be used to account for different terminology or spelling. For example, PubMed allows truncation searching, in which an asterisk can substitute for any word's beginning or ending (for instance, a search for therap* will retrieve articles with the words therapy and therapeutic ). Search-field descriptors and wildcard characters should be applied to any newly identified keywords and index terms in the logic grid (see Table 4 ).

Once all search terms, including both free-text words and indexing terms, have been collected and finalized, a second search can then be undertaken across all selected citation databases. Initially, the key terms and synonyms within each column in the logic grid are combined using “OR.” (Most databases use some form of Boolean logic—search terms connected by the Boolean operators “OR” and “AND,” among others.) This will direct the database to find articles containing any of the search terms within the indicated fields. To do this in PubMed, select the “Advanced” search box and clear the search history. Copy and paste the first set of terms into PubMed and run the search.

For example, an initial search for articles related to different types of dementia might look like this:

“Dementia [tw] OR Alzheimer [tw] OR Huntington* [tw] OR Kluver [tw] OR Lewy [tw] OR Dementia [mh] OR Alzheimer disease [mh]"

This search could yield more than 100,000 citations. Following this, clear the search box and repeat the process with search terms from the second column in Table 4 . It is easier to search each column of the logic grid individually—particularly if each column contains an extensive list of search terms—rather than combining all the search sets in one go. Furthermore, by running each search successively you can determine if a component of the search string is producing many irrelevant results and easily adjust the search strategy. In our example, if you add the term aggress* [tw] to capture aggressive and aggression in the title or abstract, you will get an overwhelming number of irrelevant results because these terms are also used to describe the spread of certain cancers.

Once you complete the searches aligned to each concept, click on the “Advanced” option again. This allows for display of the “search history” and for a ready combination of the individual searches using the Boolean operators “AND” and “OR.” Using this method, parentheses are automatically placed around each set of terms to maintain the logical structure of the search. For example, the search for articles on animal-assisted therapy versus music therapy to treat aggression in patients with dementia might look like this:

“(Dementia [tw] OR Alzheimer [tw] OR Huntington* [tw] OR Kluver [tw] OR Lewy [tw] OR Dementia [mh] OR Alzheimer disease [mh]) AND (Animal assisted therapy [tw] OR Animal assisted activit* [tiab] OR Animal assisted intervention* [tiab] OR Animal therapy [tw] OR Pet therapy [tw] OR Dog therapy [tw] OR Dog assisted therapy [tw] OR Canine assisted therapy [tw] OR Aquarium [tiab] OR Animal Assisted Therapy [mh] OR Pets [mh] OR Dogs [mh] OR Cats [mh] OR Birds [mh] OR Bonding, Human-Pet [mh] OR Animals, Domestic [mh]) OR (Music* [tw] OR Music therapy [tw] OR Singing [tw] OR Sing [tw] OR Auditory stimulat* [tw] OR Music [mh] OR Music Therapy [mh] OR Acoustic Stimulation [mh] OR Singing [mh]) AND (Aggression [tw] OR Neuropsychiatric [tiab] OR Apathy inventory [tiab] OR Cornell scale [tiab] OR Cohen Mansfield [tiab] OR BEHAVE-AD [tiab] OR CERAD-BRSD [tiab] OR Behavior* [tiab] OR Behaviour* [tiab] OR Aggression [mh] OR Personality inventory [mh] OR Psychomotor agitation [mh])"

Once the final search has been conducted, you can further refine search results by publication date, study groups, language, or any other limits appropriate to the review topic by selecting the relevant filter (left-hand side of the screen in PubMed) from the range available. PubMed also provides predefined search filters that restrict search results to specific clinical study categories or subject matters (such as clinical queries). You will have determined the date range for the search at the protocol development stage. Given that your aim is to summarize the evidence surrounding a particular question, you should justify any limits to the publication date of included studies in the background section of the protocol. The chosen time frame will vary depending on the review question. For example, reviewers may impose a start date for a search that coincides with the introduction of a new intervention and the advent of the preceding clinical research on it.

The structure of the search strategy will remain the same regardless of the search platform used to search a database. But since most major databases use a unique controlled vocabulary to index their articles, the indexing terms will need to be adapted to each database; in most cases the key terms remain the same across different databases. These differences in indexing terms are the main reason it is not recommended to search bibliographic citation databases for a systematic review using a federated search engine or platform—that is, one that searches multiple databases and sources at once.

You should also be aware that the platforms used to search citation databases often use different wildcard characters or commands. For this reason, beginning searchers should use the online tutorials and help pages of the various platforms and databases. For example, while Ovid's search platform can also be used to search the MEDLINE database, the terms used for truncation searching are quite different: an asterisk (*) is used for unlimited truncation within PubMed and a dollar symbol ($) in Ovid. Moreover, in Ovid the question mark (?) wildcard can be used within or at the end of a word to substitute for one character or no characters ( behavio?r will retrieve articles with the words behaviour and behavior ); the number sign (#) wildcard can substitute for a single character ( wom#n will retrieve articles with both woman and women ). The use of wildcards for substitution of characters is not supported in PubMed.

Because searching is an iterative process, you won't want to predetermine when it will end. Consequently, it is important to look at the results of the search continually as you develop the search strategy to determine whether the results are relevant. One way to do this is to check if already identified relevant articles are being captured by the search. If not, the search strategy will need to be modified accordingly.

Once the search is complete, the results can be exported to bibliographic management software such as EndNote or Reference Manager. These tools are useful for organizing the search results, removing duplicate citations, and selecting studies (the next step of the systematic review process, to be discussed in the next article in this series).

WHERE TO SEARCH?

Developing the search strategy and search filters for use within each database is an important and time-consuming part of the search process, often more so than the search itself! Another important consideration is where to search. A search for a systematic review should be comprehensive and attempt to identify all of the available evidence. This can be an enormous undertaking.

Generally, a systematic review to inform health care practice and policy should search the major medical databases including MEDLINE from the NLM in North America and searchable through PubMed, and Embase, a product of Elsevier that indexes many European biomedical journals; the controlled vocabulary for Embase is searchable through Emtree, which also contains all MeSH terms ( www.elsevier.com/online-tools/embase/emtree ). Nurses undertaking systematic reviews will find that much literature relevant to nursing practice is also available in the Cumulative Index to Nursing and Allied Health Literature (CINAHL) database by EBSCO. Beyond these, there are many others: Web of Science, PsycINFO, Scopus, JSTOR, Academic Search Premier, Academic Onefile, the Cochrane Nursing Care Field trials register, and the list goes on.

You should establish which databases index articles relevant to the topic at hand. Some databases have a specific topic focus, such as PsycINFO, which should be searched for a question related to mental health. The JBI Database of Systematic Reviews and Implementation Reports is, as the name suggests, a repository for systematic reviews and would be unnecessary for most review searches (systematic reviews rarely include other systematic reviews among their inclusion criteria). Similarly, a quick Google search (“What information is in… ?”) to establish the content and coverage of other databases is worthwhile and will help in identifying unnecessary overlap in the search strategy.

Hand searching. You may also wish to consider more traditional means of locating evidence. Screening the reference lists of studies already selected for inclusion in the review is often a valuable means of identifying other pertinent studies. Similarly, hand searching specific journals is often used by systematic review authors to locate studies. Journals selected for hand searching should be identified as relevant from database or preliminary searching; the likelihood is that these journals may contain relevant studies. Because hand searching can be an onerous task, it's recommended that no more than two or three relevant journals should be hand searched for a review.

Finding experts is another method of locating evidence. While contacting authors to clarify details of studies and to request data are relatively common pursuits for the systematic reviewer during the appraisal and extraction processes, doing so to identify relevant studies can also be useful. Such experts can often provide papers that even a comprehensive search may have failed to identify.

SHADES OF GRAY

Systematic reviews that purport to have conducted a comprehensive search should have made some attempt to search for gray literature. The International Conference on Grey Literature in Luxembourg defined it in 1997 (and expanded on it in 2004) as literature “produced at all levels of government, academic, business and industry in electronic and print formats not controlled by commercial publishing.” 3 However, this definition is often broadened to include any study or paper that has not been formally published or peer reviewed. Gray literature often appears in the form of government or institution reports and newsletters and even in blogs, conference proceedings, census reports, or nonindependent research papers. As a result, these reports or manuscripts are often not as widely available and are generally more difficult to locate.

Nonetheless, the inclusion of gray literature in systematic reviews has emerged as an important adjunct to commercially published research, as it often reflects a source of timely or alternative information that can help to minimize publication bias and provide a more accurate and thorough account of the evidence. 4, 5

There are three common ways to search for gray literature. The first involves searching or browsing the Web sites of organizations relevant to the review question (such as the World Health Organization or the National Institute for Health and Care Excellence). The second involves searching databases that collate and index gray literature. Although gray literature is rarely indexed, two commonly used sources are OpenGrey ( www.opengrey.eu ), an open access database to gray literature from Europe, and the Grey Literature Report ( www.greylit.org ), a bimonthly report from the New York Academy of Medicine. Reviewers will find that such databases do not have an extensive or advanced search capability, and therefore searching them is often limited to the use of a few critical keywords. Furthermore, they lack indexing or subject headings; without this feature a search can be quite time consuming. The third approach is to use online search engines. Search engines such as Google do not use a controlled vocabulary and so performing a simple search of a few select keywords is best. Such sites will yield a large number of results. To make results more manageable, you can try limiting the search to terms that appear on the title page of an article only 6 or by using keywords that limit the results to specific documents (such as guidelines). Searches can also be limited by language or sources (for example, adding site:gov to a Google search will limit results to government Web sites). An example of a tool that can also help is the federated search engine MedNar ( http://mednar.com/mednar ) that searches across a range of government and organizational sites, as well as commercial databases.

Other sources of gray literature can be found in numerous guides developed to assist researchers. For example, the Canadian Agency for Drugs and Technologies in Health's Grey Matters provides an extensive list of gray literature sources that can be searched. 7 Developed with the systematic reviewer in mind, the tool kit provides a checklist that aids users in documenting the search process and in ensuring it has been conducted in a standardized way.

REPORTING THE SEARCH STRATEGY

The final consideration is reporting the details of the search strategy, including the filters (such as language, date limits) and databases and other sources used. A hallmark of a systematic review is its reproducibility; another researcher should be able to review the same question and arrive at similar conclusions. Without a transparent reporting of the search strategy—one that allows readers to assess the quality of the search and its sources, and in turn, make a judgment on the likely credibility of the review 8, 9 —this would not be possible.

Most journals that publish systematic reviews now espouse the PRISMA guidelines (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; online at www.prisma-statement.org ), which dictate that the full search strategy for at least one major database should be reported in an appendix and published along with the review. 10 Online repositories of systematic reviews, such as the JBI Database of Systematic Reviews and Implementation Reports and the Cochrane Database of Systematic Reviews , allow for publication of all the search filters and strategies across the databases and sites used. A systematic reviewer will appreciate that reporting only the search filters used is inadequate. The methods section of a review should list all of the bibliographic citation databases searched, ideally with the platform used to search them, as well as the dates they were searched and any limits used. The results of the search should be adequately reported, as well; this is often quite simple to convey in a flow diagram, which is also detailed in the PRISMA guidelines. 10

Once the search is complete and the results from each source have been exported, the next step, study selection, can begin. This is where titles, abstracts, and sometimes the full text of studies found are screened against the inclusion and exclusion criteria. This step of the process will be the focus of the next article in this series.

evidence; gray literature; literature search; review question; systematic review

• + Favorites
• View in Gallery

Readers Of this Article Also Read

The systematic review: an overview, evidence-based practice: step by step: igniting a spirit of inquiry, evidence-based practice: step by step: the seven steps of evidence-based..., evidence-based practice, step by step: asking the clinical question: a key step ..., developing the review question and inclusion criteria.

Systematic Reviews and Meta Analysis

• Getting Started
• Guides and Standards
• Review Protocols
• Databases and Sources
• Randomized Controlled Trials
• Controlled Clinical Trials
• Observational Designs
• Tests of Diagnostic Accuracy
• Software and Tools
• Where do I get all those articles?
• Collaborations
• EPI 233/528
• Countway Mediated Search
• Risk of Bias (RoB)

Systematic review Q & A

What is a systematic review.

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

For details about carrying out systematic reviews, see the Guides and Standards section of this guide.

Is my research topic appropriate for systematic review methods?

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of the findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

If not a systematic review, then what?

You might consider performing a scoping review . This framework allows iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. The framework includes built-in mechanisms to adjust the analysis as the work progresses and more is learned about the topic. A scoping review won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

Can my student complete a systematic review during her summer project?

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion criteria and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

How can I know if my topic has been been reviewed already?

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be captured by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist. Cochrane systematic reviews do not include 'systematic' in the title, however. It's worth checking the Cochrane Database of Systematic Reviews independently.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO , a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library .

• Next: Guides and Standards >>
• Last Updated: Feb 26, 2024 3:17 PM
• URL: https://guides.library.harvard.edu/meta-analysis

Introducing the Literature Grid: Helping Undergraduates Consistently Produce Quality Literature Reviews

APSA 2012 Teaching & Learning Conference Paper

20 Pages Posted: 2 Feb 2012 Last revised: 14 Feb 2012

Peter Rocco Yacobucci

SUNY Buffalo State College

Date Written: 2012

The literature review or critical review essay is an increasingly common component of the undergraduate political science research paper. Unfortunately very little has been written discussing how such a literature review should be crafted (Knopf 2006). As new and more powerful search engines become available to our students, the difficulty of conducting a search of the relevant literature has been displaced by the complexity of organizing and effectively summarizing the large amount of literature found. This article introduces the Literature Grid, a heuristic device I have developed and provide my students in my undergraduate research design classes that simplifies the crafting of a quality literature review. No longer am I burdened by reading student literature reviews that are either mild modifications of annotated bibliographies or a jumble of references that have no direction or synthesis. By using the Literature Grid my students are able to analyze the relevant literature on their chosen topic based on important causal inferences and variable impacts. My students’ literature reviews have become a meaningful segment of their research and have enlivened their independent research to further the field. This article will outline how a Literature Grid is created, what it can and cannot do for your students and provide a practical, easy-to-follow heuristic device that will inspire your students’ critical review essays to summarize and engage the current literature.

Keywords: Literature review, review of literature, integrative review, literature grid, literature grids

Suggested Citation: Suggested Citation

Peter Rocco Yacobucci (Contact Author)

Suny buffalo state college ( email ).

1300 Elmwood Ave Buffalo, NY 14222 United States 7168786116 (Phone)

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, social sciences education ejournal.

Subscribe to this fee journal for more curated articles on this topic

Political Science Educator: Courses, Cases & Teaching eJournal

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PMC10248995

Guidance to best tools and practices for systematic reviews

Kat kolaski.

1 Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC USA

Lynne Romeiser Logan

2 Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY USA

John P. A. Ioannidis

3 Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA USA

Associated Data

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.

A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.

Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13643-023-02255-9.

Part 1. The state of evidence synthesis

Evidence syntheses are commonly regarded as the foundation of evidence-based medicine (EBM). They are widely accredited for providing reliable evidence and, as such, they have significantly influenced medical research and clinical practice. Despite their uptake throughout health care and ubiquity in contemporary medical literature, some important aspects of evidence syntheses are generally overlooked or not well recognized. Evidence syntheses are mostly retrospective exercises, they often depend on weak or irreparably flawed data, and they may use tools that have acknowledged or yet unrecognized limitations. They are complicated and time-consuming undertakings prone to bias and errors. Production of a good evidence synthesis requires careful preparation and high levels of organization in order to limit potential pitfalls [ 1 ]. Many authors do not recognize the complexity of such an endeavor and the many methodological challenges they may encounter. Failure to do so is likely to result in research and resource waste.

Given their potential impact on people’s lives, it is crucial for evidence syntheses to correctly report on the current knowledge base. In order to be perceived as trustworthy, reliable demonstration of the accuracy of evidence syntheses is equally imperative [ 2 ]. Concerns about the trustworthiness of evidence syntheses are not recent developments. From the early years when EBM first began to gain traction until recent times when thousands of systematic reviews are published monthly [ 3 ] the rigor of evidence syntheses has always varied. Many systematic reviews and meta-analyses had obvious deficiencies because original methods and processes had gaps, lacked precision, and/or were not widely known. The situation has improved with empirical research concerning which methods to use and standardization of appraisal tools. However, given the geometrical increase in the number of evidence syntheses being published, a relatively larger pool of unreliable evidence syntheses is being published today.

Publication of methodological studies that critically appraise the methods used in evidence syntheses is increasing at a fast pace. This reflects the availability of tools specifically developed for this purpose [ 4 – 6 ]. Yet many clinical specialties report that alarming numbers of evidence syntheses fail on these assessments. The syntheses identified report on a broad range of common conditions including, but not limited to, cancer, [ 7 ] chronic obstructive pulmonary disease, [ 8 ] osteoporosis, [ 9 ] stroke, [ 10 ] cerebral palsy, [ 11 ] chronic low back pain, [ 12 ] refractive error, [ 13 ] major depression, [ 14 ] pain, [ 15 ] and obesity [ 16 , 17 ]. The situation is even more concerning with regard to evidence syntheses included in clinical practice guidelines (CPGs) [ 18 – 20 ]. Astonishingly, in a sample of CPGs published in 2017–18, more than half did not apply even basic systematic methods in the evidence syntheses used to inform their recommendations [ 21 ].

These reports, while not widely acknowledged, suggest there are pervasive problems not limited to evidence syntheses that evaluate specific kinds of interventions or include primary research of a particular study design (eg, randomized versus non-randomized) [ 22 ]. Similar concerns about the reliability of evidence syntheses have been expressed by proponents of EBM in highly circulated medical journals [ 23 – 26 ]. These publications have also raised awareness about redundancy, inadequate input of statistical expertise, and deficient reporting. These issues plague primary research as well; however, there is heightened concern for the impact of these deficiencies given the critical role of evidence syntheses in policy and clinical decision-making.

Methods and guidance to produce a reliable evidence synthesis

Several international consortiums of EBM experts and national health care organizations currently provide detailed guidance (Table ​ (Table1). 1 ). They draw criteria from the reporting and methodological standards of currently recommended appraisal tools, and regularly review and update their methods to reflect new information and changing needs. In addition, they endorse the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system for rating the overall quality of a body of evidence [ 27 ]. These groups typically certify or commission systematic reviews that are published in exclusive databases (eg, Cochrane, JBI) or are used to develop government or agency sponsored guidelines or health technology assessments (eg, National Institute for Health and Care Excellence [NICE], Scottish Intercollegiate Guidelines Network [SIGN], Agency for Healthcare Research and Quality [AHRQ]). They offer developers of evidence syntheses various levels of methodological advice, technical and administrative support, and editorial assistance. Use of specific protocols and checklists are required for development teams within these groups, but their online methodological resources are accessible to any potential author.

Guidance for development of evidence syntheses

Notably, Cochrane is the largest single producer of evidence syntheses in biomedical research; however, these only account for 15% of the total [ 28 ]. The World Health Organization requires Cochrane standards be used to develop evidence syntheses that inform their CPGs [ 29 ]. Authors investigating questions of intervention effectiveness in syntheses developed for Cochrane follow the Methodological Expectations of Cochrane Intervention Reviews [ 30 ] and undergo multi-tiered peer review [ 31 , 32 ]. Several empirical evaluations have shown that Cochrane systematic reviews are of higher methodological quality compared with non-Cochrane reviews [ 4 , 7 , 9 , 11 , 14 , 32 – 35 ]. However, some of these assessments have biases: they may be conducted by Cochrane-affiliated authors, and they sometimes use scales and tools developed and used in the Cochrane environment and by its partners. In addition, evidence syntheses published in the Cochrane database are not subject to space or word restrictions, while non-Cochrane syntheses are often limited. As a result, information that may be relevant to the critical appraisal of non-Cochrane reviews is often removed or is relegated to online-only supplements that may not be readily or fully accessible [ 28 ].

Influences on the state of evidence synthesis

Many authors are familiar with the evidence syntheses produced by the leading EBM organizations but can be intimidated by the time and effort necessary to apply their standards. Instead of following their guidance, authors may employ methods that are discouraged or outdated 28]. Suboptimal methods described in in the literature may then be taken up by others. For example, the Newcastle–Ottawa Scale (NOS) is a commonly used tool for appraising non-randomized studies [ 36 ]. Many authors justify their selection of this tool with reference to a publication that describes the unreliability of the NOS and recommends against its use [ 37 ]. Obviously, the authors who cite this report for that purpose have not read it. Authors and peer reviewers have a responsibility to use reliable and accurate methods and not copycat previous citations or substandard work [ 38 , 39 ]. Similar cautions may potentially extend to automation tools. These have concentrated on evidence searching [ 40 ] and selection given how demanding it is for humans to maintain truly up-to-date evidence [ 2 , 41 ]. Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) and studies related to COVID-19, [ 2 , 42 ] but such tools are not yet commonly used [ 43 ]. The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.

Editorials about unreliable or misleading systematic reviews highlight several of the intertwining factors that may contribute to continued publication of unreliable evidence syntheses: shortcomings and inconsistencies of the peer review process, lack of endorsement of current standards on the part of journal editors, the incentive structure of academia, industry influences, publication bias, and the lure of “predatory” journals [ 44 – 48 ]. At this juncture, clarification of the extent to which each of these factors contribute remains speculative, but their impact is likely to be synergistic.

Over time, the generalized acceptance of the conclusions of systematic reviews as incontrovertible has affected trends in the dissemination and uptake of evidence. Reporting of the results of evidence syntheses and recommendations of CPGs has shifted beyond medical journals to press releases and news headlines and, more recently, to the realm of social media and influencers. The lay public and policy makers may depend on these outlets for interpreting evidence syntheses and CPGs. Unfortunately, communication to the general public often reflects intentional or non-intentional misrepresentation or “spin” of the research findings [ 49 – 52 ] News and social media outlets also tend to reduce conclusions on a body of evidence and recommendations for treatment to binary choices (eg, “do it” versus “don’t do it”) that may be assigned an actionable symbol (eg, red/green traffic lights, smiley/frowning face emoji).

Strategies for improvement

Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [ 46 , 53 ]. Informing them about research methodology could increase the likelihood they will apply rigorous methods [ 25 , 33 , 45 ]. We tackle this challenge, from both a theoretical and a practical perspective, by offering guidance applicable to any specialty. It is based on recent methodological research that is extensively referenced to promote self-study. However, the information presented is not intended to be substitute for committed training in evidence synthesis methodology; instead, we hope to inspire our target audience to seek such training. We also hope to inform a broader audience of clinicians and guideline developers influenced by evidence syntheses. Notably, these communities often include the same members who serve in different capacities.

In the following sections, we highlight methodological concepts and practices that may be unfamiliar, problematic, confusing, or controversial. In Part 2, we consider various types of evidence syntheses and the types of research evidence summarized by them. In Part 3, we examine some widely used (and misused) tools for the critical appraisal of systematic reviews and reporting guidelines for evidence syntheses. In Part 4, we discuss how to meet methodological conduct standards applicable to key components of systematic reviews. In Part 5, we describe the merits and caveats of rating the overall certainty of a body of evidence. Finally, in Part 6, we summarize suggested terminology, methods, and tools for development and evaluation of evidence syntheses that reflect current best practices.

Part 2. Types of syntheses and research evidence

A good foundation for the development of evidence syntheses requires an appreciation of their various methodologies and the ability to correctly identify the types of research potentially available for inclusion in the synthesis.

Types of evidence syntheses

Systematic reviews have historically focused on the benefits and harms of interventions; over time, various types of systematic reviews have emerged to address the diverse information needs of clinicians, patients, and policy makers [ 54 ] Systematic reviews with traditional components have become defined by the different topics they assess (Table 2.1 ). In addition, other distinctive types of evidence syntheses have evolved, including overviews or umbrella reviews, scoping reviews, rapid reviews, and living reviews. The popularity of these has been increasing in recent years [ 55 – 58 ]. A summary of the development, methods, available guidance, and indications for these unique types of evidence syntheses is available in Additional File 2 A.

Types of traditional systematic reviews

Both Cochrane [ 30 , 59 ] and JBI [ 60 ] provide methodologies for many types of evidence syntheses; they describe these with different terminology, but there is obvious overlap (Table 2.2 ). The majority of evidence syntheses published by Cochrane (96%) and JBI (62%) are categorized as intervention reviews. This reflects the earlier development and dissemination of their intervention review methodologies; these remain well-established [ 30 , 59 , 61 ] as both organizations continue to focus on topics related to treatment efficacy and harms. In contrast, intervention reviews represent only about half of the total published in the general medical literature, and several non-intervention review types contribute to a significant proportion of the other half.

Evidence syntheses published by Cochrane and JBI

a Data from https://www.cochranelibrary.com/cdsr/reviews . Accessed 17 Sep 2022

b Data obtained via personal email communication on 18 Sep 2022 with Emilie Francis, editorial assistant, JBI Evidence Synthesis

c Includes the following categories: prevalence, scoping, mixed methods, and realist reviews

d This methodology is not supported in the current version of the JBI Manual for Evidence Synthesis

Types of research evidence

There is consensus on the importance of using multiple study designs in evidence syntheses; at the same time, there is a lack of agreement on methods to identify included study designs. Authors of evidence syntheses may use various taxonomies and associated algorithms to guide selection and/or classification of study designs. These tools differentiate categories of research and apply labels to individual study designs (eg, RCT, cross-sectional). A familiar example is the Design Tree endorsed by the Centre for Evidence-Based Medicine [ 70 ]. Such tools may not be helpful to authors of evidence syntheses for multiple reasons.

Suboptimal levels of agreement and accuracy even among trained methodologists reflect challenges with the application of such tools [ 71 , 72 ]. Problematic distinctions or decision points (eg, experimental or observational, controlled or uncontrolled, prospective or retrospective) and design labels (eg, cohort, case control, uncontrolled trial) have been reported [ 71 ]. The variable application of ambiguous study design labels to non-randomized studies is common, making them especially prone to misclassification [ 73 ]. In addition, study labels do not denote the unique design features that make different types of non-randomized studies susceptible to different biases, including those related to how the data are obtained (eg, clinical trials, disease registries, wearable devices). Given this limitation, it is important to be aware that design labels preclude the accurate assignment of non-randomized studies to a “level of evidence” in traditional hierarchies [ 74 ].

These concerns suggest that available tools and nomenclature used to distinguish types of research evidence may not uniformly apply to biomedical research and non-health fields that utilize evidence syntheses (eg, education, economics) [ 75 , 76 ]. Moreover, primary research reports often do not describe study design or do so incompletely or inaccurately; thus, indexing in PubMed and other databases does not address the potential for misclassification [ 77 ]. Yet proper identification of research evidence has implications for several key components of evidence syntheses. For example, search strategies limited by index terms using design labels or study selection based on labels applied by the authors of primary studies may cause inconsistent or unjustified study inclusions and/or exclusions [ 77 ]. In addition, because risk of bias (RoB) tools consider attributes specific to certain types of studies and study design features, results of these assessments may be invalidated if an inappropriate tool is used. Appropriate classification of studies is also relevant for the selection of a suitable method of synthesis and interpretation of those results.

An alternative to these tools and nomenclature involves application of a few fundamental distinctions that encompass a wide range of research designs and contexts. While these distinctions are not novel, we integrate them into a practical scheme (see Fig. ​ Fig.1) 1 ) designed to guide authors of evidence syntheses in the basic identification of research evidence. The initial distinction is between primary and secondary studies. Primary studies are then further distinguished by: 1) the type of data reported (qualitative or quantitative); and 2) two defining design features (group or single-case and randomized or non-randomized). The different types of studies and study designs represented in the scheme are described in detail in Additional File 2 B. It is important to conceptualize their methods as complementary as opposed to contrasting or hierarchical [ 78 ]; each offers advantages and disadvantages that determine their appropriateness for answering different kinds of research questions in an evidence synthesis.

Distinguishing types of research evidence

Application of these basic distinctions may avoid some of the potential difficulties associated with study design labels and taxonomies. Nevertheless, debatable methodological issues are raised when certain types of research identified in this scheme are included in an evidence synthesis. We briefly highlight those associated with inclusion of non-randomized studies, case reports and series, and a combination of primary and secondary studies.

Non-randomized studies

When investigating an intervention’s effectiveness, it is important for authors to recognize the uncertainty of observed effects reported by studies with high RoB. Results of statistical analyses that include such studies need to be interpreted with caution in order to avoid misleading conclusions [ 74 ]. Review authors may consider excluding randomized studies with high RoB from meta-analyses. Non-randomized studies of intervention (NRSI) are affected by a greater potential range of biases and thus vary more than RCTs in their ability to estimate a causal effect [ 79 ]. If data from NRSI are synthesized in meta-analyses, it is helpful to separately report their summary estimates [ 6 , 74 ].

Nonetheless, certain design features of NRSI (eg, which parts of the study were prospectively designed) may help to distinguish stronger from weaker ones. Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [ 79 , 80 ] This process is facilitated by a study design feature checklist; guidance on using the checklist is included with developers’ description of the tool [ 73 , 74 ]. Authors collect information about these design features during data extraction and then consider it when making final study selection decisions and when performing RoB assessments of the included NRSI.

Case reports and case series

Correctly identified case reports and case series can contribute evidence not well captured by other designs [ 81 ]; in addition, some topics may be limited to a body of evidence that consists primarily of uncontrolled clinical observations. Murad and colleagues offer a framework for how to include case reports and series in an evidence synthesis [ 82 ]. Distinguishing between cohort studies and case series in these syntheses is important, especially for those that rely on evidence from NRSI. Additional data obtained from studies misclassified as case series can potentially increase the confidence in effect estimates. Mathes and Pieper provide authors of evidence syntheses with specific guidance on distinguishing between cohort studies and case series, but emphasize the increased workload involved [ 77 ].

Primary and secondary studies

Synthesis of combined evidence from primary and secondary studies may provide a broad perspective on the entirety of available literature on a topic. This is, in fact, the recommended strategy for scoping reviews that may include a variety of sources of evidence (eg, CPGs, popular media). However, except for scoping reviews, the synthesis of data from primary and secondary studies is discouraged unless there are strong reasons to justify doing so.

Combining primary and secondary sources of evidence is challenging for authors of other types of evidence syntheses for several reasons [ 83 ]. Assessments of RoB for primary and secondary studies are derived from conceptually different tools, thus obfuscating the ability to make an overall RoB assessment of a combination of these study types. In addition, authors who include primary and secondary studies must devise non-standardized methods for synthesis. Note this contrasts with well-established methods available for updating existing evidence syntheses with additional data from new primary studies [ 84 – 86 ]. However, a new review that synthesizes data from primary and secondary studies raises questions of validity and may unintentionally support a biased conclusion because no existing methodological guidance is currently available [ 87 ].

Recommendations

We suggest that journal editors require authors to identify which type of evidence synthesis they are submitting and reference the specific methodology used for its development. This will clarify the research question and methods for peer reviewers and potentially simplify the editorial process. Editors should announce this practice and include it in the instructions to authors. To decrease bias and apply correct methods, authors must also accurately identify the types of research evidence included in their syntheses.

Part 3. Conduct and reporting

The need to develop criteria to assess the rigor of systematic reviews was recognized soon after the EBM movement began to gain international traction [ 88 , 89 ]. Systematic reviews rapidly became popular, but many were very poorly conceived, conducted, and reported. These problems remain highly prevalent [ 23 ] despite development of guidelines and tools to standardize and improve the performance and reporting of evidence syntheses [ 22 , 28 ]. Table 3.1  provides some historical perspective on the evolution of tools developed specifically for the evaluation of systematic reviews, with or without meta-analysis.

Tools specifying standards for systematic reviews with and without meta-analysis

a Currently recommended

b Validated tool for systematic reviews of interventions developed for use by authors of overviews or umbrella reviews

These tools are often interchangeably invoked when referring to the “quality” of an evidence synthesis. However, quality is a vague term that is frequently misused and misunderstood; more precisely, these tools specify different standards for evidence syntheses. Methodological standards address how well a systematic review was designed and performed [ 5 ]. RoB assessments refer to systematic flaws or limitations in the design, conduct, or analysis of research that distort the findings of the review [ 4 ]. Reporting standards help systematic review authors describe the methodology they used and the results of their synthesis in sufficient detail [ 92 ]. It is essential to distinguish between these evaluations: a systematic review may be biased, it may fail to report sufficient information on essential features, or it may exhibit both problems; a thoroughly reported systematic evidence synthesis review may still be biased and flawed while an otherwise unbiased one may suffer from deficient documentation.

We direct attention to the currently recommended tools listed in Table 3.1  but concentrate on AMSTAR-2 (update of AMSTAR [A Measurement Tool to Assess Systematic Reviews]) and ROBIS (Risk of Bias in Systematic Reviews), which evaluate methodological quality and RoB, respectively. For comparison and completeness, we include PRISMA 2020 (update of the 2009 Preferred Reporting Items for Systematic Reviews of Meta-Analyses statement), which offers guidance on reporting standards. The exclusive focus on these three tools is by design; it addresses concerns related to the considerable variability in tools used for the evaluation of systematic reviews [ 28 , 88 , 96 , 97 ]. We highlight the underlying constructs these tools were designed to assess, then describe their components and applications. Their known (or potential) uptake and impact and limitations are also discussed.

Evaluation of conduct

Development.

AMSTAR [ 5 ] was in use for a decade prior to the 2017 publication of AMSTAR-2; both provide a broad evaluation of methodological quality of intervention systematic reviews, including flaws arising through poor conduct of the review [ 6 ]. ROBIS, published in 2016, was developed to specifically assess RoB introduced by the conduct of the review; it is applicable to systematic reviews of interventions and several other types of reviews [ 4 ]. Both tools reflect a shift to a domain-based approach as opposed to generic quality checklists. There are a few items unique to each tool; however, similarities between items have been demonstrated [ 98 , 99 ]. AMSTAR-2 and ROBIS are recommended for use by: 1) authors of overviews or umbrella reviews and CPGs to evaluate systematic reviews considered as evidence; 2) authors of methodological research studies to appraise included systematic reviews; and 3) peer reviewers for appraisal of submitted systematic review manuscripts. For authors, these tools may function as teaching aids and inform conduct of their review during its development.

Description

Systematic reviews that include randomized and/or non-randomized studies as evidence can be appraised with AMSTAR-2 and ROBIS. Other characteristics of AMSTAR-2 and ROBIS are summarized in Table 3.2 . Both tools define categories for an overall rating; however, neither tool is intended to generate a total score by simply calculating the number of responses satisfying criteria for individual items [ 4 , 6 ]. AMSTAR-2 focuses on the rigor of a review’s methods irrespective of the specific subject matter. ROBIS places emphasis on a review’s results section— this suggests it may be optimally applied by appraisers with some knowledge of the review’s topic as they may be better equipped to determine if certain procedures (or lack thereof) would impact the validity of a review’s findings [ 98 , 100 ]. Reliability studies show AMSTAR-2 overall confidence ratings strongly correlate with the overall RoB ratings in ROBIS [ 100 , 101 ].

Comparison of AMSTAR-2 and ROBIS

a ROBIS includes an optional first phase to assess the applicability of the review to the research question of interest. The tool may be applicable to other review types in addition to the four specified, although modification of this initial phase will be needed (Personal Communication via email, Penny Whiting, 28 Jan 2022)

b AMSTAR-2 item #9 and #11 require separate responses for RCTs and NRSI

Interrater reliability has been shown to be acceptable for AMSTAR-2 [ 6 , 11 , 102 ] and ROBIS [ 4 , 98 , 103 ] but neither tool has been shown to be superior in this regard [ 100 , 101 , 104 , 105 ]. Overall, variability in reliability for both tools has been reported across items, between pairs of raters, and between centers [ 6 , 100 , 101 , 104 ]. The effects of appraiser experience on the results of AMSTAR-2 and ROBIS require further evaluation [ 101 , 105 ]. Updates to both tools should address items shown to be prone to individual appraisers’ subjective biases and opinions [ 11 , 100 ]; this may involve modifications of the current domains and signaling questions as well as incorporation of methods to make an appraiser’s judgments more explicit. Future revisions of these tools may also consider the addition of standards for aspects of systematic review development currently lacking (eg, rating overall certainty of evidence, [ 99 ] methods for synthesis without meta-analysis [ 105 ]) and removal of items that assess aspects of reporting that are thoroughly evaluated by PRISMA 2020.

Application

A good understanding of what is required to satisfy the standards of AMSTAR-2 and ROBIS involves study of the accompanying guidance documents written by the tools’ developers; these contain detailed descriptions of each item’s standards. In addition, accurate appraisal of a systematic review with either tool requires training. Most experts recommend independent assessment by at least two appraisers with a process for resolving discrepancies as well as procedures to establish interrater reliability, such as pilot testing, a calibration phase or exercise, and development of predefined decision rules [ 35 , 99 – 101 , 103 , 104 , 106 ]. These methods may, to some extent, address the challenges associated with the diversity in methodological training, subject matter expertise, and experience using the tools that are likely to exist among appraisers.

The standards of AMSTAR, AMSTAR-2, and ROBIS have been used in many methodological studies and epidemiological investigations. However, the increased publication of overviews or umbrella reviews and CPGs has likely been a greater influence on the widening acceptance of these tools. Critical appraisal of the secondary studies considered evidence is essential to the trustworthiness of both the recommendations of CPGs and the conclusions of overviews. Currently both Cochrane [ 55 ] and JBI [ 107 ] recommend AMSTAR-2 and ROBIS in their guidance for authors of overviews or umbrella reviews. However, ROBIS and AMSTAR-2 were released in 2016 and 2017, respectively; thus, to date, limited data have been reported about the uptake of these tools or which of the two may be preferred [ 21 , 106 ]. Currently, in relation to CPGs, AMSTAR-2 appears to be overwhelmingly popular compared to ROBIS. A Google Scholar search of this topic (search terms “AMSTAR 2 AND clinical practice guidelines,” “ROBIS AND clinical practice guidelines” 13 May 2022) found 12,700 hits for AMSTAR-2 and 1,280 for ROBIS. The apparent greater appeal of AMSTAR-2 may relate to its longer track record given the original version of the tool was in use for 10 years prior to its update in 2017.

Barriers to the uptake of AMSTAR-2 and ROBIS include the real or perceived time and resources necessary to complete the items they include and appraisers’ confidence in their own ratings [ 104 ]. Reports from comparative studies available to date indicate that appraisers find AMSTAR-2 questions, responses, and guidance to be clearer and simpler compared with ROBIS [ 11 , 101 , 104 , 105 ]. This suggests that for appraisal of intervention systematic reviews, AMSTAR-2 may be a more practical tool than ROBIS, especially for novice appraisers [ 101 , 103 – 105 ]. The unique characteristics of each tool, as well as their potential advantages and disadvantages, should be taken into consideration when deciding which tool should be used for an appraisal of a systematic review. In addition, the choice of one or the other may depend on how the results of an appraisal will be used; for example, a peer reviewer’s appraisal of a single manuscript versus an appraisal of multiple systematic reviews in an overview or umbrella review, CPG, or systematic methodological study.

Authors of overviews and CPGs report results of AMSTAR-2 and ROBIS appraisals for each of the systematic reviews they include as evidence. Ideally, an independent judgment of their appraisals can be made by the end users of overviews and CPGs; however, most stakeholders, including clinicians, are unlikely to have a sophisticated understanding of these tools. Nevertheless, they should at least be aware that AMSTAR-2 and ROBIS ratings reported in overviews and CPGs may be inaccurate because the tools are not applied as intended by their developers. This can result from inadequate training of the overview or CPG authors who perform the appraisals, or to modifications of the appraisal tools imposed by them. The potential variability in overall confidence and RoB ratings highlights why appraisers applying these tools need to support their judgments with explicit documentation; this allows readers to judge for themselves whether they agree with the criteria used by appraisers [ 4 , 108 ]. When these judgments are explicit, the underlying rationale used when applying these tools can be assessed [ 109 ].

Theoretically, we would expect an association of AMSTAR-2 with improved methodological rigor and an association of ROBIS with lower RoB in recent systematic reviews compared to those published before 2017. To our knowledge, this has not yet been demonstrated; however, like reports about the actual uptake of these tools, time will tell. Additional data on user experience is also needed to further elucidate the practical challenges and methodological nuances encountered with the application of these tools. This information could potentially inform the creation of unifying criteria to guide and standardize the appraisal of evidence syntheses [ 109 ].

Evaluation of reporting

Complete reporting is essential for users to establish the trustworthiness and applicability of a systematic review’s findings. Efforts to standardize and improve the reporting of systematic reviews resulted in the 2009 publication of the PRISMA statement [ 92 ] with its accompanying explanation and elaboration document [ 110 ]. This guideline was designed to help authors prepare a complete and transparent report of their systematic review. In addition, adherence to PRISMA is often used to evaluate the thoroughness of reporting of published systematic reviews [ 111 ]. The updated version, PRISMA 2020 [ 93 ], and its guidance document [ 112 ] were published in 2021. Items on the original and updated versions of PRISMA are organized by the six basic review components they address (title, abstract, introduction, methods, results, discussion). The PRISMA 2020 update is a considerably expanded version of the original; it includes standards and examples for the 27 original and 13 additional reporting items that capture methodological advances and may enhance the replicability of reviews [ 113 ].

The original PRISMA statement fostered the development of various PRISMA extensions (Table 3.3 ). These include reporting guidance for scoping reviews and reviews of diagnostic test accuracy and for intervention reviews that report on the following: harms outcomes, equity issues, the effects of acupuncture, the results of network meta-analyses and analyses of individual participant data. Detailed reporting guidance for specific systematic review components (abstracts, protocols, literature searches) is also available.

PRISMA extensions

PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses

a Note the abstract reporting checklist is now incorporated into PRISMA 2020 [ 93 ]

Uptake and impact

The 2009 PRISMA standards [ 92 ] for reporting have been widely endorsed by authors, journals, and EBM-related organizations. We anticipate the same for PRISMA 2020 [ 93 ] given its co-publication in multiple high-impact journals. However, to date, there is a lack of strong evidence for an association between improved systematic review reporting and endorsement of PRISMA 2009 standards [ 43 , 111 ]. Most journals require a PRISMA checklist accompany submissions of systematic review manuscripts. However, the accuracy of information presented on these self-reported checklists is not necessarily verified. It remains unclear which strategies (eg, authors’ self-report of checklists, peer reviewer checks) might improve adherence to the PRISMA reporting standards; in addition, the feasibility of any potentially effective strategies must be taken into consideration given the structure and limitations of current research and publication practices [ 124 ].

Pitfalls and limitations of PRISMA, AMSTAR-2, and ROBIS

Misunderstanding of the roles of these tools and their misapplication may be widespread problems. PRISMA 2020 is a reporting guideline that is most beneficial if consulted when developing a review as opposed to merely completing a checklist when submitting to a journal; at that point, the review is finished, with good or bad methodological choices. However, PRISMA checklists evaluate how completely an element of review conduct was reported, but do not evaluate the caliber of conduct or performance of a review. Thus, review authors and readers should not think that a rigorous systematic review can be produced by simply following the PRISMA 2020 guidelines. Similarly, it is important to recognize that AMSTAR-2 and ROBIS are tools to evaluate the conduct of a review but do not substitute for conceptual methodological guidance. In addition, they are not intended to be simple checklists. In fact, they have the potential for misuse or abuse if applied as such; for example, by calculating a total score to make a judgment about a review’s overall confidence or RoB. Proper selection of a response for the individual items on AMSTAR-2 and ROBIS requires training or at least reference to their accompanying guidance documents.

Not surprisingly, it has been shown that compliance with the PRISMA checklist is not necessarily associated with satisfying the standards of ROBIS [ 125 ]. AMSTAR-2 and ROBIS were not available when PRISMA 2009 was developed; however, they were considered in the development of PRISMA 2020 [ 113 ]. Therefore, future studies may show a positive relationship between fulfillment of PRISMA 2020 standards for reporting and meeting the standards of tools evaluating methodological quality and RoB.

Choice of an appropriate tool for the evaluation of a systematic review first involves identification of the underlying construct to be assessed. For systematic reviews of interventions, recommended tools include AMSTAR-2 and ROBIS for appraisal of conduct and PRISMA 2020 for completeness of reporting. All three tools were developed rigorously and provide easily accessible and detailed user guidance, which is necessary for their proper application and interpretation. When considering a manuscript for publication, training in these tools can sensitize peer reviewers and editors to major issues that may affect the review’s trustworthiness and completeness of reporting. Judgment of the overall certainty of a body of evidence and formulation of recommendations rely, in part, on AMSTAR-2 or ROBIS appraisals of systematic reviews. Therefore, training on the application of these tools is essential for authors of overviews and developers of CPGs. Peer reviewers and editors considering an overview or CPG for publication must hold their authors to a high standard of transparency regarding both the conduct and reporting of these appraisals.

Part 4. Meeting conduct standards

Many authors, peer reviewers, and editors erroneously equate fulfillment of the items on the PRISMA checklist with superior methodological rigor. For direction on methodology, we refer them to available resources that provide comprehensive conceptual guidance [ 59 , 60 ] as well as primers with basic step-by-step instructions [ 1 , 126 , 127 ]. This section is intended to complement study of such resources by facilitating use of AMSTAR-2 and ROBIS, tools specifically developed to evaluate methodological rigor of systematic reviews. These tools are widely accepted by methodologists; however, in the general medical literature, they are not uniformly selected for the critical appraisal of systematic reviews [ 88 , 96 ].

To enable their uptake, Table 4.1  links review components to the corresponding appraisal tool items. Expectations of AMSTAR-2 and ROBIS are concisely stated, and reasoning provided.

Systematic review components linked to appraisal with AMSTAR-2 and ROBIS a

CoI conflict of interest, MA meta-analysis, NA not addressed, PICO participant, intervention, comparison, outcome, PRISMA-P Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols, RoB risk of bias

a Components shown in bold are chosen for elaboration in Part 4 for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors; and/or 2) the component is evaluated by standards of an AMSTAR-2 “critical” domain

b Critical domains of AMSTAR-2 are indicated by *

Issues involved in meeting the standards for seven review components (identified in bold in Table 4.1 ) are addressed in detail. These were chosen for elaboration for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors based on consistent reports of their frequent AMSTAR-2 or ROBIS deficiencies [ 9 , 11 , 15 , 88 , 128 , 129 ]; and/or 2) the review component is judged by standards of an AMSTAR-2 “critical” domain. These have the greatest implications for how a systematic review will be appraised: if standards for any one of these critical domains are not met, the review is rated as having “critically low confidence.”

Research question

Specific and unambiguous research questions may have more value for reviews that deal with hypothesis testing. Mnemonics for the various elements of research questions are suggested by JBI and Cochrane (Table 2.1 ). These prompt authors to consider the specialized methods involved for developing different types of systematic reviews; however, while inclusion of the suggested elements makes a review compliant with a particular review’s methods, it does not necessarily make a research question appropriate. Table 4.2  lists acronyms that may aid in developing the research question. They include overlapping concepts of importance in this time of proliferating reviews of uncertain value [ 130 ]. If these issues are not prospectively contemplated, systematic review authors may establish an overly broad scope, or develop runaway scope allowing them to stray from predefined choices relating to key comparisons and outcomes.

Research question development

a Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Hulley SB, Cummings SR, Browner WS, editors. Designing clinical research: an epidemiological approach; 4th edn. Lippincott Williams & Wilkins; 2007. p. 14–22

b Doran, GT. There’s a S.M.A.R.T. way to write management’s goals and objectives. Manage Rev. 1981;70:35-6.

c Johnson BT, Hennessy EA. Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc Sci Med. 2019;233:237–51

Once a research question is established, searching on registry sites and databases for existing systematic reviews addressing the same or a similar topic is necessary in order to avoid contributing to research waste [ 131 ]. Repeating an existing systematic review must be justified, for example, if previous reviews are out of date or methodologically flawed. A full discussion on replication of intervention systematic reviews, including a consensus checklist, can be found in the work of Tugwell and colleagues [ 84 ].

Protocol development is considered a core component of systematic reviews [ 125 , 126 , 132 ]. Review protocols may allow researchers to plan and anticipate potential issues, assess validity of methods, prevent arbitrary decision-making, and minimize bias that can be introduced by the conduct of the review. Registration of a protocol that allows public access promotes transparency of the systematic review’s methods and processes and reduces the potential for duplication [ 132 ]. Thinking early and carefully about all the steps of a systematic review is pragmatic and logical and may mitigate the influence of the authors’ prior knowledge of the evidence [ 133 ]. In addition, the protocol stage is when the scope of the review can be carefully considered by authors, reviewers, and editors; this may help to avoid production of overly ambitious reviews that include excessive numbers of comparisons and outcomes or are undisciplined in their study selection.

An association with attainment of AMSTAR standards in systematic reviews with published prospective protocols has been reported [ 134 ]. However, completeness of reporting does not seem to be different in reviews with a protocol compared to those without one [ 135 ]. PRISMA-P [ 116 ] and its accompanying elaboration and explanation document [ 136 ] can be used to guide and assess the reporting of protocols. A final version of the review should fully describe any protocol deviations. Peer reviewers may compare the submitted manuscript with any available pre-registered protocol; this is required if AMSTAR-2 or ROBIS are used for critical appraisal.

There are multiple options for the recording of protocols (Table 4.3 ). Some journals will peer review and publish protocols. In addition, many online sites offer date-stamped and publicly accessible protocol registration. Some of these are exclusively for protocols of evidence syntheses; others are less restrictive and offer researchers the capacity for data storage, sharing, and other workflow features. These sites document protocol details to varying extents and have different requirements [ 137 ]. The most popular site for systematic reviews, the International Prospective Register of Systematic Reviews (PROSPERO), for example, only registers reviews that report on an outcome with direct relevance to human health. The PROSPERO record documents protocols for all types of reviews except literature and scoping reviews. Of note, PROSPERO requires authors register their review protocols prior to any data extraction [ 133 , 138 ]. The electronic records of most of these registry sites allow authors to update their protocols and facilitate transparent tracking of protocol changes, which are not unexpected during the progress of the review [ 139 ].

Options for protocol registration of evidence syntheses

a Authors are advised to contact their target journal regarding submission of systematic review protocols

b Registration is restricted to approved review projects

c The JBI registry lists review projects currently underway by JBI-affiliated entities. These records include a review’s title, primary author, research question, and PICO elements. JBI recommends that authors register eligible protocols with PROSPERO

d See Pieper and Rombey [ 137 ] for detailed characteristics of these five registries

e See Pieper and Rombey [ 137 ] for other systematic review data repository options

Study design inclusion

For most systematic reviews, broad inclusion of study designs is recommended [ 126 ]. This may allow comparison of results between contrasting study design types [ 126 ]. Certain study designs may be considered preferable depending on the type of review and nature of the research question. However, prevailing stereotypes about what each study design does best may not be accurate. For example, in systematic reviews of interventions, randomized designs are typically thought to answer highly specific questions while non-randomized designs often are expected to reveal greater information about harms or real-word evidence [ 126 , 140 , 141 ]. This may be a false distinction; randomized trials may be pragmatic [ 142 ], they may offer important (and more unbiased) information on harms [ 143 ], and data from non-randomized trials may not necessarily be more real-world-oriented [ 144 ].

Moreover, there may not be any available evidence reported by RCTs for certain research questions; in some cases, there may not be any RCTs or NRSI. When the available evidence is limited to case reports and case series, it is not possible to test hypotheses nor provide descriptive estimates or associations; however, a systematic review of these studies can still offer important insights [ 81 , 145 ]. When authors anticipate that limited evidence of any kind may be available to inform their research questions, a scoping review can be considered. Alternatively, decisions regarding inclusion of indirect as opposed to direct evidence can be addressed during protocol development [ 146 ]. Including indirect evidence at an early stage of intervention systematic review development allows authors to decide if such studies offer any additional and/or different understanding of treatment effects for their population or comparison of interest. Issues of indirectness of included studies are accounted for later in the process, during determination of the overall certainty of evidence (see Part 5 for details).

Evidence search

Both AMSTAR-2 and ROBIS require systematic and comprehensive searches for evidence. This is essential for any systematic review. Both tools discourage search restrictions based on language and publication source. Given increasing globalism in health care, the practice of including English-only literature should be avoided [ 126 ]. There are many examples in which language bias (different results in studies published in different languages) has been documented [ 147 , 148 ]. This does not mean that all literature, in all languages, is equally trustworthy [ 148 ]; however, the only way to formally probe for the potential of such biases is to consider all languages in the initial search. The gray literature and a search of trials may also reveal important details about topics that would otherwise be missed [ 149 – 151 ]. Again, inclusiveness will allow review authors to investigate whether results differ in gray literature and trials [ 41 , 151 – 153 ].

Authors should make every attempt to complete their review within one year as that is the likely viable life of a search. (1) If that is not possible, the search should be updated close to the time of completion [ 154 ]. Different research topics may warrant less of a delay, for example, in rapidly changing fields (as in the case of the COVID-19 pandemic), even one month may radically change the available evidence.

Excluded studies

AMSTAR-2 requires authors to provide references for any studies excluded at the full text phase of study selection along with reasons for exclusion; this allows readers to feel confident that all relevant literature has been considered for inclusion and that exclusions are defensible.

Risk of bias assessment of included studies

The design of the studies included in a systematic review (eg, RCT, cohort, case series) should not be equated with appraisal of its RoB. To meet AMSTAR-2 and ROBIS standards, systematic review authors must examine RoB issues specific to the design of each primary study they include as evidence. It is unlikely that a single RoB appraisal tool will be suitable for all research designs. In addition to tools for randomized and non-randomized studies, specific tools are available for evaluation of RoB in case reports and case series [ 82 ] and single-case experimental designs [ 155 , 156 ]. Note the RoB tools selected must meet the standards of the appraisal tool used to judge the conduct of the review. For example, AMSTAR-2 identifies four sources of bias specific to RCTs and NRSI that must be addressed by the RoB tool(s) chosen by the review authors. The Cochrane RoB-2 [ 157 ] tool for RCTs and ROBINS-I [ 158 ] for NRSI for RoB assessment meet the AMSTAR-2 standards. Appraisers on the review team should not modify any RoB tool without complete transparency and acknowledgment that they have invalidated the interpretation of the tool as intended by its developers [ 159 ]. Conduct of RoB assessments is not addressed AMSTAR-2; to meet ROBIS standards, two independent reviewers should complete RoB assessments of included primary studies.

Implications of the RoB assessments must be explicitly discussed and considered in the conclusions of the review. Discussion of the overall RoB of included studies may consider the weight of the studies at high RoB, the importance of the sources of bias in the studies being summarized, and if their importance differs in relationship to the outcomes reported. If a meta-analysis is performed, serious concerns for RoB of individual studies should be accounted for in these results as well. If the results of the meta-analysis for a specific outcome change when studies at high RoB are excluded, readers will have a more accurate understanding of this body of evidence. However, while investigating the potential impact of specific biases is a useful exercise, it is important to avoid over-interpretation, especially when there are sparse data.

Synthesis methods for quantitative data

Syntheses of quantitative data reported by primary studies are broadly categorized as one of two types: meta-analysis, and synthesis without meta-analysis (Table 4.4 ). Before deciding on one of these methods, authors should seek methodological advice about whether reported data can be transformed or used in other ways to provide a consistent effect measure across studies [ 160 , 161 ].

Common methods for quantitative synthesis

CI confidence interval (or credible interval, if analysis is done in Bayesian framework)

a See text for descriptions of the types of data combined in each of these approaches

b See Additional File 4  for guidance on the structure and presentation of forest plots

c General approach is similar to aggregate data meta-analysis but there are substantial differences relating to data collection and checking and analysis [ 162 ]. This approach to syntheses is applicable to intervention, diagnostic, and prognostic systematic reviews [ 163 ]

d Examples include meta-regression, hierarchical and multivariate approaches [ 164 ]

e In-depth guidance and illustrations of these methods are provided in Chapter 12 of the Cochrane Handbook [ 160 ]

Meta-analysis

Systematic reviews that employ meta-analysis should not be referred to simply as “meta-analyses.” The term meta-analysis strictly refers to a specific statistical technique used when study effect estimates and their variances are available, yielding a quantitative summary of results. In general, methods for meta-analysis involve use of a weighted average of effect estimates from two or more studies. If considered carefully, meta-analysis increases the precision of the estimated magnitude of effect and can offer useful insights about heterogeneity and estimates of effects. We refer to standard references for a thorough introduction and formal training [ 165 – 167 ].

There are three common approaches to meta-analysis in current health care–related systematic reviews (Table 4.4 ). Aggregate meta-analyses is the most familiar to authors of evidence syntheses and their end users. This standard meta-analysis combines data on effect estimates reported by studies that investigate similar research questions involving direct comparisons of an intervention and comparator. Results of these analyses provide a single summary intervention effect estimate. If the included studies in a systematic review measure an outcome differently, their reported results may be transformed to make them comparable [ 161 ]. Forest plots visually present essential information about the individual studies and the overall pooled analysis (see Additional File 4  for details).

Less familiar and more challenging meta-analytical approaches used in secondary research include individual participant data (IPD) and network meta-analyses (NMA); PRISMA extensions provide reporting guidelines for both [ 117 , 118 ]. In IPD, the raw data on each participant from each eligible study are re-analyzed as opposed to the study-level data analyzed in aggregate data meta-analyses [ 168 ]. This may offer advantages, including the potential for limiting concerns about bias and allowing more robust analyses [ 163 ]. As suggested by the description in Table 4.4 , NMA is a complex statistical approach. It combines aggregate data [ 169 ] or IPD [ 170 ] for effect estimates from direct and indirect comparisons reported in two or more studies of three or more interventions. This makes it a potentially powerful statistical tool; while multiple interventions are typically available to treat a condition, few have been evaluated in head-to-head trials [ 171 ]. Both IPD and NMA facilitate a broader scope, and potentially provide more reliable and/or detailed results; however, compared with standard aggregate data meta-analyses, their methods are more complicated, time-consuming, and resource-intensive, and they have their own biases, so one needs sufficient funding, technical expertise, and preparation to employ them successfully [ 41 , 172 , 173 ].

Several items in AMSTAR-2 and ROBIS address meta-analysis; thus, understanding the strengths, weaknesses, assumptions, and limitations of methods for meta-analyses is important. According to the standards of both tools, plans for a meta-analysis must be addressed in the review protocol, including reasoning, description of the type of quantitative data to be synthesized, and the methods planned for combining the data. This should not consist of stock statements describing conventional meta-analysis techniques; rather, authors are expected to anticipate issues specific to their research questions. Concern for the lack of training in meta-analysis methods among systematic review authors cannot be overstated. For those with training, the use of popular software (eg, RevMan [ 174 ], MetaXL [ 175 ], JBI SUMARI [ 176 ]) may facilitate exploration of these methods; however, such programs cannot substitute for the accurate interpretation of the results of meta-analyses, especially for more complex meta-analytical approaches.

Synthesis without meta-analysis

There are varied reasons a meta-analysis may not be appropriate or desirable [ 160 , 161 ]. Syntheses that informally use statistical methods other than meta-analysis are variably referred to as descriptive, narrative, or qualitative syntheses or summaries; these terms are also applied to syntheses that make no attempt to statistically combine data from individual studies. However, use of such imprecise terminology is discouraged; in order to fully explore the results of any type of synthesis, some narration or description is needed to supplement the data visually presented in tabular or graphic forms [ 63 , 177 ]. In addition, the term “qualitative synthesis” is easily confused with a synthesis of qualitative data in a qualitative or mixed methods review. “Synthesis without meta-analysis” is currently the preferred description of other ways to combine quantitative data from two or more studies. Use of this specific terminology when referring to these types of syntheses also implies the application of formal methods (Table 4.4 ).

Methods for syntheses without meta-analysis involve structured presentations of the data in any tables and plots. In comparison to narrative descriptions of each study, these are designed to more effectively and transparently show patterns and convey detailed information about the data; they also allow informal exploration of heterogeneity [ 178 ]. In addition, acceptable quantitative statistical methods (Table 4.4 ) are formally applied; however, it is important to recognize these methods have significant limitations for the interpretation of the effectiveness of an intervention [ 160 ]. Nevertheless, when meta-analysis is not possible, the application of these methods is less prone to bias compared with an unstructured narrative description of included studies [ 178 , 179 ].

Vote counting is commonly used in systematic reviews and involves a tally of studies reporting results that meet some threshold of importance applied by review authors. Until recently, it has not typically been identified as a method for synthesis without meta-analysis. Guidance on an acceptable vote counting method based on direction of effect is currently available [ 160 ] and should be used instead of narrative descriptions of such results (eg, “more than half the studies showed improvement”; “only a few studies reported adverse effects”; “7 out of 10 studies favored the intervention”). Unacceptable methods include vote counting by statistical significance or magnitude of effect or some subjective rule applied by the authors.

AMSTAR-2 and ROBIS standards do not explicitly address conduct of syntheses without meta-analysis, although AMSTAR-2 items 13 and 14 might be considered relevant. Guidance for the complete reporting of syntheses without meta-analysis for systematic reviews of interventions is available in the Synthesis without Meta-analysis (SWiM) guideline [ 180 ] and methodological guidance is available in the Cochrane Handbook [ 160 , 181 ].

Familiarity with AMSTAR-2 and ROBIS makes sense for authors of systematic reviews as these appraisal tools will be used to judge their work; however, training is necessary for authors to truly appreciate and apply methodological rigor. Moreover, judgment of the potential contribution of a systematic review to the current knowledge base goes beyond meeting the standards of AMSTAR-2 and ROBIS. These tools do not explicitly address some crucial concepts involved in the development of a systematic review; this further emphasizes the need for author training.

We recommend that systematic review authors incorporate specific practices or exercises when formulating a research question at the protocol stage, These should be designed to raise the review team’s awareness of how to prevent research and resource waste [ 84 , 130 ] and to stimulate careful contemplation of the scope of the review [ 30 ]. Authors’ training should also focus on justifiably choosing a formal method for the synthesis of quantitative and/or qualitative data from primary research; both types of data require specific expertise. For typical reviews that involve syntheses of quantitative data, statistical expertise is necessary, initially for decisions about appropriate methods, [ 160 , 161 ] and then to inform any meta-analyses [ 167 ] or other statistical methods applied [ 160 ].

Part 5. Rating overall certainty of evidence

Report of an overall certainty of evidence assessment in a systematic review is an important new reporting standard of the updated PRISMA 2020 guidelines [ 93 ]. Systematic review authors are well acquainted with assessing RoB in individual primary studies, but much less familiar with assessment of overall certainty across an entire body of evidence. Yet a reliable way to evaluate this broader concept is now recognized as a vital part of interpreting the evidence.

Historical systems for rating evidence are based on study design and usually involve hierarchical levels or classes of evidence that use numbers and/or letters to designate the level/class. These systems were endorsed by various EBM-related organizations. Professional societies and regulatory groups then widely adopted them, often with modifications for application to the available primary research base in specific clinical areas. In 2002, a report issued by the AHRQ identified 40 systems to rate quality of a body of evidence [ 182 ]. A critical appraisal of systems used by prominent health care organizations published in 2004 revealed limitations in sensibility, reproducibility, applicability to different questions, and usability to different end users [ 183 ]. Persistent use of hierarchical rating schemes to describe overall quality continues to complicate the interpretation of evidence. This is indicated by recent reports of poor interpretability of systematic review results by readers [ 184 – 186 ] and misleading interpretations of the evidence related to the “spin” systematic review authors may put on their conclusions [ 50 , 187 ].

Recognition of the shortcomings of hierarchical rating systems raised concerns that misleading clinical recommendations could result even if based on a rigorous systematic review. In addition, the number and variability of these systems were considered obstacles to quick and accurate interpretations of the evidence by clinicians, patients, and policymakers [ 183 ]. These issues contributed to the development of the GRADE approach. An international working group, that continues to actively evaluate and refine it, first introduced GRADE in 2004 [ 188 ]. Currently more than 110 organizations from 19 countries around the world have endorsed or are using GRADE [ 189 ].

GRADE approach to rating overall certainty

GRADE offers a consistent and sensible approach for two separate processes: rating the overall certainty of a body of evidence and the strength of recommendations. The former is the expected conclusion of a systematic review, while the latter is pertinent to the development of CPGs. As such, GRADE provides a mechanism to bridge the gap from evidence synthesis to application of the evidence for informed clinical decision-making [ 27 , 190 ]. We briefly examine the GRADE approach but only as it applies to rating overall certainty of evidence in systematic reviews.

In GRADE, use of “certainty” of a body of evidence is preferred over the term “quality.” [ 191 ] Certainty refers to the level of confidence systematic review authors have that, for each outcome, an effect estimate represents the true effect. The GRADE approach to rating confidence in estimates begins with identifying the study type (RCT or NRSI) and then systematically considers criteria to rate the certainty of evidence up or down (Table 5.1 ).

GRADE criteria for rating certainty of evidence

a Applies to randomized studies

b Applies to non-randomized studies

This process results in assignment of one of the four GRADE certainty ratings to each outcome; these are clearly conveyed with the use of basic interpretation symbols (Table 5.2 ) [ 192 ]. Notably, when multiple outcomes are reported in a systematic review, each outcome is assigned a unique certainty rating; thus different levels of certainty may exist in the body of evidence being examined.

GRADE certainty ratings and their interpretation symbols a

a From the GRADE Handbook [ 192 ]

GRADE’s developers acknowledge some subjectivity is involved in this process [ 193 ]. In addition, they emphasize that both the criteria for rating evidence up and down (Table 5.1 ) as well as the four overall certainty ratings (Table 5.2 ) reflect a continuum as opposed to discrete categories [ 194 ]. Consequently, deciding whether a study falls above or below the threshold for rating up or down may not be straightforward, and preliminary overall certainty ratings may be intermediate (eg, between low and moderate). Thus, the proper application of GRADE requires systematic review authors to take an overall view of the body of evidence and explicitly describe the rationale for their final ratings.

Advantages of GRADE

Outcomes important to the individuals who experience the problem of interest maintain a prominent role throughout the GRADE process [ 191 ]. These outcomes must inform the research questions (eg, PICO [population, intervention, comparator, outcome]) that are specified a priori in a systematic review protocol. Evidence for these outcomes is then investigated and each critical or important outcome is ultimately assigned a certainty of evidence as the end point of the review. Notably, limitations of the included studies have an impact at the outcome level. Ultimately, the certainty ratings for each outcome reported in a systematic review are considered by guideline panels. They use a different process to formulate recommendations that involves assessment of the evidence across outcomes [ 201 ]. It is beyond our scope to describe the GRADE process for formulating recommendations; however, it is critical to understand how these two outcome-centric concepts of certainty of evidence in the GRADE framework are related and distinguished. An in-depth illustration using examples from recently published evidence syntheses and CPGs is provided in Additional File 5 A (Table AF5A-1).

The GRADE approach is applicable irrespective of whether the certainty of the primary research evidence is high or very low; in some circumstances, indirect evidence of higher certainty may be considered if direct evidence is unavailable or of low certainty [ 27 ]. In fact, most interventions and outcomes in medicine have low or very low certainty of evidence based on GRADE and there seems to be no major improvement over time [ 202 , 203 ]. This is still a very important (even if sobering) realization for calibrating our understanding of medical evidence. A major appeal of the GRADE approach is that it offers a common framework that enables authors of evidence syntheses to make complex judgments about evidence certainty and to convey these with unambiguous terminology. This prevents some common mistakes made by review authors, including overstating results (or under-reporting harms) [ 187 ] and making recommendations for treatment. This is illustrated in Table AF5A-2 (Additional File 5 A), which compares the concluding statements made about overall certainty in a systematic review with and without application of the GRADE approach.

Theoretically, application of GRADE should improve consistency of judgments about certainty of evidence, both between authors and across systematic reviews. In one empirical evaluation conducted by the GRADE Working Group, interrater reliability of two individual raters assessing certainty of the evidence for a specific outcome increased from ~ 0.3 without using GRADE to ~ 0.7 by using GRADE [ 204 ]. However, others report variable agreement among those experienced in GRADE assessments of evidence certainty [ 190 ]. Like any other tool, GRADE requires training in order to be properly applied. The intricacies of the GRADE approach and the necessary subjectivity involved suggest that improving agreement may require strict rules for its application; alternatively, use of general guidance and consensus among review authors may result in less consistency but provide important information for the end user [ 190 ].

GRADE caveats

Simply invoking “the GRADE approach” does not automatically ensure GRADE methods were employed by authors of a systematic review (or developers of a CPG). Table 5.3 lists the criteria the GRADE working group has established for this purpose. These criteria highlight the specific terminology and methods that apply to rating the certainty of evidence for outcomes reported in a systematic review [ 191 ], which is different from rating overall certainty across outcomes considered in the formulation of recommendations [ 205 ]. Modifications of standard GRADE methods and terminology are discouraged as these may detract from GRADE’s objectives to minimize conceptual confusion and maximize clear communication [ 206 ].

Criteria for using GRADE in a systematic review a

a Adapted from the GRADE working group [ 206 ]; this list does not contain the additional criteria that apply to the development of a clinical practice guideline

Nevertheless, GRADE is prone to misapplications [ 207 , 208 ], which can distort a systematic review’s conclusions about the certainty of evidence. Systematic review authors without proper GRADE training are likely to misinterpret the terms “quality” and “grade” and to misunderstand the constructs assessed by GRADE versus other appraisal tools. For example, review authors may reference the standard GRADE certainty ratings (Table 5.2 ) to describe evidence for their outcome(s) of interest. However, these ratings are invalidated if authors omit or inadequately perform RoB evaluations of each included primary study. Such deficiencies in RoB assessments are unacceptable but not uncommon, as reported in methodological studies of systematic reviews and overviews [ 104 , 186 , 209 , 210 ]. GRADE ratings are also invalidated if review authors do not formally address and report on the other criteria (Table 5.1 ) necessary for a GRADE certainty rating.

Other caveats pertain to application of a GRADE certainty of evidence rating in various types of evidence syntheses. Current adaptations of GRADE are described in Additional File 5 B and included on Table 6.3 , which is introduced in the next section.

Concise Guide to best practices for evidence syntheses, version 1.0 a

AMSTAR A MeaSurement Tool to Assess Systematic Reviews, CASP Critical Appraisal Skills Programme, CERQual Confidence in the Evidence from Reviews of Qualitative research, ConQual Establishing Confidence in the output of Qualitative research synthesis, COSMIN COnsensus-based Standards for the selection of health Measurement Instruments, DTA diagnostic test accuracy, eMERGe meta-ethnography reporting guidance, ENTREQ enhancing transparency in reporting the synthesis of qualitative research, GRADE Grading of Recommendations Assessment, Development and Evaluation, MA meta-analysis, NRSI non-randomized studies of interventions, P protocol, PRIOR Preferred Reporting Items for Overviews of Reviews, PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PROBAST Prediction model Risk Of Bias ASsessment Tool, QUADAS quality assessment of studies of diagnostic accuracy included in systematic reviews, QUIPS Quality In Prognosis Studies, RCT randomized controlled trial, RoB risk of bias, ROBINS-I Risk Of Bias In Non-randomised Studies of Interventions, ROBIS Risk of Bias in Systematic Reviews, ScR scoping review, SWiM systematic review without meta-analysis

a Superscript numbers represent citations provided in the main reference list. Additional File 6 lists links to available online resources for the methods and tools included in the Concise Guide

b The MECIR manual [ 30 ] provides Cochrane’s specific standards for both reporting and conduct of intervention systematic reviews and protocols

c Editorial and peer reviewers can evaluate completeness of reporting in submitted manuscripts using these tools. Authors may be required to submit a self-reported checklist for the applicable tools

d The decision flowchart described by Flemming and colleagues [ 223 ] is recommended for guidance on how to choose the best approach to reporting for qualitative reviews

e SWiM was developed for intervention studies reporting quantitative data. However, if there is not a more directly relevant reporting guideline, SWiM may prompt reviewers to consider the important details to report. (Personal Communication via email, Mhairi Campbell, 14 Dec 2022)

f JBI recommends their own tools for the critical appraisal of various quantitative primary study designs included in systematic reviews of intervention effectiveness, prevalence and incidence, and etiology and risk as well as for the critical appraisal of systematic reviews included in umbrella reviews. However, except for the JBI Checklists for studies reporting prevalence data and qualitative research, the development, validity, and reliability of these tools are not well documented

g Studies that are not RCTs or NRSI require tools developed specifically to evaluate their design features. Examples include single case experimental design [ 155 , 156 ] and case reports and series [ 82 ]

h The evaluation of methodological quality of studies included in a synthesis of qualitative research is debatable [ 224 ]. Authors may select a tool appropriate for the type of qualitative synthesis methodology employed. The CASP Qualitative Checklist [ 218 ] is an example of a published, commonly used tool that focuses on assessment of the methodological strengths and limitations of qualitative studies. The JBI Critical Appraisal Checklist for Qualitative Research [ 219 ] is recommended for reviews using a meta-aggregative approach

i Consider including risk of bias assessment of included studies if this information is relevant to the research question; however, scoping reviews do not include an assessment of the overall certainty of a body of evidence

j Guidance available from the GRADE working group [ 225 , 226 ]; also recommend consultation with the Cochrane diagnostic methods group

k Guidance available from the GRADE working group [ 227 ]; also recommend consultation with Cochrane prognostic methods group

l Used for syntheses in reviews with a meta-aggregative approach [ 224 ]

m Chapter 5 in the JBI Manual offers guidance on how to adapt GRADE to prevalence and incidence reviews [ 69 ]

n Janiaud and colleagues suggest criteria for evaluating evidence certainty for meta-analyses of non-randomized studies evaluating risk factors [ 228 ]

o The COSMIN user manual provides details on how to apply GRADE in systematic reviews of measurement properties [ 229 ]

The expected culmination of a systematic review should be a rating of overall certainty of a body of evidence for each outcome reported. The GRADE approach is recommended for making these judgments for outcomes reported in systematic reviews of interventions and can be adapted for other types of reviews. This represents the initial step in the process of making recommendations based on evidence syntheses. Peer reviewers should ensure authors meet the minimal criteria for supporting the GRADE approach when reviewing any evidence synthesis that reports certainty ratings derived using GRADE. Authors and peer reviewers of evidence syntheses unfamiliar with GRADE are encouraged to seek formal training and take advantage of the resources available on the GRADE website [ 211 , 212 ].

Part 6. Concise Guide to best practices

Accumulating data in recent years suggest that many evidence syntheses (with or without meta-analysis) are not reliable. This relates in part to the fact that their authors, who are often clinicians, can be overwhelmed by the plethora of ways to evaluate evidence. They tend to resort to familiar but often inadequate, inappropriate, or obsolete methods and tools and, as a result, produce unreliable reviews. These manuscripts may not be recognized as such by peer reviewers and journal editors who may disregard current standards. When such a systematic review is published or included in a CPG, clinicians and stakeholders tend to believe that it is trustworthy. A vicious cycle in which inadequate methodology is rewarded and potentially misleading conclusions are accepted is thus supported. There is no quick or easy way to break this cycle; however, increasing awareness of best practices among all these stakeholder groups, who often have minimal (if any) training in methodology, may begin to mitigate it. This is the rationale for inclusion of Parts 2 through 5 in this guidance document. These sections present core concepts and important methodological developments that inform current standards and recommendations. We conclude by taking a direct and practical approach.

Inconsistent and imprecise terminology used in the context of development and evaluation of evidence syntheses is problematic for authors, peer reviewers and editors, and may lead to the application of inappropriate methods and tools. In response, we endorse use of the basic terms (Table 6.1 ) defined in the PRISMA 2020 statement [ 93 ]. In addition, we have identified several problematic expressions and nomenclature. In Table 6.2 , we compile suggestions for preferred terms less likely to be misinterpreted.

Terms relevant to the reporting of health care–related evidence syntheses a

a Reproduced from Page and colleagues [ 93 ]

Terminology suggestions for health care–related evidence syntheses

a For example, meta-aggregation, meta-ethnography, critical interpretative synthesis, realist synthesis

b This term may best apply to the synthesis in a mixed methods systematic review in which data from different types of evidence (eg, qualitative, quantitative, economic) are summarized [ 64 ]

We also propose a Concise Guide (Table 6.3 ) that summarizes the methods and tools recommended for the development and evaluation of nine types of evidence syntheses. Suggestions for specific tools are based on the rigor of their development as well as the availability of detailed guidance from their developers to ensure their proper application. The formatting of the Concise Guide addresses a well-known source of confusion by clearly distinguishing the underlying methodological constructs that these tools were designed to assess. Important clarifications and explanations follow in the guide’s footnotes; associated websites, if available, are listed in Additional File 6 .

To encourage uptake of best practices, journal editors may consider adopting or adapting the Concise Guide in their instructions to authors and peer reviewers of evidence syntheses. Given the evolving nature of evidence synthesis methodology, the suggested methods and tools are likely to require regular updates. Authors of evidence syntheses should monitor the literature to ensure they are employing current methods and tools. Some types of evidence syntheses (eg, rapid, economic, methodological) are not included in the Concise Guide; for these, authors are advised to obtain recommendations for acceptable methods by consulting with their target journal.

We encourage the appropriate and informed use of the methods and tools discussed throughout this commentary and summarized in the Concise Guide (Table 6.3 ). However, we caution against their application in a perfunctory or superficial fashion. This is a common pitfall among authors of evidence syntheses, especially as the standards of such tools become associated with acceptance of a manuscript by a journal. Consequently, published evidence syntheses may show improved adherence to the requirements of these tools without necessarily making genuine improvements in their performance.

In line with our main objective, the suggested tools in the Concise Guide address the reliability of evidence syntheses; however, we recognize that the utility of systematic reviews is an equally important concern. An unbiased and thoroughly reported evidence synthesis may still not be highly informative if the evidence itself that is summarized is sparse, weak and/or biased [ 24 ]. Many intervention systematic reviews, including those developed by Cochrane [ 203 ] and those applying GRADE [ 202 ], ultimately find no evidence, or find the evidence to be inconclusive (eg, “weak,” “mixed,” or of “low certainty”). This often reflects the primary research base; however, it is important to know what is known (or not known) about a topic when considering an intervention for patients and discussing treatment options with them.

Alternatively, the frequency of “empty” and inconclusive reviews published in the medical literature may relate to limitations of conventional methods that focus on hypothesis testing; these have emphasized the importance of statistical significance in primary research and effect sizes from aggregate meta-analyses [ 183 ]. It is becoming increasingly apparent that this approach may not be appropriate for all topics [ 130 ]. Development of the GRADE approach has facilitated a better understanding of significant factors (beyond effect size) that contribute to the overall certainty of evidence. Other notable responses include the development of integrative synthesis methods for the evaluation of complex interventions [ 230 , 231 ], the incorporation of crowdsourcing and machine learning into systematic review workflows (eg the Cochrane Evidence Pipeline) [ 2 ], the shift in paradigm to living systemic review and NMA platforms [ 232 , 233 ] and the proposal of a new evidence ecosystem that fosters bidirectional collaborations and interactions among a global network of evidence synthesis stakeholders [ 234 ]. These evolutions in data sources and methods may ultimately make evidence syntheses more streamlined, less duplicative, and more importantly, they may be more useful for timely policy and clinical decision-making; however, that will only be the case if they are rigorously reported and conducted.

We look forward to others’ ideas and proposals for the advancement of methods for evidence syntheses. For now, we encourage dissemination and uptake of the currently accepted best tools and practices for their development and evaluation; at the same time, we stress that uptake of appraisal tools, checklists, and software programs cannot substitute for proper education in the methodology of evidence syntheses and meta-analysis. Authors, peer reviewers, and editors must strive to make accurate and reliable contributions to the present evidence knowledge base; online alerts, upcoming technology, and accessible education may make this more feasible than ever before. Our intention is to improve the trustworthiness of evidence syntheses across disciplines, topics, and types of evidence syntheses. All of us must continue to study, teach, and act cooperatively for that to happen.

Acknowledgements

Michelle Oakman Hayes for her assistance with the graphics, Mike Clarke for his willingness to answer our seemingly arbitrary questions, and Bernard Dan for his encouragement of this project.

Authors’ contributions

All authors participated in the development of the ideas, writing, and review of this manuscript. The author(s) read and approved the final manuscript.

The work of John Ioannidis has been supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University.

Declarations

The authors declare no competing interests.

This article has been published simultaneously in BMC Systematic Reviews, Acta Anaesthesiologica Scandinavica, BMC Infectious Diseases, British Journal of Pharmacology, JBI Evidence Synthesis, the Journal of Bone and Joint Surgery Reviews , and the Journal of Pediatric Rehabilitation Medicine .

Publisher’ s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Open access
• Published: 02 May 2019

Electric grid reliability research

• Vivian Sultan   ORCID: orcid.org/0000-0002-1066-5212 1 &
• Brian Hilton 1  

Energy Informatics volume  2 , Article number:  3 ( 2019 ) Cite this article

13k Accesses

14 Citations

2 Altmetric

Metrics details

Energy informatics (EI) is the area of research that addresses the application of technology to resolve complex problems in the energy domain. Goebel et al. (BISE 6:25–31, 2014) provided an EI research framework to encompass all aspects of EI research. Due to rapid EI improvements, many current research areas have not been incorporated into this framework. Specifically, this research posits that grid reliability is an underrepresented research area and should be incorporated into the framework. The rapidly changing nature of energy generation, and new developments in the electric-power network, are driving the need for grid reliability research. The goal of this paper is to extend the EI research framework offered by the Goebel et al. through a systematic review of current literature.

This literature review includes current publications (2015–2017) in power utility and technical reference libraries together with the earlier foundational EI papers.

The systematic literature review is based on a broad automated search of appropriate digital sources (IEEE Xplore and the Web of Science) using relevant search terms. The paper also details the components of grid reliability research as well as their accompanying use cases.

The expanded EI research framework presented in this literature review should help researchers identify areas for future research endeavors. In the extended EI research framework, service reliability is now included as a third research component adding to the existing energy efficiency and renewable-energy supply components.

Energy informatics (EI) research concerns the use of information and communication technologies to address energy challenges (Watson and Boudreau 2011 ) and to inform the application of technology to resolve complex problems in the energy domain. The U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy ( 2017 ) classified challenges within the electricity-grid domain into two types: (1) transmission-system challenges and (2) distribution-system challenges. Transmission-system challenges include grid operations, grid reliability, grid stability, models and codes, grid operators, and utilities. Distribution-system challenges includes voltage and volt-ampere reactivity regulation, unintentional islanding, power quality, protection coordination, distribution modeling, visibility and control, codes, and standards.

Goebel et al. ( 2014 ) stated that energy efficiency and renewable-energy supply are the two principal types of research movements within the energy domain. Energy efficiency research involves studying individual incentives and behavioral dynamics to influence electricity consumers’ usage behavior. This first type drives the evolution of smart energy-saving systems. The second type of research, renewable-energy supply, seeks to resolve challenges arising in the integration of such renewable sources of energy as wind and solar power into the electric grid. This, in turn drives, the advancement of smart grids.

This research paper discusses an understudied research area within EI, grid reliability. Given the rapidly changing nature of energy generation, new developments of the electric power network, the incorporation of distributed energy resources into the grid, and circuit and equipment overloads, grid reliability research has been underwhelming. According to Goebel et al. ( 2014 ), energy efficiency and the renewable-energy supply are the two principal types of research movements in the energy domain. In their EI research framework, Goebel et al. considered grid reliability a subtopic, one of the many segments underlying the renewable-energy-supply research theme. Specifically, reliability was a segment under renewable-energy research. However, rapidly shifting challenges within the electric utility industry suggest that grid-reliability research should be classified as a new and separate area of research.

Grid reliability has several definitions. The North American Electric Reliability Corporation defined reliability as “the degree to which the performances of the elements of the electric system result in power being delivered to consumers within accepted standards and in the amount desired” (Hirst and Kirby 2000 , p 7). Osborn and Kawann ( 2001 ) viewed reliability as “the ability of the power system components to deliver electricity to all points of consumption, in the quantity and with the quality demanded by the customer” (p 2). Reliability is measured by outage indices as illustrated by the Institute of Electrical and Electronics Engineers’ Standard 1366. To facilitate a unified view of grid reliability, a definition is proposed: “the ability of the electric grid to deliver electricity to customers without degradation or failure.” The argument is that today’s power systems cannot accommodate, for instance, significant variable distributed energy generation without failure (U.S. Department of Energy 2015 ).

Grid reliability aims to address challenges and remove barriers to integrating high penetration of distributed-energy generation at the transmission and distribution levels (U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy 2017 ). The subject includes many unaddressed questions: What makes the grid reliable? Why might reliability degrade? How do ongoing changes in grid use impose such a risk? What can we do to improve grid reliability? The U.S. Department of Energy has recently released its electric-grid reliability study, recommending prioritization of developments for grid resiliency and reliability (Profeta 2017 ). Additionally, in the same month, the U.S. Department of Energy, Office of Electricity announced an initial investment of nearly $900,000 to address power systems’ risks and uncertainty, enabling academic research in the United States (Laezman 2017 ).

The goal of this paper is to restructure the Goebel et al. ( 2014 ) EI research framework through a systematic literature review to produce an updated framework highlighting grid-reliability research, all of its components, and its additional use cases.

The rest of the paper is organized as follows. The second section presents research importance. The third section describes the methodology used to conduct the systematic literature review. The fourth section outlines the results of the review, proposes an enhanced EI framework, and presents a bibliography of foundational papers. Finally, the fifth section offers conclusions.

Research importance

Due to a rise in electricity use resulting from such new technologies as electric vehicles and circuit and equipment overloads, a significant number of publications from research organizations, governmental bodies, and utility companies have focused on understanding grid reliability, fault causes, and power outages. The National Academies of Sciences, Engineering, and Medicine ( 2017 ) recently published “Enhancing the Resilience of the Nation’s Electricity System” in response to Congress’s call for an independent assessment to “conduct a national-level comprehensive study on the future resilience and reliability of the nation’s electric power transmission and distribution system.” In addition, the National Academies of Sciences, Engineering, and Medicine established a committee to conduct the relevant research. Throughout this report, the committee highlighted all elements of grid reliability, resilience, and the risks of the system-wide failure that will grow as the structure of the power industry becomes more atomized and complex and laid out a wide range of actions to improve the U.S. power system’s resilience. Analytics (including machine learning, data mining, and other artificial-intelligence techniques) will play a very important role in response to the predicted attacks on the electric grid, failures, or other impairments due to their ability to generate real-time recommendations (National Academies of Sciences, Engineering, and Medicine 2017 ).

In another exemplar research, Shawn Adderly ( 2016 ) examined U.S. Department of Energy’s power outage data from 2002 to 2013 and investigated reliability trends. The research objective was to assess the correlation between utilities’ reliability and grid-investment projects such as the deployment of smart-grid assets. Using the deployment of smart meters as a proxy for grid investments, Adderly concluded that the increase of smart meters correlated strongly with a decrease in the frequency of outages. The author acknowledged that due to the presence of confounding variables, the reduction in power outage could not be attributed to any specific smart-grid investment project.

Several studies attempt to understand grid reliability. Steve Mitnick’s ( 2015 ) report prepared for the Electric Markets Research Foundation, another important resource, explains the reasons for concerns about grid reliability. The author suggests that distributed energy resources must be carefully incorporated into the grid to minimize grid-reliability risk. Another relevant study conducted by the Lawrence Berkeley National Lab (Eto et al. 2012 ) highlighted the fact that reliability-data trends might not improve due to the addition of smart-grid technology because automated outage-management systems may report service interruptions more accurately. Since the study was based on a sample of reliability data from several utilities, the authors did not attempt to make claims about overall U.S. power reliability.

With respect to the power-outage-causes study domain, the majority of the outages in the United States are the result of events that occur on the grid’s distribution side. Few outages are caused by the external factors. The three main causes for electrical outages are (1) hardware and technical failures, (2) environmental incidents, and (3) human errors. Among hardware and technical failures, outages are experienced due to equipment overload and short circuits, to name a few (Westar Energy 2017 ; Diesel Service and Supply 2017 ; Rocky Mountain Power 2017 ). These failures are often attributed to unmet peak demand, outdated equipment, and malfunctions of backup power systems (Chayanam 2005 ).

Environmental incidents, the largest portion of power outage causes, can be classified into two distinct categories: weather and other factors. Such publications as Wisconsin Public Service ( 2017 ) have highlighted the weather-related causes of power outages. The Edison Electric Institute states that 70% of U.S. power outages are weather related (as cited in Campbell 2012 ). Kenward and Raja ( 2014 ) analyzed power outage data over a 28-year period and noted that between 2003 and 2012, 80% of all outages were caused by weather. Similarly, Campbell ( 2012 ) highlighted the damage to the electrical grid caused by seasonal storms, rain, and high winds.

Besides weather, other external forces create power outages. As a byproduct of heavy weather patterns, falling tree branches disrupt the flow of electricity (National Academies of Sciences, Engineering, and Medicine 2017 ). Animals are another culprit of power disruption. The Edison Electric Institute study also indicated that animals, such as large birds, contacting power lines accounted for 11% of U.S. outages (as cited in Campbell 2012 ).

Human-error incidents are the last cause of power outages. Chayanam ( 2005 ) indicated that training is essential for technicians and staff to battle outages with proper maintenance procedures. This ensures a reduction in the frequency of power outages. Car accidents are another major source of power outages (Wisconsin Public Service 2017 ).

In a 2013 white paper entitled “the Smart Grid Investment Grant Program Progress Report,” the Department of Energy highlighted reliability improvements observed through decreasing reliability indices and highlighted that such projects as automated feeder switching were able to reduce the frequency of outages. No statistics were shown in the report to demonstrate the correlation between reliability indices and spending. However, the study identified the progress made by utilities as a result of receiving federal funding (U.S. Department of Energy 2013 ).

Interrupted power supply is no longer a mere inconvenience. As the duration and spatial extent of electrical outages increases, costs and inconvenience grow. Critical social services such as medical care, police and other emergency services, and communications systems rely upon electricity to function at minimally acceptable levels. Failures can bring catastrophic outcomes; lives can be lost. We must better understand the causes to be more ready to implement redundancy and resilience in the electric grid.

To heed this call, this paper presents a systematic grid-reliability literature review to help understand the topic’s current knowledge base. A systematic literature review is a particularly influential tool in the hands of researchers because it allows a scholar to gather and recap all the information about research in a specific field (Spanos and Angelis 2016 ). In this first systematic grid-reliability literature review, the focus is on the different grid-reliability topics and their specific characteristics. This article should enrich future literature reviews while integrating the most current articles into the body of knowledge.

Methodology

The systematic literature review offered in this paper follows the three stages in a systematic review: the Planning Stage, the Conducting Stage, and the Reporting Stage (Kitchenham 2004 ; Kitchenham and Charters 2007 ).

The first step is the identification of a need for a systematic review. As described in the previous two sections, although several studies have investigated electric-grid reliability, these studies should be summarized to update the current knowledgebase. Therefore, the urgent need for a systematic literature review providing solid foundations and equipping researchers with pertinent information is clear.

The second step is the development of the review protocol. This section presents the research questions, search strategy, inclusion/exclusion criteria, quality-assessment criteria, and the data extracted from the studies.

Defining research questions is an essential step in every systematic review. By answering the following questions, the literature review can accomplish its aim.

How many research studies have examined electric-grid reliability?

What are the types of reliability-research questions?

What are the studies’ results?

What research methods are used?

To conduct the systematic literature review, it was decided to do a broad automated search, a method that includes the selection of the most appropriate digital sources (digital libraries and indexing systems) and the determination of the search terms (Spanos and Angelis 2016 ). The digital libraries of IEEE Xplore and the Web of Science were selected for the systematic review. The Web of Science database provides a wide breadth and depth, whereas the IEEE database provides more narrowly focused and very recent research. These searches relied on papers’ titles to avoid receiving duplicate or irrelevant papers as search results. The following search strings were used.

IEEE Xplore Boolean/Phrase

(((((“Document Title”:Electricity Reliability) OR “Document Title”:Electric Grid Reliability) OR “Document Title”:Power System Reliability) OR “Document Title”:Electric Circuit Reliability) OR “Document Title”:Power Outage Research)

refined by.

Content Type: Conference Publications Journals & Magazines Books & eBooks Year: 2015–2017

The year range is limited to those 3 years to ensure the search captures the reliability impact of integrating distribution-energy resources into the electric grid.

Web of Science Boolean/Phrase

TITLE:(Electricity Reliability) OR TITLE:(Electric Grid Reliability) OR TITLE:(Power System Reliability) OR TITLE:(Electric Circuit Reliability) OR TITLE:(Power Outage Research) AND TITLE: (“estimate*” OR “assess*”)

DOCUMENT TYPES: (PROCEEDINGS PAPER OR ARTICLE)

Timespan: 2015–2017. Indexes: SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, ESCI.

The inclusion/exclusion selection criteria of any systematic literature review must be distinct and clearly stated (Spanos and Angelis 2016 ). The following selection criteria were used for the systematic review.

Full-article publication (not just an abstract),

English-language publication,

Year of publication sufficiently recent (since year 2015) to ensure that the reliability impact of integrating distribution energy resources into the electric grid has been captured, and

Study relevance.

The following exclusion criterion was used.

Duplicate publications (to avoid double counting studies)

The quality-assessment criteria are provided to ensure that all the included studies in a systematic literature review contain an adequate level of quality. The following were the criteria.

Availability and the description of the data,

Description of the used methodology, and

Presentation of the results.

The last task is related to the development of the review protocol and the selection of the data features that will be extracted from the papers found by the search. The data features for this review are as follows.

Year of publication,

Type of paper,

The research question(s),

Method(s) used, and

This section presents the initial results of the systematic literature review: 209 papers from the IEEE Xplore database and 294 papers from the Web of Science core collection. Table  1 summarizes the characteristics of the documents identified and the counts of papers by year.

The top publishing authors were M Fotuhi-Firuzabad (10 publications), C Singh (8), LF Wang (8), YM Xing (8), and M Benidris (6).

After the excluding duplicate, irrelevant, and low-quality papers, 231 papers remained from the search in the two databases. Table  2 summarizes the search results (503 total paper count from the initial search), the number of non-relevant, the duplicate papers, those not-included after the quality assessment, and the final number of relevant papers included in the systematic literature review for analysis.

Preliminary analysis of the relevant papers identified four key types of research question:

How does one assess or evaluate reliability of the power grid?

How does one improve or enhance the reliability of the electric-power system?

How should one plan reliability of the smart grid?

What are the impacts of changes, including adding distributed-energy resources, new regulations, and investment projects, on the reliability of the electric-power system?

Research methods within this research domain can be classified either as analytical or as Monte Carlo simulation. Analytical techniques represent the system by a mathematical model and evaluate the reliability indices from this model using mathematical solutions. Monte Carlo simulation methods, on the other hand, estimate the reliability indices by simulating the actual process and the system’s random behavior. Simulation treats problems as a series of real experiments. There are advantages and disadvantages to both methods.

The foremost research methods are modeling and simulation. Simulation modeling is the process of creating and analyzing a prototype of a physical model to predict its performance in the real world. Simulation modeling is used to help researchers understand whether, under what conditions, and in which ways a part could fail and what loads it can withstand. Researchers have used various modeling and simulation tools to perform the analysis, but Monte Carlo simulation is the most dominant research method.

Based on the systematic literature review, the Goebel et al. ( 2014 ) EI research framework was extended. The following section illustrates the impact of this extension and highlights the addition of grid reliability research, all of its components, and its use cases.

Energy informatics enhanced research framework

In the proposed Energy Informatics enhanced research framework (Fig.  1 ), energy efficiency, renewable-energy supply, and service reliability are the three types of research streams in the energy domain. These streams reflect the topics identified in the systematic literature review. The restructured Goebel et al. ( 2014 ) framework includes service reliability as a third research stream in addition to energy efficiency and the renewable-energy supply to distinguish this understudied research area. The first theme, energy efficiency, drives the evolution of smart energy-saving systems. The second theme, renewable-energy supply, drives the advancement of smart grids. Finally, the third additional theme, service reliability, drives smart-grid reliability and resiliency.

Energy informatics enhanced research framework—Enriched with reliability research

In the context of the service-reliability research theme, use cases (a collection of possible scenarios) were classified into four transmission scales: subcontinental, regional, local distribution system, and feeder. Power systems and renewable energy can also be viewed as additional use cases due to their impact on grid reliability. The rapidly changing nature of energy generation and the new developments of the electric power network have fueled the rise of grid-reliability research to justify considering it a separate research stream.

The first category of use cases, the subcontinental scale, examines large, relatively self-contained landmasses forming a subdivision of a continent. Within this category, multiple grids, transmission, and distribution systems may exist and be interconnected.

In the second category of use cases, regional transmission, studies examine high-voltage transmission networks that enables power to travel long distances from generating units to substations closer to local end-use customers where the voltage is stepped back down and sent into the distribution system for delivery to consumers. Given the interconnected configuration of the high-voltage grid, events in one place can propagate across the transmission system in seconds or a few minutes, potentially causing cascading blackouts that can affect customers hundreds of miles from the initial disturbance. Outage events on the transmission system can result in large-area impacts (National Academies of Sciences, Engineering, and Medicine ( 2017 ).

In the third type of use cases, the electric distribution system moves power from the energy system to the meters of electricity customers. Typically, power is delivered to distribution substations from two or more transmission lines, where its voltage is reduced, and sent to customers over distribution feeders. Although distribution-system outages are more common than those occurring on transmission facilities, their impacts are smaller in scale and generally easier to repair (National Academies of Sciences, Engineering, and Medicine 2017 ).

The fourth category of use cases includes the feeder scale. Customers on radial systems are exposed to interruption when their feeder experiences an outage. In metropolitan areas, these feeders typically have switches that can be reconfigured to support restoration from an outage or regular maintenance. When a component fails in these systems, customers on unaffected sections of the feeder are switched manually or automatically to an adjacent, functioning, circuit. However, this still exposes critical services such as hospitals or police stations to potential outages, so these facilities are often connected to a second feeder for redundancy (National Academies of Sciences, Engineering, and Medicine 2017 ).

The definition of service reliability—“the ability of the electric grid to deliver electricity to customers without degradation or failure”—is used to outline the service-reliability research theme. Recent developments such as the integration of distributed-energy resources into the smart grid make information collection, integration, management, and analysis of vital importance. That is why EI has flourished in the research community.

Here, the aim is to contribute to a holistic understanding of problem identification and resolution through the use of tools such as geographic information systems, databases, big-data management, machine learning, information security, and optimization and control. Analytics using these tools could transform the way we think, act, and use energy and help elucidate a problem’s root cause, define a solution through data, and implement the solution with continuous monitoring and management.

Power-system reliability research framework

In addition to the EI enhanced research framework, Fig.  2 illustrates a framework for power-system reliability founded on the previous literature. The focus within the power-system-reliability research theme can be organized into one of two main types: the bulk-power system (BPS) and the local-distribution system. The first research focus, can be defined as a large interconnected electrical system made up of generation and transmission facilities and their control systems. A BPS does not include facilities used in the local distribution of electric energy. If a BPS is disrupted, the effects are felt in more than one location. In the United States, BPS are managed by the North American Electric Reliability Corporation (National Academies of Sciences, Engineering, and Medicine 2017 ). Reliability of power supply and system operation, regular evaluations of expected or emerging changes, and system maintenance throughout changes in the electric industry are all possible goals within the BPS research focus.

The local-distribution system, the second research focus, provides power to individual consumer premises. Distribution networks usually consist of distribution substations, primary distribution feeders, distribution transformers, distributors, and service mains (National Academies of Sciences, Engineering, and Medicine 2017 ). Maintenance and repair of the distribution network, public safety, and operating cost are the possible goals within the local-distribution system research focus.

The combined transmission and distribution network is known as the “power grid” or simply “the grid.” North America’s BPS involves four different power grids (interconnections). The Eastern Interconnection serves the eastern two thirds of the continental United States and Canada from the Great Plains to the Eastern Seaboard. The Western Interconnection covers the western third of the continental United States, the Canadian provinces of Alberta and British Columbia, and a portion of Baja California Norte in Mexico. The Texas Interconnection includes most of the State of Texas. Finally, the Canadian province of Quebec is served by the fourth North American interconnection. The grid systems in Hawaii and Alaska are not connected to the grids in the continental states (U.S. Department of Energy, Office of Electricity 2015 ).

In the context of the power-system reliability research theme, possible scenarios, or use cases, can be classified into: infrastructure addition, infrastructure retirement, changing demand conditions, evolving technologies, changes driven by cost, policies, technological change, events, substation reliability, electric-circuit reliability, electric short circuits, cybersecurity risks, natural disasters and climate change, demand response and flexibility technologies, information and communication technologies, operating protocol, and other risks and disturbances.

Foundational papers on the subject of electric-grid reliability

One way to grasp the main core of a subject is to look at the references cited in the current articles and highlight papers that are continually referenced. This step was particularly helpful in identifying the foundational papers.

The same search string was used without an exclusion criterion (no year range restriction) to pull all journal and magazine publications from the database. The results (366 extracted articles) were sorted in descending order based on the number of citations (the number of other papers’ reference lists that included them), the standard deviation for each article’s number of citations was calculated, and the outliers (articles whose number of citations was more one standard deviations) were identified. Based on this analysis, 59 foundational papers were identified. After excluding the irrelevant papers and those not meeting the quality criteria, 39 relevant papers remained (Table  3 ).

This section presents a bibliography and the analysis of foundational papers on the subject of electric-grid reliability (Table  4 ).

Table  5 summarizes the research methods identified in the foundational papers. Though modeling and simulation are dominant research methods within the literature, articles using analytical approaches seem to be gaining more attention considering how often they are cited.

Bearing in mind the research themes and the methods illustrated in the foundational papers, analytics has been a popular topic in research and more research is needed in the area of reliability planning and improvement, particularly in the energy field. Through the use of GIS, machine learning, and data-mining techniques, analytics would help the research community plan and improve the smart grid.

This paper identifies the use of analytics to predict reliability issues and plan for grid reliability. To engender cutting-edge grid-reliability research activity, prospective authors in this space could examine topics such as:

Modeling grid behavior,

Detecting outages,

Location analytics on smart-grid resiliency,

Planning secondary and tertiary backup for the smart grid,

Data management and pipeline to enable analytics,

Internet of things (IOT) and sensors for analytics, and

Recent advances in analytics and smart grids.

This systematic literature review of grid-reliability research provides a solid foundation to equip researchers with the most pertinent information, offers an enhanced EI research framework, and provides directions for future research in this domain.

Abbreviations

Bulk-power system

Energy Informatics

Adderly S (2016) Reviewing power outage trends, electric reliability indices and smart grid funding. Thesis, University of Vermont. http://scholarworks.uvm.edu/cgi/viewcontent.cgi?article=1530&context=graddis . Accessed 15 Sept 2017

Google Scholar  

Campbell RJ (2012) Weather-related power outages and electric system resiliency. (Congressional Research Service Report, Vol. August 28, pp. 1–15). https://fas.org/sgp/crs/misc/R42696.pdf . Accessed 15 Sept 2017

Chayanam K (2005) Analysis of telecommunications power outages. Thesis. https://etd.ohiolink.edu/rws_etd/document/get/ohiou1125024491/inline . Accessed 15 Sept 2017

Diesel Service and Supply (2017). The many causes of power failures. http://www.dieselserviceandsupply.com/Causes_of_Power_Failures.aspx . Accessed 11 Sept 2017

Eto JH, Lacommare, KH, Larsen P, Todd A, Fisher E (2012) An examination of temporal trends in electricity reliability based on reports from U.S. electric utilities. https://emp.lbl.gov/publications/examination-temporal-trends . Accessed 15 Sept 2017

Book   Google Scholar  

Goebel C, Jacobsen H-A, del Razo V, Doblander C, Rivera J, Ilg J et al (2014) Energy informatics—current and future research directions. BISE 6:25–31. https://doi.org/10.1007/s12599-013-0304-2

Article   Google Scholar  

Hirst E, Kirby B (2000) Bulk-power basics: reliability and commerce. Consulting in Electric-Industry Restructuring, Oak Ridge http://www.consultkirby.com/files/RAPBPBasics.pdf

Kenward A, Raja U (2014) Blackout: extreme weather, climate change and power outages. https://www.eenews.net/assets/2014/04/14/document_ew_01.pdf . Accessed 15 Sept 2017

Kitchenham B (2004) Procedures for performing systematic reviews (NICTA Technical Report 0400011T.1). Durham University and Keele University, Durham http://www.inf.ufsc.br/~aldo.vw/kitchenham.pdf . Accessed 11 Sept 2017

Kitchenham B, Charters S (2007). Guidelines for performing systematic literature reviews in software engineering (Keele University and Durham University Joint Report). https://userpages.uni-koblenz.de/~laemmel/esecourse/slides/slr.pdf

Laezman R (2017) Office of Energy invests in grid reliability research. Electrical Contractor Magazine. http://www.ecmag.com/section/systems/office-energy-invests-grid-reliability-research . Accessed 11 Sept 2017

Mitnick S (2015) Changing uses of the electric grid: reliability challenges and concerns. http://www.emrf.net/uploads/3/4/4/6/34469793/emrf_business_models_final_web_version.pdf . Accessed 11 Sept 2017

National Academies of Sciences, Engineering, and Medicine (2017) Enhancing the resilience of the nation’s electricity system. The National Academies Press, Washington, DC. https://doi.org/10.17226/24836

Osborn J, Kawann C (2001) Reliability of the U.S. electricity system: recent trends and current issues. Energy Analysis Department, Ernest Orlando Lawrence Berkeley National Laboratory, Berkeley https://emp.lbl.gov/sites/all/files/REPORTlbnl-47043.pdf . Accessed 11 Sept 2017

Profeta T (2017) The climate post: grid reliability study released as climate change panel disbands. http://www.huffingtonpost.com/entry/the-climate-post-grid-reliability-study-released-as_us_599ef43ae4b0a62d0987ad61 . Accessed 11 Sept 2017

Rocky Mountain Power (2017) Key causes of power outages. https://www.rockymountainpower.net/ed/po/or/kcopo.html . Accessed 11 Sept 2017

Spanos G, Angelis L (2016) The impact of information security events to the stock market: a systematic literature review. Comput Secur 58:216–229. https://doi.org/10.1016/j.cose.2015.12.006

U.S. Department of Energy (2013) Smart Grid Investment Grant Program progress report II. https://www.smartgrid.gov/files/SGIG_progress_report_2013.pdf . Accessed 15 Sept 2017

U.S. Department of Energy (2015) Enabling modernization of the electric power system. In: Quadrennial technology review: An assessment of energy technologies and research opportunities http://energy.gov/downloads/chapter-3-enabling-modernization-electric-power-system . Accessed 11Sept 2017

U.S. Department of Energy, Office of Electricity (2015). United States electricity industry primer (DOE/OE-0017). https://www.energy.gov/sites/prod/files/2015/12/f28/united-states-electricity-industry-primer.pdf . Accessed 11 Sept 2017

U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy (2017) Performance and operation. https://energy.gov/eere/sunshot/grid-performance-and-reliability . Accessed 11 Sept 2017

Watson RT, Boudreau M-C (2011) Energy informatics. Green ePress, Athens

Westar Energy (2017). What causes power outages? https://www.westarenergy.com/outage-causes . Accessed 11 Sept 2017

Wisconsin Public Service (2017) Why power outages occur. http://www.wisconsinpublicservice.com/home/power_occur.aspx . Accessed 11 Sept 2017

Download references

Acknowledgements

The first author thanks her entire dissertation committee for their helpful advice.

This literature review is based on the first author’s doctoral dissertation earned from Claremont Graduate University. No additional funding was used for this project.

Availability of data and materials

Additional supporting data is available from the first author.

Author information

Authors and affiliations.

Claremont Graduate University, Claremont, CA, 91711, USA

Vivian Sultan & Brian Hilton

You can also search for this author in PubMed   Google Scholar

Contributions

The first author completed the literature review. The second author was the chair of the first author’s dissertation committee, designed the paper structure, reviewed the findings, and proofread/edited the final paper. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Vivian Sultan .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Cite this article.

Sultan, V., Hilton, B. Electric grid reliability research. Energy Inform 2 , 3 (2019). https://doi.org/10.1186/s42162-019-0069-z

Download citation

Received : 18 February 2019

Accepted : 12 April 2019

Published : 02 May 2019

DOI : https://doi.org/10.1186/s42162-019-0069-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Energy informatics
• Grid reliability
• EI research framework

Smart Grid Security: A Systematic Literature Review

• Conference paper
• First Online: 27 May 2023
• Cite this conference paper

• Vivian Sultan 11 ,
• Arun Aryal 11 ,
• Crystal Wu 11 &
• Hector Lopez 11  

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 700))

Included in the following conference series:

• International Conference on Advances in Computing Research

504 Accesses

The smart grid is an upgrade to the current electrical power grid. This upgrade is in response to changing consumer requirements for the 21st century. The smart grid is a large and complex system spread out over a large geographic area. The grid as we know it today already relies on a wide variety of digital devices and computerized controls to function. Managing security threats to power utility systems is a complex issue. Several security risks are present in the smart grid, so this project systematically reviewed smart-grid security to define the unique research streams within the domain, enhance the field’s knowledge, and to guide future research and enrich the body of knowledge. A systematic literature review is a particularly influential tool in the hands of researchers since it allows a scholar to gather and recap all the information about research in a specific field.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Infrastructure Security for Smart Electric Grids: A Survey

Smart grid security – an overview of standards and guidelines

Reliability-Centric Studies in Smart Grids: Adequacy and Vulnerability Considerations

Amin, S., Ali, S., Hanzo, L.: A Survey of security and privacy challenges on the internet of things. IEEE Commun. Surv. Tutorials 21 (1), 49–83 (2019)

Google Scholar  

Bashar, A., Muhammad, S., Mohammad, N., Khan, M.: Modeling and analysis of MDP-Based security risk assessment system for smart grids. In: Fourth International Conference on Inventive Systems and Control, pp. 25–30 (2020)

Boell, S.K., Cecez-Kecmanovic, D.: On being ‘systematic’ in literature reviews in IS. J. Inf. Technol. 30 (2), 161–173 (2015)

Article   Google Scholar  

Chen, Y.T.: Modeling information security threats for smart grid applications by using software engineering and risk management. In: IEEE International Conference on Smart Energy Grid Engineering, pp. 128–132 (2018)

Dumitrache, I., Dogaru, D.I.: Smart grid overview: infrastructure, cyber-physical security and challenges. In: 20th International Conference on Control Systems and Computer Science, pp. 693–699 (2015)

Ekanayake, B., Kintada, S., Jayasinghe, S.: Smart grid: an overview of the technology and its integration with renewable energy systems. Renew. Sustain. Energy Rev. 82 , 685–697 (2018)

Elzinga, D.: Electricity system development: a focus on smart grids. Overview of activities and players in smart grids UNECE (2015). https://www.unece.org/fileadmin/DAM/energy/se/pdfs/eneff/eneff_h.news/Smart.Grids.Overview.pdf . Accessed 12 June 2016

El Shafie, A., Niyato, D., Hamila, R., Al-Dhahir, N.: Impact of the wireless network’s PHY security and reliability on demand-side management cost in the smart grid. IEEE Access 5 , 5678–5689 (2017)

Ericsson, G.N.: Cyber security and power system communication—essential parts of a smart grid infrastructure. IEEE Trans. Power Deliv. 25 (3), 1501–1507 (2010). https://doi.org/10.1109/TPWRD.2010.2046654

Faulin, J., Juan, A.A., Martorell, S., Ramírez-Márquez, J.-E.: Simulation Methods for Reliability and Availability of Complex Systems. Springer, London (2010). https://doi.org/10.1007/978-1-84882-213-9

Book   MATH   Google Scholar  

Hahn, A., Ashok, A., Sridhar, S., Govindarasu, M.: Cyber-physical security testbeds: architecture, application, and evaluation for smart grid. IEEE Trans. Smart Grid 4 (2), 847–855 (2013). https://doi.org/10.1109/TSG.2012.2226919

Hasan, M.K., et al.: Blockchain technology on smart grid, energy trading, and big data: security issues, challenges, and recommendations. Wirel. Commun. Mob. Comput. 2022 (2022). Art. 9065768. https://doi.org/10.1155/2022/9065768

Khan, M.K., Imran, M., Ali, S., Hanzo, L.: Smart grid communications: state-of-the-art, challenges and future research directions. IEEE Commun. Surv. Tutorials 21 (1), 49–83 (2019)

Kimani, K., Oduol, V., Langat, K.: Cyber security challenges for IoT-based smart grid networks. Int. J. Crit. Infrastruct. Prot. 25 , 36–49 (2019)

Kitchenham, B.: Procedures for performing systematic reviews. Keele University Technical report TR/SE-0401. Keele, UK. https://www.researchgate.net/profile/Barbara-Kitchenham/publication/228756057_Procedures_for_Performing_Systematic_Reviews . Accessed Sept 2022

Kitchenham, B., Brereton, O.P., Budgen, D., Turner, M., Bailey, J., Linkman, S.: Systematic literature reviews in software engineering—a systematic literature review. Inf. Softw. Technol. 51 (1), 7–15 (2009)

Knopf, J.W.: Doing a literature review. PS: Polit. Sci. Polit. 39 (1), 127–132 (2006)

Komninos, N., Philippou, E., Pitsillides, A.: Survey in smart grid and smart home security: issues, challenges and countermeasures. IEEE Commun. Surv. Tutorials 16 (4), 1933–1954 (2014). https://doi.org/10.1109/COMST.2014.2320093

Kumar, V., Gupta, C.P.: Cyber security issue in smart grid. In: IEEE 4th International Conference on Computing, Power and Communication Technologies, pp. 1–9 (2021)

Li, Z., Liang, Q., Zhang, B., Wu, X.: Home area network security in smart grid: a hybrid model of wireless and power line communications. In: Mu, J., Liang, Q., Wang, W., Zhang, B., Pi, Y. (eds.) The Proceedings of the Third International Conference on Communications, Signal Processing, and Systems. LNEE, vol. 322, pp. 79–87. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-08991-1_9

Chapter   Google Scholar  

Liu, J., Xiao, Y., Li, S., Liang, W., Chen, C.L.P.: Cyber security and privacy issues in smart grids. IEEE Commun. Surv. Tutorials 14 (4), 981–997 (2012). https://doi.org/10.1109/SURV.2011.122111.00145

Llaria, A., Dos Santos, J., Terrasson, G., Boussaada, Z., Merlo, C., Curea, O.: Intelligent buildings in smart grids: a survey on security and privacy issues related to energy management. Energies 14 (2021). Art. 2733. https://doi.org/10.3390/en14092733

Malina, L., Hajny, J., Zeman, V., Vrba, K.: Security and privacy in the smart grid services. In: 38th International Conference on Telecommunications and Signal Processing, pp. 71–75. Prague, Czech Republic (2015). https://doi.org/10.1109/TSP.2015.7296226

Malik, N.S.: US power grids attacks reach all-time high in 2022. Bloomberg (2023). https://www.bloomberg.com/news/articles/2023-02-01/attacks-on-us-power-grids-rise-to-all-time-high-in-2022 . Accessed 11 Feb 2023

Metke, A.R., Ekl, R.L.: Security technology for smart grid networks. IEEE Trans. Smart Grid 1 (1), 99–107 (2010). https://doi.org/10.1109/TSG.2010.2046347

Mo, Y., et al.: Cyber-physical security of a smart grid infrastructure. Proc. IEEE 100 (1), 195–209 (2012). https://doi.org/10.1109/JPROC.2011.2161428

Molokomme, D.N., Onumanyi, A.J., Abu-Mahfouz, A.M.: Edge intelligence in smart grids: a survey on architectures, offloading models, cyber security measures, and challenges. J. Sens. Actuator Netw. 11 (3) (2022). Art. 47. https://doi.org/10.3390/jsan11030047

Raychaudhuri, S.: Introduction to Monte Carlo simulation. In: IEEE Winter Simulation Conference, pp. 91–100 (2008)

Reed, J.H., Gonzalez, C.R.A.: Enhancing smart grid cyber security using power fingerprinting: integrity assessment and intrusion detection. In: IEEE Future of Instrumentation International Workshop Proceedings, pp. 1–3 (2012)

Spanos, G., Angelis, L.: The impact of information security events to the stock market: a systematic literature review. Comput. Secur. 58 , 216–229 (2016)

Sultan, V., Aryal, A., Chang, H., Kral, J.: Integration of EVs into the smart grid: a systematic literature review. Energy Inf. J. Comput. Sci. 5 (65) (2022). https://doi.org/10.1186/s42162-022-00251-2 . Accessed 18 Feb 2023

U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy. OE 417 Annual Summary Report (2022). https://www.oe.netl.doe.gov/OE417_annual_summary.aspx . Accessed 11 Feb 2023

Yan, X., Du, J.G., Li, J.R.: The digital watermarking used to the smart power grid security. Adv. Mater. Res. 181 , 770–775 (2011)

Yan, Y., Qian, Y., Sharif, H., Tipper, D.: A survey on cyber security for smart grid communications. IEEE Commun. Surv. Tutorials 14 (4), 998–1010 (2012). https://doi.org/10.1109/SURV.2012.010912.00035

Download references

Author information

Authors and affiliations.

College of Business and Economics, California State University, Los Angeles, USA

Vivian Sultan, Arun Aryal, Crystal Wu & Hector Lopez

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Vivian Sultan .

Editor information

Editors and affiliations.

University of Detroit Mercy, Detroit, MI, USA

Kevin Daimi

Charles Sturt University, Sydney, NSW, Australia

Abeer Al Sadoon

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Sultan, V., Aryal, A., Wu, C., Lopez, H. (2023). Smart Grid Security: A Systematic Literature Review. In: Daimi, K., Al Sadoon, A. (eds) Proceedings of the 2023 International Conference on Advances in Computing Research (ACR’23). ACR 2023. Lecture Notes in Networks and Systems, vol 700. Springer, Cham. https://doi.org/10.1007/978-3-031-33743-7_28

Download citation

DOI : https://doi.org/10.1007/978-3-031-33743-7_28

Published : 27 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-33742-0

Online ISBN : 978-3-031-33743-7

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Grab your spot at the free arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Neural and Evolutionary Computing

Title: optimization algorithms in smart grids: a systematic literature review.

Abstract: Electrical smart grids are units that supply electricity from power plants to the users to yield reduced costs, power failures/loss, and maximized energy management. Smart grids (SGs) are well-known devices due to their exceptional benefits such as bi-directional communication, stability, detection of power failures, and inter-connectivity with appliances for monitoring purposes. SGs are the outcome of different modern applications that are used for managing data and security, i.e., modeling, monitoring, optimization, and/or Artificial Intelligence. Hence, the importance of SGs as a research field is increasing with every passing year. This paper focuses on novel features and applications of smart grids in domestic and industrial sectors. Specifically, we focused on Genetic algorithm, Particle Swarm Optimization, and Grey Wolf Optimization to study the efforts made up till date for maximized energy management and cost minimization in SGs. Therefore, we collected 145 research works (2011 to 2022) in this systematic literature review. This research work aims to figure out different features and applications of SGs proposed in the last decade and investigate the trends in popularity of SGs for different regions of world. Our finding is that the most popular optimization algorithm being used by researchers to bring forward new solutions for energy management and cost effectiveness in SGs is Particle Swarm Optimization. We also provide a brief overview of objective functions and parameters used in the solutions for energy and cost effectiveness as well as discuss different open research challenges for future research works.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Solar photovoltaics value chain and end-of-life management practices: a systematic literature review.

1. Introduction

• waste management practices [ 4 , 5 ];
• human health [ 6 , 7 ];
• the natural environment [ 4 , 8 , 9 ];
• water sources specifically [ 7 ];
• economic performance [ 2 , 10 ].

2. Generations of Photovoltaic Technology

3. methodology of the study, 4. bibliometric distribution of the selected paper, 5. thematic results and discussion, 5.1. raw materials, 5.2. manufacturing technology, 5.3. supply chain collaboration, 5.4. pv installation, 5.5. job market, 5.6. business models and pv industry.

• Product–service system: Under this model, customers have the advantage of a product–service system (PSS) because they can use products or services without purchasing them directly. The PSS bypasses the upfront costs of installation and maintenance costs, encouraging customer adoption and promoting PV circularity [ 64 ] through the reuse, repair and refurbishment of products. Schmidt-Costa et al. [ 64 ] identified use-oriented and result-oriented PSS alternatives that may overcome obstacles such as initial costs, a lack of financing options and limited access to appropriate installation sites, even in a region with highly favourable conditions for solar PVs, such as California. A use-oriented PSS includes maintenance and repair services in the rental or lease contracts [ 65 ]. In a result-oriented PSS, the consumer pays a predetermined fee for the energy generated, usually measured in kWh. Payment is determined by the results or product performance, such as the amount of energy produced by a PV array. Reim et al. [ 76 ] identified three structures for PSS models. A product-oriented PSS holds the provider responsible for the products and services, which the customer pays for. In a use-oriented PSS, the provider is responsible for the usability, which the customer pays for over time. A result-oriented PSS has the provider responsible for results and the customer pays for units of outcome. The PSS model has the advantages of increased solar technology affordability and availability, customers have less financial risk and resource efficiency is increased [ 64 ]. On the other hand, the PSS model must address key issues such as active stakeholder engagement and communication, clearly defined value propositions and pricing structures, which can be complex.
• Second-life model. Under this model, PV products are repurposed at the end of their original life to reduce waste, save resources and generate economic benefits [ 66 ]. In a second-life cycle, components can be integrated for new use into building materials and furniture, or they can be repaired for reuse in solar panels. Bocken et al. [ 25 ] propose that solar panel collection systems be combined with product design strategies to facilitate the reuse, refurbishment and recycling of materials and components to create new products. However, effective second-life models have supportive legislation that puts the responsibility onto producers to manage waste and feed it back into second-life processes [ 20 ]. Such product stewardship programs also emphasise consumer education about the challenges of waste associated with solar PV panels and encourage demand for second-life products and services [ 20 ].
• EOL management. A recycling system to recover materials and components from EOL products is another CBM that can promote circularity in the solar industry. This CBM promotes the creation of new products, a reduction in raw material consumption, and increased resource efficiency. Recycling systems can be designed to recover silver, copper and aluminium from used solar panels [ 67 ]. Reduced landfill is another advantage of recycling PV waste.

5.8. Government and Other Institutions

• Tax credits, subsidies or feed-in tariffs that encourage the adoption of circular economic practices and ensure the useful recycling of economic resources;
• Extended producer responsibility (EPR) policies that can shift the financial burden of EOL management to manufacturers, encouraging them to use and generate recyclable materials by applying circular economic principles;
• Landfill bans or charges for solar PV waste disposal.

5.9. Health and Environment

5.10. end-of life (eof) closed loop (reuse, recycle), 5.11. ev batteries/lithium-ion batteries (lib), 5.12. pv system benefits, 6. sustainability and pv management.

• RQ1: In the complex value chain of solar PV systems, there are multiple factors that hinder the integration of a circular economy approach. First are conflicting interests, where manufacturers may place cost effectiveness above environmental sustainability and recyclers favour profitability. Second, there is inadequate supportive legal frameworks or enforcement provisions that ensure compliance with proper waste management and recycling practices. Third, there is the absence of an effective communication or education campaign to enhances stakeholder awareness and knowledge on the benefits of implementing circular economy approaches for solar PV systems.
• RQ2: The role of stakeholders in the development of sustainable practices for end-of-life PV panels is essential, and it should be seen in collaboration and partnerships. There are some excellent examples of collaboration from the EU, and such best cases should be seen as models to be adopted across the world given the identified lack of a suitable framework to guide the CBM for the solar PV panel industry. They include extended producer responsibility (EPR) schemes which encourage manufacturers, importers and recyclers to cooperate, such as the Waste Electrical and Electronic Equipment (WEEE) Directive from the European Union [ 23 ] and voluntary take-back and recycling program by the PV CYCLE association in Europe [ 81 ]. In Australia, the Circular PV Alliance developed circular economy solutions for solar PV waste management as a collaborative initiative involving industry, government and research organisations [ 82 ]. Partnerships are a deeper form of collaboration between governments, manufacturers and recyclers to develop recycling infrastructure and promote circular economy practices, which is often also formalised with agreements. They express long-term commitments for finding solutions based on forecasting the scale and persistence of the issue [ 83 ]. Cooperation and partnerships can be adopted for stakeholders at different stages of the supply chain to avoid landfill treatment and to procure a green pathway for waste management [ 32 ].
• RQ3: We could not find an explicit answer to the question about how a country such as Australia should manage future solar PV panel waste, including through existing policies. The practical relevance and impact on current environmental and economic challenges are yet to be fully and properly addressed in the available literature. Only a few studies bridge the gap between research and real-world applications, including policy development, industry best practices and technological innovations for recycling and waste management. The review shows there is need for more interdisciplinary studies that combine technological solutions with the economic, environmental and social dimensions of the problem.

7. Future Research

• What will be the effects of the lack of raw materials for making solar panels or the effect of toxic substances released during the recovery of failed panels? Is the risk of recycling panels superior to the risk of causing negative health impacts?
• What is the estimated recovery rate and cost of reusing or recycling waste?
• If the recovery of solar panels creates jobs, would the increase in jobs created be significant?
• Given the lack of raw materials and the inability of some countries to produce or recover solar cells, when could solar PV technology be considered as a main method of energy production in the majority of countries?
• What are the circular policies, economic programs and circular technologies of advanced countries for solar PV systems?

8. Conclusions

Supplementary materials, author contributions, institutional review board statement, data availability statement, conflicts of interest.

• Oteng, D.; Zuo, J.; Sharifi, E. An expert-based evaluation on end-of-life solar photovoltaic management: An application of Fuzzy Delphi Technique. Sustain. Horiz. 2022 , 4 , 100036. [ Google Scholar ] [ CrossRef ]
• AL-Zoubi, O.H.; Shawaqfah, M.; Almomani, F.; Damash, R.A.; Al-Zboon, K. Photovoltaic Solar Cells and Panels Waste in Jordan: Figures, Facts, and Concerns. Sustainability 2022 , 14 , 13313. [ Google Scholar ] [ CrossRef ]
• Kiwan, S.; Al-Gharibeh, E. Jordan toward a 100% renewable electricity system. Renew. Energy 2019 , 147 , 423–436. [ Google Scholar ] [ CrossRef ]
• Herceg, S.; Pinto Bautista, S.; Weiß, K.-A. Influence of Waste Management on the Environmental Footprint of Electricity Produced by Photovoltaic Systems. Energies 2020 , 13 , 2146. [ Google Scholar ] [ CrossRef ]
• Vargas, C.; Chesney, M. End of life decommissioning and recycling of solar panels in the United States. A real options analysis. J. Sustain. Financ. Invest. 2021 , 11 , 82–102. [ Google Scholar ] [ CrossRef ]
• Bang, Y.-Y.; Hong, N.-J.; Lee, D.S.; Lim, S.-R. Comparative assessment of solar photovoltaic panels based on metal-derived hazardous waste, resource depletion, and toxicity potentials. Int. J. Green Energy 2018 , 15 , 550–557. [ Google Scholar ] [ CrossRef ]
• Nain, P.; Kumar, A. Initial metal contents and leaching rate constants of metals leached from end-of-life solar photovoltaic waste: An integrative literature review and analysis. Renew. Sustain. Energy Rev. 2020 , 119 , 109592. [ Google Scholar ] [ CrossRef ]
• Kwak, J.I.; Nam, S.-H.; Kim, L.; An, Y.-J. Potential environmental risk of solar cells: Current knowledge and future challenges. J. Hazard. Mater. 2020 , 392 , 122297. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Stamford, L.; Azapagic, A. Environmental impacts of photovoltaics: The effects of technological improvements and transfer of manufacturing from Europe to China. Energy Technol. 2018 , 6 , 1148–1160. [ Google Scholar ] [ CrossRef ]
• D’Adamo, I.; Miliacca, M.; Rosa, P. Economic Feasibility for Recycling of Waste Crystalline Silicon Photovoltaic Modules. Int. J. Photoenergy 2017 , 2017 , 4184676. [ Google Scholar ] [ CrossRef ]
• Yu, M.; Halog, A. Solar Photovoltaic Development in Australia—A Life Cycle Sustainability Assessment Study. Sustainability 2015 , 7 , 1213–1247. [ Google Scholar ] [ CrossRef ]
• Australian Government. Office of Northern Australia. Available online: https://www.infrastructure.gov.au/territories-regions-cities/regional-australia/office-northern-australia (accessed on 8 August 2024).
• Parkinson, G. Households to Be “Driving Force” of Energy Transition with Solar, Batteries and EVs. 2024. Available online: https://reneweconomy.com.au/households-to-be-driving-force-of-energy-transition-with-solar-batteries-and-evs/ (accessed on 8 August 2024).
• Barbosa, R.R.; Schultz, H.S.; Garcia, L.d.C.; Martins, D.D.; Carvalho, M. Economic and greenhouse gas assessments for two hot water industrial systems: Solar vs. natural gas. Clean. Eng. Technol. 2022 , 6 , 100365. [ Google Scholar ] [ CrossRef ]
• Lassio, J.G.; Branco, D.C.; Magrini, A.; Matos, D. Environmental life cycle-based analysis of fixed and single-axis tracking systems for photovoltaic power plants: A case study in Brazil. Clean. Eng. Technol. 2022 , 11 , 100586. [ Google Scholar ] [ CrossRef ]
• Gautam, A.; Shankar, R.; Vrat, P. End-of-life solar photovoltaic e-waste assessment in India: A step towards a circular economy. Sustain. Prod. Consum. 2021 , 26 , 65–77. [ Google Scholar ] [ CrossRef ]
• Irena, I.-p. End-of-Life Management: Solar Photovoltaic Panels ; International Renewable Energy Agency and International Energy Agency Photovoltaic Power Systems: Abu Dhabi, United Arab Emirates, 2016. [ Google Scholar ]
• Van Opstal, W.; Smeets, A. Circular economy strategies as enablers for solar PV adoption in organizational market segments. Sustain. Prod. Consum. 2023 , 35 , 40–54. [ Google Scholar ] [ CrossRef ]
• Ndzibah, E.; La Cruz, G.A.P.-D.; Shamsuzzoha, A. End of life analysis of solar photovoltaic panel: Roadmap for developing economies. Int. J. Energy Sect. Manag. 2022 , 16 , 112–128. [ Google Scholar ] [ CrossRef ]
• Majewski, P.; Al-shammari, W.; Dudley, M.; Jit, J.; Lee, S.-H.; Myoung-Kug, K.; Sung-Jim, K. Recycling of solar PV panels-product stewardship and regulatory approaches. Energy Policy 2021 , 149 , 112062. [ Google Scholar ] [ CrossRef ]
• Baldwin, S.; Bindewald, G.; Brown, A.; Chen, C.; Cheung, K.; Clark, C.; Cresko, J.; Crozat, M.; Daniels, J.; Edmonds, J. Quadrennial Technology Review: An Assessment of Energy Technologies and Research Opportunities ; US Department of Energy: Washington, DC, USA, 2015.
• Muteri, V.; Cellura, M.; Curto, D.; Franzitta, V.; Longo, S.; Mistretta, M.; Parisi, M.L. Review on life cycle assessment of solar photovoltaic panels. Energies 2020 , 13 , 252. [ Google Scholar ] [ CrossRef ]
• Radavičius, T.; van der Heide, A.; Palitzsch, W.; Rommens, T.; Denafas, J.; Tvaronavičienė, M. Circular solar industry supply chain through product technological design changes. Insights Reg. Dev. 2021 , 3 , 10–30. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Atasu, A.; Duran, S.; Van Wassenhove, L.N. The dark side of solar power. Harv. Bus. Rev. 2021 . Available online: https://hbr.org/2021/06/the-dark-side-of-solar-power (accessed on 8 August 2024).
• Bocken, N.M.; De Pauw, I.; Bakker, C.; Van Der Grinten, B. Product design and business model strategies for a circular economy. J. Ind. Prod. Eng. 2016 , 33 , 308–320. [ Google Scholar ] [ CrossRef ]
• Bocken, N.M.; Antikainen, M. Circular business model experimentation: Concept and approaches. In Proceedings of the Sustainable Design and Manufacturing 2018: Proceedings of the 5th International Conference on Sustainable Design and Manufacturing (KES-SDM-18) 5, Gold Coast, Australia, 24–26 June 2018. [ Google Scholar ]
• Moro, S.R.; Cauchick-Miguel, P.A.; Mendes, G.H.d.S. Adding sustainable value in product-service systems business models design: A conceptual review towards a framework proposal. Sustain. Prod. Consum. 2022 , 32 , 492–504. [ Google Scholar ] [ CrossRef ]
• Geels, F.W.; Sovacool, B.K.; Schwanen, T.; Sorrell, S. The socio-technical dynamics of low-carbon transitions. Joule 2017 , 1 , 463–479. [ Google Scholar ] [ CrossRef ]
• Oteng, D.; Zuo, J.; Sharifi, E. A scientometric review of trends in solar photovoltaic waste management research. Sol. Energy 2021 , 224 , 545–562. [ Google Scholar ] [ CrossRef ]
• ISO 14044 ; Environmental Management—Life Cycle Assessment—Requirements and Guidelines. International Organization for Standardization: Geneva, Switzerland, 2006.
• Ali, A.; Koch, T.W.; Volk, T.A.; Malmsheimer, R.W.; Eisenbies, M.H.; Kloster, D.; Brown, T.R.; Naim, N.; Therasme, O. The Environmental Life Cycle Assessment of Electricity Production in New York State from Distributed Solar Photovoltaic Systems. Energies 2022 , 15 , 7278. [ Google Scholar ] [ CrossRef ]
• Sheoran, M.; Kumar, P.; Sharma, S.; Bukya, M. Current situation analysis of solar PV waste management in India. Mater. Today Proc. 2022 , 58 , 773–782. [ Google Scholar ] [ CrossRef ]
• Pastuszak, J.; Węgierek, P. Photovoltaic cell generations and current research directions for their development. Materials 2022 , 15 , 5542. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Alami, A.H.; Alasad, S.; Aljaghoub, H.; Ayoub, M.; Alashkar, A.; Mdallal, A.; Hasan, R. First-Generation Photovoltaics: History and Conventional Manufacturing. In PV Technology and Manufacturing ; Alami, A.H., Ed.; Springer: Berlin/Heidelberg, Germany, 2023; pp. 1–17. [ Google Scholar ] [ CrossRef ]
• Aamir Iqbal, M.; Malik, M.; Shahid, W.; Zaheer Ud Din, S.; Anwar, N.; Ikram, M.; Idrees, F. Materials for Photovoltaics: Overview, Generations, Recent Advancements and Future Prospects ; IntechOpen: London, UK, 2022. [ Google Scholar ] [ CrossRef ]
• Conibeer, G. Third-generation photovoltaics. Mater. Today 2007 , 10 , 42–50. [ Google Scholar ] [ CrossRef ]
• Battaglia, C.; Cuevas, A.; De Wolf, S. High-efficiency crystalline silicon solar cells: Status and perspectives. Energy Environ. Sci. 2016 , 9 , 1552–1576. [ Google Scholar ] [ CrossRef ]
• Gao, J.; Tang, L.; Shen, Z.; Dong, Y.; Wang, Z.; Lyu, J.; Li, J.; Yu, H. Efficient solar-light conversion for optimizing the catalytic ozonation of gaseous toluene with PdOx-LaFeO 3 . Appl. Catal. B Environ. 2021 , 288 , 120004. [ Google Scholar ] [ CrossRef ]
• Heath, G.A.; Silverman, T.J.; Kempe, M.; Deceglie, M.; Ravikumar, D.; Remo, T.; Cui, H.; Sinha, P.; Libby, C.; Shaw, S.; et al. Research and development priorities for silicon photovoltaic module recycling to support a circular economy. Nat. Energy 2020 , 5 , 502–510. [ Google Scholar ] [ CrossRef ]
• Huang, B.; Zhao, J.; Chai, J.; Xue, B.; Zhao, F.; Wang, X. Environmental influence assessment of China’s multi-crystalline silicon (multi-Si) photovoltaic modules considering recycling process. Sol. Energy 2017 , 143 , 132–141. [ Google Scholar ] [ CrossRef ]
• Oteng, D.; Zuo, J.; Sharifi, E. An evaluation of the impact framework for product stewardship on end-of-life solar photovoltaic modules: An environmental lifecycle assessment. J. Clean. Prod. 2023 , 411 , 137357. [ Google Scholar ] [ CrossRef ]
• Tasnim, S.S.; Rahman, M.M.; Hasan, M.M.; Shammi, M.; Tareq, S.M. Current challenges and future perspectives of solar-PV cell waste in Bangladesh. Heliyon 2022 , 8 , e08970. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ndzibah, E.; Pinilla-De La Cruz, G.A.; Shamsuzzoha, A. Collaboration towards value creation for end-of-life solar photovoltaic panel in Ghana. J. Clean. Prod. 2022 , 333 , 129969. [ Google Scholar ] [ CrossRef ]
• Cherrington, R.; Goodship, V.; Longfield, A.; Kirwan, K. The feed-in tariff in the UK: A case study focus on domestic photovoltaic systems. Renew. Energy 2013 , 50 , 421–426. [ Google Scholar ] [ CrossRef ]
• Priyadarshini, P.; Abhilash, P.C. Circular economy practices within energy and waste management sectors of India: A meta-analysis. Bioresour. Technol. 2020 , 304 , 123018. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Snyder, H. Literature review as a research methodology: An overview and guidelines. J. Bus. Res. 2019 , 104 , 333–339. [ Google Scholar ] [ CrossRef ]
• Minunno, R.; O’Grady, T.; Morrison, G.M.; Gruner, R.L. Investigating the embodied energy and carbon of buildings: A systematic literature review and meta-analysis of life cycle assessments. Renew. Sustain. Energy Rev. 2021 , 143 , 110935. [ Google Scholar ] [ CrossRef ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, S.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Syst. Rev. 2021 , 10 , 89. [ Google Scholar ] [ CrossRef ]
• Scimago Journal & Country Rank. Sustainability. Available online: https://www.scimagojr.com/journalsearch.php?q=21100240100&tip=sid&clean=0 (accessed on 8 August 2024).
• Davy, E.; Hansen, U.E.; Nygaard, I. Localizing the solar value chain in Kenya? Innov. Dev. 2024 , 14 , 217–240. [ Google Scholar ] [ CrossRef ]
• Sah, D.; Chitra; Kumar, S. Investigation and recovery of copper from waste silicon solar module. Mater. Chem. Phys. 2023 , 296 , 127205. [ Google Scholar ] [ CrossRef ]
• Kolathur, S.; Khatiwada, D.; Khan, E.U. Life cycle assessment and life cycle costing of a building-scale, solar-driven water purification system. Energy Nexus 2023 , 10 , 100208. [ Google Scholar ] [ CrossRef ]
• Senthil, R. A technical note on integrating thermal energy systems into solar photovoltaic panels toward a circular economy. Renew. Energy Focus 2022 , 42 , 97–100. [ Google Scholar ] [ CrossRef ]
• Sinha, P. Life cycle materials and water management for CdTe photovoltaics. Sol. Energy Mater. Sol. Cells 2013 , 119 , 271–275. [ Google Scholar ] [ CrossRef ]
• Venkatachary, S.K.; Samikannu, R.; Murugesan, S.; Dasari, N.R.; Subramaniyam, R.U. Economics and im-pact of recycling solar waste materials on the environment and health care. Environ. Technol. Innov. 2020 , 20 , 101130. [ Google Scholar ] [ CrossRef ]
• Bonilla-Alicea, R.J.; Fu, K. Social life-cycle assessment (S-LCA) of residential rooftop solar panels using challenge-derived framework. Energy Sustain. Soc. 2022 , 12 , 7. [ Google Scholar ] [ CrossRef ]
• Maani, T.; Celik, I.; Heben, M.J.; Ellingson, R.J.; Apul, D. Environmental impacts of recycling crystalline silicon (c-SI) and cadmium telluride (CDTE) solar panels. Sci. Total Environ. 2020 , 735 , 138827. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Perkins, G. Techno-economic comparison of the levelised cost of electricity generation from solar PV and battery storage with solar PV and combustion of bio-crude using fast pyrolysis of biomass. Energy Convers. Manag. 2018 , 171 , 1573–1588. [ Google Scholar ] [ CrossRef ]
• Tannous, S.; Manneh, R.; Harajli, H.; El Zakhem, H. Comparative cradle-to-grave life cycle assessment of traditional grid connected and solar stand-alone street light systems: A case study for rural areas in Lebanon. J. Clean. Prod. 2018 , 186 , 963–977. [ Google Scholar ] [ CrossRef ]
• Farrell, C.; Osman, A.; Doherty, R.; Saad, M.; Zhang, X.; Murphy, A.; Harrison, J.; Vennard, A.; Kumaravel, V.; Al-Muhtaseb, A.; et al. Technical challenges and opportunities in realising a circular economy for waste photovoltaic modules. Renew. Sustain. Energy Rev. 2020 , 128 , 109911. [ Google Scholar ] [ CrossRef ]
• Rao, R.R.; Priyadarshani, S.; Mani, M. Examining the use of End-of-Life (EoL) PV panels in housing and sustainability. Sol. Energy 2023 , 257 , 210–220. [ Google Scholar ] [ CrossRef ]
• Lema, R.; Bhamidipati, P.L.; Gregersen, C.; Hansen, U.E.; Kirchherr, J. China’s investments in renewable energy in Africa: Creating co-benefits or just cashing-in? World Dev. 2021 , 141 , 105365. [ Google Scholar ] [ CrossRef ]
• Mkhwebane, E.; Ntuli, N. Alternatives for small, medium and micro scale enterprises participation in the renewable energy industry-small scale embedded generation review. J. Energy S. Afr. 2019 , 30 , 144–151. [ Google Scholar ] [ CrossRef ]
• Schmidt-Costa, J.R.; Uriona-Maldonado, M.; Possamai, O. Product-service systems in solar PV deployment programs: What can we learn from the California Solar Initiative? Resour. Conserv. Recy. 2018 , 140 , 145. [ Google Scholar ] [ CrossRef ]
• Borg, D.; Mont, O.; Schoonover, H. Consumer acceptance and value in use-oriented product-service systems: Lessons from Swedish consumer goods companies. Sustainability 2020 , 12 , 8079. [ Google Scholar ] [ CrossRef ]
• Lieder, M.; Rashid, A. Towards circular economy implementation: A comprehensive review in context of manufacturing industry. J. Clean. Prod. 2016 , 115 , 36–51. [ Google Scholar ] [ CrossRef ]
• Chowdhury, M.S.; Rahman, K.S.; Chowdhury, T.; Nuthammachot, N.; Techato, K.; Akhtaruzzaman, M.; Tiong, S.K.; Sopian, K.; Amin, N. An overview of solar photovoltaic panels’ end-of-life material recycling. Energy Strat. Rev. 2020 , 27 , 100431. [ Google Scholar ] [ CrossRef ]
• Guerin, T.F. Using integrated risk assessment to enable on-site re-purposing of construction wastes from renewable energy projects to confirm highest value re-use. Environ. Sci. Pollut. Res. 2022 , 29 , 8909–8920. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Franco, M.A.; Groesser, S.N. A systematic literature review of the solar photovoltaic value chain for a circular economy. Sustainability 2021 , 13 , 9615. [ Google Scholar ] [ CrossRef ]
• Zhang, Q.; Liu, C.; Zheng, S. Investment and pricing in solar photovoltaic waste recycling with government intervention: A supply chain perspective. Comput. Ind. Eng. 2023 , 177 , 109044. [ Google Scholar ] [ CrossRef ]
• Le LUU, Q.; Binh, V.; Nguyen, N.Q.; Nguyen, N.H. Life Cycle Assessment (LCA) of an Integrated Solar PV and Wind Power System in Vietnam. J. Asian Energy Stud. 2020 , 4 , 36–47. [ Google Scholar ] [ CrossRef ]
• Oteng, D.; Zuo, J.; Sharifi, E. Environmental emissions influencing solar photovoltaic waste management in Australia: An optimised system network of waste collection facilities. J. Environ. Manag. 2022 , 314 , 115007. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Volan, T.; Vaz, C.R.; Uriona-Maldonado, M. Scenarios for end-of-life (EOL) electric vehicle batteries in China. Rev. Gestão 2021 , 28 , 335–357. [ Google Scholar ] [ CrossRef ]
• Amiryar, M.E.; Pullen, K.R. Assessment of the Carbon and Cost Savings of a Combined Diesel Generator, Solar Photovoltaic, and Flywheel Energy Storage Islanded Grid System. Energies 2019 , 12 , 3356. [ Google Scholar ] [ CrossRef ]
• Maso, M.D.; Olsen, K.H.; Dong, Y.; Pedersen, M.B.; Hauschild, M.Z. Sustainable development impacts of nationally determined contributions: Assessing the case of mini-grids in Kenya. Clim. Policy 2020 , 20 , 815–831. [ Google Scholar ] [ CrossRef ]
• Reim, W.; Parida, V.; Örtqvist, D. Product–Service Systems (PSS) business models and tactics—A systematic literature review. J. Clean. Prod. 2015 , 97 , 61–75. [ Google Scholar ] [ CrossRef ]
• Maqbool, R.; Akubo, S.A.; Rashid, Y.; Ashfaq, S.; Saiba, M.R. Solar energy and the circular economy policies for sustainable management. In Solar Energy Harvesting, Conversion, and Storage: Materials, Technologies, and Applications ; Khalid, M., Walvekar, R., Panchal, H., Vaka, M., Eds.; Elsevier: Amsterdam, The Netherlands, 2023; pp. 363–376. [ Google Scholar ] [ CrossRef ]
• Gerbinet, S.; Belboom, S.; Léonard, A. Life Cycle Analysis (LCA) of photovoltaic panels: A review. Renew. Sustain. Energy Rev. 2014 , 38 , 747–753. [ Google Scholar ] [ CrossRef ]
• Kabir, E.; Kumar, P.; Kumar, S.; Adelodun, A.A.; Kim, K.H. Solar energy: Potential and future prospects. Renew. Sustain. Energy Rev. 2018 , 82 , 894–900. [ Google Scholar ] [ CrossRef ]
• Klugmann-Radziemska, E.; Ostrowski, P. Chemical treatment of crystalline silicon solar cells as a method of recovering pure silicon from photovoltaic modules. Renew. Energy 2010 , 35 , 1751–1759. [ Google Scholar ] [ CrossRef ]
• Institute for Energy Economics and Financial Analysis [IEEFA]. Role of Recycling for Resilient Solar Photovoltaic Supply Chains. (2 April 2024). Available online: https://ieefa.org/resources/role-recycling-resilient-solar-photovoltaic-supply-chains (accessed on 8 August 2024).
• Circular PV Alliance. Research|Circular PV Alliance. Available online: https://www.circularpv.com.au/research (accessed on 8 August 2024).
• Mahmoudi, S.; Huda, N.; Behnia, M. Photovoltaic waste assessment: Forecasting and screening of emerging waste in Australia. Resour. Conserv. Recycl. 2019 , 146 , 192–205. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Amrollahi Biyouki, Z.; Zaman, A.; Marinova, D.; Minunno, R.; Askari Shayegan, M. Solar Photovoltaics Value Chain and End-of-Life Management Practices: A Systematic Literature Review. Sustainability 2024 , 16 , 7038. https://doi.org/10.3390/su16167038

Amrollahi Biyouki Z, Zaman A, Marinova D, Minunno R, Askari Shayegan M. Solar Photovoltaics Value Chain and End-of-Life Management Practices: A Systematic Literature Review. Sustainability . 2024; 16(16):7038. https://doi.org/10.3390/su16167038

Amrollahi Biyouki, Zahra, Atiq Zaman, Dora Marinova, Roberto Minunno, and Maryam Askari Shayegan. 2024. "Solar Photovoltaics Value Chain and End-of-Life Management Practices: A Systematic Literature Review" Sustainability 16, no. 16: 7038. https://doi.org/10.3390/su16167038

Article Metrics

Article access statistics, supplementary material.

ZIP-Document (ZIP, 93 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals