literature in qualitative research

Archer Library

Qualitative research: literature review .

• Archer Library This link opens in a new window
• Schedule a Reference Appointment This link opens in a new window
• Qualitative Research Handout This link opens in a new window
• Locating Books
• ebook Collections This link opens in a new window
• A to Z Database List This link opens in a new window
• Research & Stats
• Literature Review Resources
• Citation & Reference

Exploring the literature review 

Literature review model: 6 steps.

Adapted from The Literature Review , Machi & McEvoy (2009, p. 13).

Your Literature Review

Step 2: search, boolean search strategies, search limiters, ★ ebsco & google drive.

1. Select a Topic

"All research begins with curiosity" (Machi & McEvoy, 2009, p. 14)

Selection of a topic, and fully defined research interest and question, is supervised (and approved) by your professor. Tips for crafting your topic include:

• Be specific. Take time to define your interest.
• Topic Focus. Fully describe and sufficiently narrow the focus for research.
• Academic Discipline. Learn more about your area of research & refine the scope.
• Avoid Bias. Be aware of bias that you (as a researcher) may have.
• Document your research. Use Google Docs to track your research process.
• Research apps. Consider using Evernote or Zotero to track your research.

Consider Purpose

What will your topic and research address?

In The Literature Review: A Step-by-Step Guide for Students , Ridley presents that literature reviews serve several purposes (2008, p. 16-17).  Included are the following points:

• Historical background for the research;
• Overview of current field provided by "contemporary debates, issues, and questions;"
• Theories and concepts related to your research;
• Introduce "relevant terminology" - or academic language - being used it the field;
• Connect to existing research - does your work "extend or challenge [this] or address a gap;" 
• Provide "supporting evidence for a practical problem or issue" that your research addresses.

★ Schedule a research appointment

At this point in your literature review, take time to meet with a librarian. Why? Understanding the subject terminology used in databases can be challenging. Archer Librarians can help you structure a search, preparing you for step two. How? Contact a librarian directly or use the online form to schedule an appointment. Details are provided in the adjacent Schedule an Appointment box.

2. Search the Literature

Collect & Select Data: Preview, select, and organize

AU Library is your go-to resource for this step in your literature review process. The literature search will include books and ebooks, scholarly and practitioner journals, theses and dissertations, and indexes. You may also choose to include web sites, blogs, open access resources, and newspapers. This library guide provides access to resources needed to complete a literature review.

Books & eBooks: Archer Library & OhioLINK

Databases: Scholarly & Practitioner Journals

Review the Library Databases tab on this library guide, it provides links to recommended databases for Education & Psychology, Business, and General & Social Sciences.

Expand your journal search; a complete listing of available AU Library and OhioLINK databases is available on the Databases  A to Z list . Search the database by subject, type, name, or do use the search box for a general title search. The A to Z list also includes open access resources and select internet sites.

Databases: Theses & Dissertations

Review the Library Databases tab on this guide, it includes Theses & Dissertation resources. AU library also has AU student authored theses and dissertations available in print, search the library catalog for these titles.

Did you know? If you are looking for particular chapters within a dissertation that is not fully available online, it is possible to submit an ILL article request . Do this instead of requesting the entire dissertation.

Newspapers:  Databases & Internet

Consider current literature in your academic field. AU Library's database collection includes The Chronicle of Higher Education and The Wall Street Journal .  The Internet Resources tab in this guide provides links to newspapers and online journals such as Inside Higher Ed , COABE Journal , and Education Week .

The Chronicle of Higher Education has the nation’s largest newsroom dedicated to covering colleges and universities.  Source of news, information, and jobs for college and university faculty members and administrators

The Chronicle features complete contents of the latest print issue; daily news and advice columns; current job listings; archive of previously published content; discussion forums; and career-building tools such as online CV management and salary databases. Dates covered: 1970-present.

Offers in-depth coverage of national and international business and finance as well as first-rate coverage of hard news--all from America's premier financial newspaper. Covers complete bibliographic information and also subjects, companies, people, products, and geographic areas. 

Comprehensive coverage back to 1984 is available from the world's leading financial newspaper through the ProQuest database. 

Newspaper Source provides cover-to-cover full text for hundreds of national (U.S.), international and regional newspapers. In addition, it offers television and radio news transcripts from major networks.

Provides complete television and radio news transcripts from CBS News, CNN, CNN International, FOX News, and more.

Search Strategies & Boolean Operators

There are three basic boolean operators:  AND, OR, and NOT.

Used with your search terms, boolean operators will either expand or limit results. What purpose do they serve? They help to define the relationship between your search terms. For example, using the operator AND will combine the terms expanding the search. When searching some databases, and Google, the operator AND may be implied.

Overview of boolean terms

About the example: Boolean searches were conducted on November 4, 2019; result numbers may vary at a later date. No additional database limiters were set to further narrow search returns.

Database Search Limiters

Database strategies for targeted search results.

Most databases include limiters, or additional parameters, you may use to strategically focus search results.  EBSCO databases, such as Education Research Complete & Academic Search Complete provide options to:

• Limit results to full text;
• Limit results to scholarly journals, and reference available;
• Select results source type to journals, magazines, conference papers, reviews, and newspapers
• Publication date

Keep in mind that these tools are defined as limiters for a reason; adding them to a search will limit the number of results returned.  This can be a double-edged sword.  How? 

• If limiting results to full-text only, you may miss an important piece of research that could change the direction of your research. Interlibrary loan is available to students, free of charge. Request articles that are not available in full-text; they will be sent to you via email.
• If narrowing publication date, you may eliminate significant historical - or recent - research conducted on your topic.
• Limiting resource type to a specific type of material may cause bias in the research results.

Use limiters with care. When starting a search, consider opting out of limiters until the initial literature screening is complete. The second or third time through your research may be the ideal time to focus on specific time periods or material (scholarly vs newspaper).

★ Truncating Search Terms

Expanding your search term at the root.

Truncating is often referred to as 'wildcard' searching. Databases may have their own specific wildcard elements however, the most commonly used are the asterisk (*) or question mark (?).  When used within your search. they will expand returned results.

Asterisk (*) Wildcard

Using the asterisk wildcard will return varied spellings of the truncated word. In the following example, the search term education was truncated after the letter "t."

Explore these database help pages for additional information on crafting search terms.

• EBSCO Connect: Searching with Wildcards and Truncation Symbols
• EBSCO Connect: Searching with Boolean Operators
• EBSCO Connect: EBSCOhost Search Tips
• EBSCO Connect: Basic Searching with EBSCO
• ProQuest Help: Search Tips
• ERIC: How does ERIC search work?

★ EBSCO Databases & Google Drive

Tips for saving research directly to Google drive.

Researching in an EBSCO database?

It is possible to save articles (PDF and HTML) and abstracts in EBSCOhost databases directly to Google drive. Select the Google Drive icon, authenticate using a Google account, and an EBSCO folder will be created in your account. This is a great option for managing your research. If documenting your research in a Google Doc, consider linking the information to actual articles saved in drive.

EBSCO Databases & Google Drive

EBSCOHost Databases & Google Drive: Managing your Research

This video features an overview of how to use Google Drive with EBSCO databases to help manage your research. It presents information for connecting an active Google account to EBSCO and steps needed to provide permission for EBSCO to manage a folder in Drive.

About the Video:  Closed captioning is available, select CC from the video menu.  If you need to review a specific area on the video, view on YouTube and expand the video description for access to topic time stamps.  A video transcript is provided below.

• EBSCOhost Databases & Google Scholar

Defining Literature Review

What is a literature review.

A definition from the Online Dictionary for Library and Information Sciences .

A literature review is "a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works" (Reitz, 2014). 

A systemic review is "a literature review focused on a specific research question, which uses explicit methods to minimize bias in the identification, appraisal, selection, and synthesis of all the high-quality evidence pertinent to the question" (Reitz, 2014).

Recommended Reading

About this page

EBSCO Connect [Discovery and Search]. (2022). Searching with boolean operators. Retrieved May, 3, 2022 from https://connect.ebsco.com/s/?language=en_US

EBSCO Connect [Discover and Search]. (2022). Searching with wildcards and truncation symbols. Retrieved May 3, 2022; https://connect.ebsco.com/s/?language=en_US

Machi, L.A. & McEvoy, B.T. (2009). The literature review . Thousand Oaks, CA: Corwin Press: 

Reitz, J.M. (2014). Online dictionary for library and information science. ABC-CLIO, Libraries Unlimited . Retrieved from https://www.abc-clio.com/ODLIS/odlis_A.aspx

Ridley, D. (2008). The literature review: A step-by-step guide for students . Thousand Oaks, CA: Sage Publications, Inc.

Archer Librarians

Schedule an appointment.

Contact a librarian directly (email), or submit a request form. If you have worked with someone before, you can request them on the form.

• ★ Archer Library Help • Online Reqest Form
• Carrie Halquist • Reference & Instruction
• Jessica Byers • Reference & Curation
• Don Reams • Corrections Education & Reference
• Diane Schrecker • Education & Head of the IRC
• Tanaya Silcox • Technical Services & Business
• Sarah Thomas • Acquisitions & ATS Librarian
• << Previous: Research & Stats
• Next: Literature Review Resources >>
• Last Updated: Jun 27, 2024 11:14 AM
• URL: https://libguides.ashland.edu/qualitative

Archer Library • Ashland University © Copyright 2023. An Equal Opportunity/Equal Access Institution.

Criteria for Good Qualitative Research: A Comprehensive Review

• Regular Article
• Open access
• Published: 18 September 2021
• Volume 31 , pages 679–689, ( 2022 )

Cite this article

You have full access to this open access article

• Drishti Yadav   ORCID: orcid.org/0000-0002-2974-0323 1  

95k Accesses

41 Citations

70 Altmetric

Explore all metrics

This review aims to synthesize a published set of evaluative criteria for good qualitative research. The aim is to shed light on existing standards for assessing the rigor of qualitative research encompassing a range of epistemological and ontological standpoints. Using a systematic search strategy, published journal articles that deliberate criteria for rigorous research were identified. Then, references of relevant articles were surveyed to find noteworthy, distinct, and well-defined pointers to good qualitative research. This review presents an investigative assessment of the pivotal features in qualitative research that can permit the readers to pass judgment on its quality and to condemn it as good research when objectively and adequately utilized. Overall, this review underlines the crux of qualitative research and accentuates the necessity to evaluate such research by the very tenets of its being. It also offers some prospects and recommendations to improve the quality of qualitative research. Based on the findings of this review, it is concluded that quality criteria are the aftereffect of socio-institutional procedures and existing paradigmatic conducts. Owing to the paradigmatic diversity of qualitative research, a single and specific set of quality criteria is neither feasible nor anticipated. Since qualitative research is not a cohesive discipline, researchers need to educate and familiarize themselves with applicable norms and decisive factors to evaluate qualitative research from within its theoretical and methodological framework of origin.

Similar content being viewed by others

Good Qualitative Research: Opening up the Debate

Beyond qualitative/quantitative structuralism: the positivist qualitative research and the paradigmatic disclaimer.

What is Qualitative in Research

Avoid common mistakes on your manuscript.

Introduction

“… It is important to regularly dialogue about what makes for good qualitative research” (Tracy, 2010 , p. 837)

To decide what represents good qualitative research is highly debatable. There are numerous methods that are contained within qualitative research and that are established on diverse philosophical perspectives. Bryman et al., ( 2008 , p. 262) suggest that “It is widely assumed that whereas quality criteria for quantitative research are well‐known and widely agreed, this is not the case for qualitative research.” Hence, the question “how to evaluate the quality of qualitative research” has been continuously debated. There are many areas of science and technology wherein these debates on the assessment of qualitative research have taken place. Examples include various areas of psychology: general psychology (Madill et al., 2000 ); counseling psychology (Morrow, 2005 ); and clinical psychology (Barker & Pistrang, 2005 ), and other disciplines of social sciences: social policy (Bryman et al., 2008 ); health research (Sparkes, 2001 ); business and management research (Johnson et al., 2006 ); information systems (Klein & Myers, 1999 ); and environmental studies (Reid & Gough, 2000 ). In the literature, these debates are enthused by the impression that the blanket application of criteria for good qualitative research developed around the positivist paradigm is improper. Such debates are based on the wide range of philosophical backgrounds within which qualitative research is conducted (e.g., Sandberg, 2000 ; Schwandt, 1996 ). The existence of methodological diversity led to the formulation of different sets of criteria applicable to qualitative research.

Among qualitative researchers, the dilemma of governing the measures to assess the quality of research is not a new phenomenon, especially when the virtuous triad of objectivity, reliability, and validity (Spencer et al., 2004 ) are not adequate. Occasionally, the criteria of quantitative research are used to evaluate qualitative research (Cohen & Crabtree, 2008 ; Lather, 2004 ). Indeed, Howe ( 2004 ) claims that the prevailing paradigm in educational research is scientifically based experimental research. Hypotheses and conjectures about the preeminence of quantitative research can weaken the worth and usefulness of qualitative research by neglecting the prominence of harmonizing match for purpose on research paradigm, the epistemological stance of the researcher, and the choice of methodology. Researchers have been reprimanded concerning this in “paradigmatic controversies, contradictions, and emerging confluences” (Lincoln & Guba, 2000 ).

In general, qualitative research tends to come from a very different paradigmatic stance and intrinsically demands distinctive and out-of-the-ordinary criteria for evaluating good research and varieties of research contributions that can be made. This review attempts to present a series of evaluative criteria for qualitative researchers, arguing that their choice of criteria needs to be compatible with the unique nature of the research in question (its methodology, aims, and assumptions). This review aims to assist researchers in identifying some of the indispensable features or markers of high-quality qualitative research. In a nutshell, the purpose of this systematic literature review is to analyze the existing knowledge on high-quality qualitative research and to verify the existence of research studies dealing with the critical assessment of qualitative research based on the concept of diverse paradigmatic stances. Contrary to the existing reviews, this review also suggests some critical directions to follow to improve the quality of qualitative research in different epistemological and ontological perspectives. This review is also intended to provide guidelines for the acceleration of future developments and dialogues among qualitative researchers in the context of assessing the qualitative research.

The rest of this review article is structured in the following fashion: Sect.  Methods describes the method followed for performing this review. Section Criteria for Evaluating Qualitative Studies provides a comprehensive description of the criteria for evaluating qualitative studies. This section is followed by a summary of the strategies to improve the quality of qualitative research in Sect.  Improving Quality: Strategies . Section  How to Assess the Quality of the Research Findings? provides details on how to assess the quality of the research findings. After that, some of the quality checklists (as tools to evaluate quality) are discussed in Sect.  Quality Checklists: Tools for Assessing the Quality . At last, the review ends with the concluding remarks presented in Sect.  Conclusions, Future Directions and Outlook . Some prospects in qualitative research for enhancing its quality and usefulness in the social and techno-scientific research community are also presented in Sect.  Conclusions, Future Directions and Outlook .

For this review, a comprehensive literature search was performed from many databases using generic search terms such as Qualitative Research , Criteria , etc . The following databases were chosen for the literature search based on the high number of results: IEEE Explore, ScienceDirect, PubMed, Google Scholar, and Web of Science. The following keywords (and their combinations using Boolean connectives OR/AND) were adopted for the literature search: qualitative research, criteria, quality, assessment, and validity. The synonyms for these keywords were collected and arranged in a logical structure (see Table 1 ). All publications in journals and conference proceedings later than 1950 till 2021 were considered for the search. Other articles extracted from the references of the papers identified in the electronic search were also included. A large number of publications on qualitative research were retrieved during the initial screening. Hence, to include the searches with the main focus on criteria for good qualitative research, an inclusion criterion was utilized in the search string.

From the selected databases, the search retrieved a total of 765 publications. Then, the duplicate records were removed. After that, based on the title and abstract, the remaining 426 publications were screened for their relevance by using the following inclusion and exclusion criteria (see Table 2 ). Publications focusing on evaluation criteria for good qualitative research were included, whereas those works which delivered theoretical concepts on qualitative research were excluded. Based on the screening and eligibility, 45 research articles were identified that offered explicit criteria for evaluating the quality of qualitative research and were found to be relevant to this review.

Figure  1 illustrates the complete review process in the form of PRISMA flow diagram. PRISMA, i.e., “preferred reporting items for systematic reviews and meta-analyses” is employed in systematic reviews to refine the quality of reporting.

PRISMA flow diagram illustrating the search and inclusion process. N represents the number of records

Criteria for Evaluating Qualitative Studies

Fundamental criteria: general research quality.

Various researchers have put forward criteria for evaluating qualitative research, which have been summarized in Table 3 . Also, the criteria outlined in Table 4 effectively deliver the various approaches to evaluate and assess the quality of qualitative work. The entries in Table 4 are based on Tracy’s “Eight big‐tent criteria for excellent qualitative research” (Tracy, 2010 ). Tracy argues that high-quality qualitative work should formulate criteria focusing on the worthiness, relevance, timeliness, significance, morality, and practicality of the research topic, and the ethical stance of the research itself. Researchers have also suggested a series of questions as guiding principles to assess the quality of a qualitative study (Mays & Pope, 2020 ). Nassaji ( 2020 ) argues that good qualitative research should be robust, well informed, and thoroughly documented.

Qualitative Research: Interpretive Paradigms

All qualitative researchers follow highly abstract principles which bring together beliefs about ontology, epistemology, and methodology. These beliefs govern how the researcher perceives and acts. The net, which encompasses the researcher’s epistemological, ontological, and methodological premises, is referred to as a paradigm, or an interpretive structure, a “Basic set of beliefs that guides action” (Guba, 1990 ). Four major interpretive paradigms structure the qualitative research: positivist and postpositivist, constructivist interpretive, critical (Marxist, emancipatory), and feminist poststructural. The complexity of these four abstract paradigms increases at the level of concrete, specific interpretive communities. Table 5 presents these paradigms and their assumptions, including their criteria for evaluating research, and the typical form that an interpretive or theoretical statement assumes in each paradigm. Moreover, for evaluating qualitative research, quantitative conceptualizations of reliability and validity are proven to be incompatible (Horsburgh, 2003 ). In addition, a series of questions have been put forward in the literature to assist a reviewer (who is proficient in qualitative methods) for meticulous assessment and endorsement of qualitative research (Morse, 2003 ). Hammersley ( 2007 ) also suggests that guiding principles for qualitative research are advantageous, but methodological pluralism should not be simply acknowledged for all qualitative approaches. Seale ( 1999 ) also points out the significance of methodological cognizance in research studies.

Table 5 reflects that criteria for assessing the quality of qualitative research are the aftermath of socio-institutional practices and existing paradigmatic standpoints. Owing to the paradigmatic diversity of qualitative research, a single set of quality criteria is neither possible nor desirable. Hence, the researchers must be reflexive about the criteria they use in the various roles they play within their research community.

Improving Quality: Strategies

Another critical question is “How can the qualitative researchers ensure that the abovementioned quality criteria can be met?” Lincoln and Guba ( 1986 ) delineated several strategies to intensify each criteria of trustworthiness. Other researchers (Merriam & Tisdell, 2016 ; Shenton, 2004 ) also presented such strategies. A brief description of these strategies is shown in Table 6 .

It is worth mentioning that generalizability is also an integral part of qualitative research (Hays & McKibben, 2021 ). In general, the guiding principle pertaining to generalizability speaks about inducing and comprehending knowledge to synthesize interpretive components of an underlying context. Table 7 summarizes the main metasynthesis steps required to ascertain generalizability in qualitative research.

Figure  2 reflects the crucial components of a conceptual framework and their contribution to decisions regarding research design, implementation, and applications of results to future thinking, study, and practice (Johnson et al., 2020 ). The synergy and interrelationship of these components signifies their role to different stances of a qualitative research study.

Essential elements of a conceptual framework

In a nutshell, to assess the rationale of a study, its conceptual framework and research question(s), quality criteria must take account of the following: lucid context for the problem statement in the introduction; well-articulated research problems and questions; precise conceptual framework; distinct research purpose; and clear presentation and investigation of the paradigms. These criteria would expedite the quality of qualitative research.

How to Assess the Quality of the Research Findings?

The inclusion of quotes or similar research data enhances the confirmability in the write-up of the findings. The use of expressions (for instance, “80% of all respondents agreed that” or “only one of the interviewees mentioned that”) may also quantify qualitative findings (Stenfors et al., 2020 ). On the other hand, the persuasive reason for “why this may not help in intensifying the research” has also been provided (Monrouxe & Rees, 2020 ). Further, the Discussion and Conclusion sections of an article also prove robust markers of high-quality qualitative research, as elucidated in Table 8 .

Quality Checklists: Tools for Assessing the Quality

Numerous checklists are available to speed up the assessment of the quality of qualitative research. However, if used uncritically and recklessly concerning the research context, these checklists may be counterproductive. I recommend that such lists and guiding principles may assist in pinpointing the markers of high-quality qualitative research. However, considering enormous variations in the authors’ theoretical and philosophical contexts, I would emphasize that high dependability on such checklists may say little about whether the findings can be applied in your setting. A combination of such checklists might be appropriate for novice researchers. Some of these checklists are listed below:

The most commonly used framework is Consolidated Criteria for Reporting Qualitative Research (COREQ) (Tong et al., 2007 ). This framework is recommended by some journals to be followed by the authors during article submission.

Standards for Reporting Qualitative Research (SRQR) is another checklist that has been created particularly for medical education (O’Brien et al., 2014 ).

Also, Tracy ( 2010 ) and Critical Appraisal Skills Programme (CASP, 2021 ) offer criteria for qualitative research relevant across methods and approaches.

Further, researchers have also outlined different criteria as hallmarks of high-quality qualitative research. For instance, the “Road Trip Checklist” (Epp & Otnes, 2021 ) provides a quick reference to specific questions to address different elements of high-quality qualitative research.

Conclusions, Future Directions, and Outlook

This work presents a broad review of the criteria for good qualitative research. In addition, this article presents an exploratory analysis of the essential elements in qualitative research that can enable the readers of qualitative work to judge it as good research when objectively and adequately utilized. In this review, some of the essential markers that indicate high-quality qualitative research have been highlighted. I scope them narrowly to achieve rigor in qualitative research and note that they do not completely cover the broader considerations necessary for high-quality research. This review points out that a universal and versatile one-size-fits-all guideline for evaluating the quality of qualitative research does not exist. In other words, this review also emphasizes the non-existence of a set of common guidelines among qualitative researchers. In unison, this review reinforces that each qualitative approach should be treated uniquely on account of its own distinctive features for different epistemological and disciplinary positions. Owing to the sensitivity of the worth of qualitative research towards the specific context and the type of paradigmatic stance, researchers should themselves analyze what approaches can be and must be tailored to ensemble the distinct characteristics of the phenomenon under investigation. Although this article does not assert to put forward a magic bullet and to provide a one-stop solution for dealing with dilemmas about how, why, or whether to evaluate the “goodness” of qualitative research, it offers a platform to assist the researchers in improving their qualitative studies. This work provides an assembly of concerns to reflect on, a series of questions to ask, and multiple sets of criteria to look at, when attempting to determine the quality of qualitative research. Overall, this review underlines the crux of qualitative research and accentuates the need to evaluate such research by the very tenets of its being. Bringing together the vital arguments and delineating the requirements that good qualitative research should satisfy, this review strives to equip the researchers as well as reviewers to make well-versed judgment about the worth and significance of the qualitative research under scrutiny. In a nutshell, a comprehensive portrayal of the research process (from the context of research to the research objectives, research questions and design, speculative foundations, and from approaches of collecting data to analyzing the results, to deriving inferences) frequently proliferates the quality of a qualitative research.

Prospects : A Road Ahead for Qualitative Research

Irrefutably, qualitative research is a vivacious and evolving discipline wherein different epistemological and disciplinary positions have their own characteristics and importance. In addition, not surprisingly, owing to the sprouting and varied features of qualitative research, no consensus has been pulled off till date. Researchers have reflected various concerns and proposed several recommendations for editors and reviewers on conducting reviews of critical qualitative research (Levitt et al., 2021 ; McGinley et al., 2021 ). Following are some prospects and a few recommendations put forward towards the maturation of qualitative research and its quality evaluation:

In general, most of the manuscript and grant reviewers are not qualitative experts. Hence, it is more likely that they would prefer to adopt a broad set of criteria. However, researchers and reviewers need to keep in mind that it is inappropriate to utilize the same approaches and conducts among all qualitative research. Therefore, future work needs to focus on educating researchers and reviewers about the criteria to evaluate qualitative research from within the suitable theoretical and methodological context.

There is an urgent need to refurbish and augment critical assessment of some well-known and widely accepted tools (including checklists such as COREQ, SRQR) to interrogate their applicability on different aspects (along with their epistemological ramifications).

Efforts should be made towards creating more space for creativity, experimentation, and a dialogue between the diverse traditions of qualitative research. This would potentially help to avoid the enforcement of one's own set of quality criteria on the work carried out by others.

Moreover, journal reviewers need to be aware of various methodological practices and philosophical debates.

It is pivotal to highlight the expressions and considerations of qualitative researchers and bring them into a more open and transparent dialogue about assessing qualitative research in techno-scientific, academic, sociocultural, and political rooms.

Frequent debates on the use of evaluative criteria are required to solve some potentially resolved issues (including the applicability of a single set of criteria in multi-disciplinary aspects). Such debates would not only benefit the group of qualitative researchers themselves, but primarily assist in augmenting the well-being and vivacity of the entire discipline.

To conclude, I speculate that the criteria, and my perspective, may transfer to other methods, approaches, and contexts. I hope that they spark dialog and debate – about criteria for excellent qualitative research and the underpinnings of the discipline more broadly – and, therefore, help improve the quality of a qualitative study. Further, I anticipate that this review will assist the researchers to contemplate on the quality of their own research, to substantiate research design and help the reviewers to review qualitative research for journals. On a final note, I pinpoint the need to formulate a framework (encompassing the prerequisites of a qualitative study) by the cohesive efforts of qualitative researchers of different disciplines with different theoretic-paradigmatic origins. I believe that tailoring such a framework (of guiding principles) paves the way for qualitative researchers to consolidate the status of qualitative research in the wide-ranging open science debate. Dialogue on this issue across different approaches is crucial for the impending prospects of socio-techno-educational research.

Amin, M. E. K., Nørgaard, L. S., Cavaco, A. M., Witry, M. J., Hillman, L., Cernasev, A., & Desselle, S. P. (2020). Establishing trustworthiness and authenticity in qualitative pharmacy research. Research in Social and Administrative Pharmacy, 16 (10), 1472–1482.

Article   Google Scholar  

Barker, C., & Pistrang, N. (2005). Quality criteria under methodological pluralism: Implications for conducting and evaluating research. American Journal of Community Psychology, 35 (3–4), 201–212.

Bryman, A., Becker, S., & Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11 (4), 261–276.

Caelli, K., Ray, L., & Mill, J. (2003). ‘Clear as mud’: Toward greater clarity in generic qualitative research. International Journal of Qualitative Methods, 2 (2), 1–13.

CASP (2021). CASP checklists. Retrieved May 2021 from https://casp-uk.net/casp-tools-checklists/

Cohen, D. J., & Crabtree, B. F. (2008). Evaluative criteria for qualitative research in health care: Controversies and recommendations. The Annals of Family Medicine, 6 (4), 331–339.

Denzin, N. K., & Lincoln, Y. S. (2005). Introduction: The discipline and practice of qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), The sage handbook of qualitative research (pp. 1–32). Sage Publications Ltd.

Google Scholar  

Elliott, R., Fischer, C. T., & Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38 (3), 215–229.

Epp, A. M., & Otnes, C. C. (2021). High-quality qualitative research: Getting into gear. Journal of Service Research . https://doi.org/10.1177/1094670520961445

Guba, E. G. (1990). The paradigm dialog. In Alternative paradigms conference, mar, 1989, Indiana u, school of education, San Francisco, ca, us . Sage Publications, Inc.

Hammersley, M. (2007). The issue of quality in qualitative research. International Journal of Research and Method in Education, 30 (3), 287–305.

Haven, T. L., Errington, T. M., Gleditsch, K. S., van Grootel, L., Jacobs, A. M., Kern, F. G., & Mokkink, L. B. (2020). Preregistering qualitative research: A Delphi study. International Journal of Qualitative Methods, 19 , 1609406920976417.

Hays, D. G., & McKibben, W. B. (2021). Promoting rigorous research: Generalizability and qualitative research. Journal of Counseling and Development, 99 (2), 178–188.

Horsburgh, D. (2003). Evaluation of qualitative research. Journal of Clinical Nursing, 12 (2), 307–312.

Howe, K. R. (2004). A critique of experimentalism. Qualitative Inquiry, 10 (1), 42–46.

Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. American Journal of Pharmaceutical Education, 84 (1), 7120.

Johnson, P., Buehring, A., Cassell, C., & Symon, G. (2006). Evaluating qualitative management research: Towards a contingent criteriology. International Journal of Management Reviews, 8 (3), 131–156.

Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23 (1), 67–93.

Lather, P. (2004). This is your father’s paradigm: Government intrusion and the case of qualitative research in education. Qualitative Inquiry, 10 (1), 15–34.

Levitt, H. M., Morrill, Z., Collins, K. M., & Rizo, J. L. (2021). The methodological integrity of critical qualitative research: Principles to support design and research review. Journal of Counseling Psychology, 68 (3), 357.

Lincoln, Y. S., & Guba, E. G. (1986). But is it rigorous? Trustworthiness and authenticity in naturalistic evaluation. New Directions for Program Evaluation, 1986 (30), 73–84.

Lincoln, Y. S., & Guba, E. G. (2000). Paradigmatic controversies, contradictions and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (2nd ed., pp. 163–188). Sage Publications.

Madill, A., Jordan, A., & Shirley, C. (2000). Objectivity and reliability in qualitative analysis: Realist, contextualist and radical constructionist epistemologies. British Journal of Psychology, 91 (1), 1–20.

Mays, N., & Pope, C. (2020). Quality in qualitative research. Qualitative Research in Health Care . https://doi.org/10.1002/9781119410867.ch15

McGinley, S., Wei, W., Zhang, L., & Zheng, Y. (2021). The state of qualitative research in hospitality: A 5-year review 2014 to 2019. Cornell Hospitality Quarterly, 62 (1), 8–20.

Merriam, S., & Tisdell, E. (2016). Qualitative research: A guide to design and implementation. San Francisco, US.

Meyer, M., & Dykes, J. (2019). Criteria for rigor in visualization design study. IEEE Transactions on Visualization and Computer Graphics, 26 (1), 87–97.

Monrouxe, L. V., & Rees, C. E. (2020). When I say… quantification in qualitative research. Medical Education, 54 (3), 186–187.

Morrow, S. L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52 (2), 250.

Morse, J. M. (2003). A review committee’s guide for evaluating qualitative proposals. Qualitative Health Research, 13 (6), 833–851.

Nassaji, H. (2020). Good qualitative research. Language Teaching Research, 24 (4), 427–431.

O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89 (9), 1245–1251.

O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19 , 1609406919899220.

Reid, A., & Gough, S. (2000). Guidelines for reporting and evaluating qualitative research: What are the alternatives? Environmental Education Research, 6 (1), 59–91.

Rocco, T. S. (2010). Criteria for evaluating qualitative studies. Human Resource Development International . https://doi.org/10.1080/13678868.2010.501959

Sandberg, J. (2000). Understanding human competence at work: An interpretative approach. Academy of Management Journal, 43 (1), 9–25.

Schwandt, T. A. (1996). Farewell to criteriology. Qualitative Inquiry, 2 (1), 58–72.

Seale, C. (1999). Quality in qualitative research. Qualitative Inquiry, 5 (4), 465–478.

Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22 (2), 63–75.

Sparkes, A. C. (2001). Myth 94: Qualitative health researchers will agree about validity. Qualitative Health Research, 11 (4), 538–552.

Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2004). Quality in qualitative evaluation: A framework for assessing research evidence.

Stenfors, T., Kajamaa, A., & Bennett, D. (2020). How to assess the quality of qualitative research. The Clinical Teacher, 17 (6), 596–599.

Taylor, E. W., Beck, J., & Ainsworth, E. (2001). Publishing qualitative adult education research: A peer review perspective. Studies in the Education of Adults, 33 (2), 163–179.

Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19 (6), 349–357.

Tracy, S. J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16 (10), 837–851.

Download references

Open access funding provided by TU Wien (TUW).

Author information

Authors and affiliations.

Faculty of Informatics, Technische Universität Wien, 1040, Vienna, Austria

Drishti Yadav

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Drishti Yadav .

Ethics declarations

Conflict of interest.

The author declares no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Yadav, D. Criteria for Good Qualitative Research: A Comprehensive Review. Asia-Pacific Edu Res 31 , 679–689 (2022). https://doi.org/10.1007/s40299-021-00619-0

Download citation

Accepted : 28 August 2021

Published : 18 September 2021

Issue Date : December 2022

DOI : https://doi.org/10.1007/s40299-021-00619-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative research
• Evaluative criteria
• Find a journal
• Publish with us
• Track your research

• Search Menu

Sign in through your institution

• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Numismatics
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Social History
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Media
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Legal System - Costs and Funding
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Restitution
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business Ethics
• Business History
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Social Issues in Business and Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic History
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Management of Land and Natural Resources (Social Science)
• Natural Disasters (Environment)
• Pollution and Threats to the Environment (Social Science)
• Social Impact of Environmental Issues (Social Science)
• Sustainability
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Politics and Law
• Politics of Development
• Public Administration
• Public Policy
• Qualitative Political Methodology
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Disability Studies
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

The Oxford Handbook of Qualitative Research (2nd edn)

Patricia Leavy Independent Scholar Kennebunk, ME, USA

• Cite Icon Cite
• Permissions Icon Permissions

The Oxford Handbook of Qualitative Research, second edition, presents a comprehensive retrospective and prospective review of the field of qualitative research. Original, accessible chapters written by interdisciplinary leaders in the field make this a critical reference work. Filled with robust examples from real-world research; ample discussion of the historical, theoretical, and methodological foundations of the field; and coverage of key issues including data collection, interpretation, representation, assessment, and teaching, this handbook aims to be a valuable text for students, professors, and researchers. This newly revised and expanded edition features up-to-date examples and topics, including seven new chapters on duoethnography, team research, writing ethnographically, creative approaches to writing, writing for performance, writing for the public, and teaching qualitative research.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jul 8, 2024 11:22 AM
• URL: https://libguides.northwestern.edu/literaturereviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is Qualitative Research? | Methods & Examples

What Is Qualitative Research? | Methods & Examples

Published on June 19, 2020 by Pritha Bhandari . Revised on June 22, 2023.

Qualitative research involves collecting and analyzing non-numerical data (e.g., text, video, or audio) to understand concepts, opinions, or experiences. It can be used to gather in-depth insights into a problem or generate new ideas for research.

Qualitative research is the opposite of quantitative research , which involves collecting and analyzing numerical data for statistical analysis.

Qualitative research is commonly used in the humanities and social sciences, in subjects such as anthropology, sociology, education, health sciences, history, etc.

• How does social media shape body image in teenagers?
• How do children and adults interpret healthy eating in the UK?
• What factors influence employee retention in a large organization?
• How is anxiety experienced around the world?
• How can teachers integrate social issues into science curriculums?

Table of contents

Approaches to qualitative research, qualitative research methods, qualitative data analysis, advantages of qualitative research, disadvantages of qualitative research, other interesting articles, frequently asked questions about qualitative research.

Qualitative research is used to understand how people experience the world. While there are many approaches to qualitative research, they tend to be flexible and focus on retaining rich meaning when interpreting data.

Common approaches include grounded theory, ethnography , action research , phenomenological research, and narrative research. They share some similarities, but emphasize different aims and perspectives.

Note that qualitative research is at risk for certain research biases including the Hawthorne effect , observer bias , recall bias , and social desirability bias . While not always totally avoidable, awareness of potential biases as you collect and analyze your data can prevent them from impacting your work too much.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Each of the research approaches involve using one or more data collection methods . These are some of the most common qualitative methods:

• Observations: recording what you have seen, heard, or encountered in detailed field notes.
• Interviews:  personally asking people questions in one-on-one conversations.
• Focus groups: asking questions and generating discussion among a group of people.
• Surveys : distributing questionnaires with open-ended questions.
• Secondary research: collecting existing data in the form of texts, images, audio or video recordings, etc.
• You take field notes with observations and reflect on your own experiences of the company culture.
• You distribute open-ended surveys to employees across all the company’s offices by email to find out if the culture varies across locations.
• You conduct in-depth interviews with employees in your office to learn about their experiences and perspectives in greater detail.

Qualitative researchers often consider themselves “instruments” in research because all observations, interpretations and analyses are filtered through their own personal lens.

For this reason, when writing up your methodology for qualitative research, it’s important to reflect on your approach and to thoroughly explain the choices you made in collecting and analyzing the data.

Qualitative data can take the form of texts, photos, videos and audio. For example, you might be working with interview transcripts, survey responses, fieldnotes, or recordings from natural settings.

Most types of qualitative data analysis share the same five steps:

• Prepare and organize your data. This may mean transcribing interviews or typing up fieldnotes.
• Review and explore your data. Examine the data for patterns or repeated ideas that emerge.
• Develop a data coding system. Based on your initial ideas, establish a set of codes that you can apply to categorize your data.
• Assign codes to the data. For example, in qualitative survey analysis, this may mean going through each participant’s responses and tagging them with codes in a spreadsheet. As you go through your data, you can create new codes to add to your system if necessary.
• Identify recurring themes. Link codes together into cohesive, overarching themes.

There are several specific approaches to analyzing qualitative data. Although these methods share similar processes, they emphasize different concepts.

Qualitative research often tries to preserve the voice and perspective of participants and can be adjusted as new research questions arise. Qualitative research is good for:

• Flexibility

The data collection and analysis process can be adapted as new ideas or patterns emerge. They are not rigidly decided beforehand.

• Natural settings

Data collection occurs in real-world contexts or in naturalistic ways.

• Meaningful insights

Detailed descriptions of people’s experiences, feelings and perceptions can be used in designing, testing or improving systems or products.

• Generation of new ideas

Open-ended responses mean that researchers can uncover novel problems or opportunities that they wouldn’t have thought of otherwise.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Researchers must consider practical and theoretical limitations in analyzing and interpreting their data. Qualitative research suffers from:

• Unreliability

The real-world setting often makes qualitative research unreliable because of uncontrolled factors that affect the data.

• Subjectivity

Due to the researcher’s primary role in analyzing and interpreting data, qualitative research cannot be replicated . The researcher decides what is important and what is irrelevant in data analysis, so interpretations of the same data can vary greatly.

• Limited generalizability

Small samples are often used to gather detailed data about specific contexts. Despite rigorous analysis procedures, it is difficult to draw generalizable conclusions because the data may be biased and unrepresentative of the wider population .

• Labor-intensive

Although software can be used to manage and record large amounts of text, data analysis often has to be checked or performed manually.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square goodness of fit test
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

There are five common approaches to qualitative research :

• Grounded theory involves collecting data in order to develop new theories.
• Ethnography involves immersing yourself in a group or organization to understand its culture.
• Narrative research involves interpreting stories to understand how people make sense of their experiences and perceptions.
• Phenomenological research involves investigating phenomena through people’s lived experiences.
• Action research links theory and practice in several cycles to drive innovative changes.

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

There are various approaches to qualitative data analysis , but they all share five steps in common:

• Prepare and organize your data.
• Review and explore your data.
• Develop a data coding system.
• Assign codes to the data.
• Identify recurring themes.

The specifics of each step depend on the focus of the analysis. Some common approaches include textual analysis , thematic analysis , and discourse analysis .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2023, June 22). What Is Qualitative Research? | Methods & Examples. Scribbr. Retrieved July 30, 2024, from https://www.scribbr.com/methodology/qualitative-research/

Is this article helpful?

Pritha Bhandari

Other students also liked, qualitative vs. quantitative research | differences, examples & methods, how to do thematic analysis | step-by-step guide & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

4.2 Definitions and Characteristics of Qualitative Research

Qualitative research aims to uncover the meaning and understanding of phenomena that cannot be broken down into measurable elements. It is based on naturalistic, interpretative and humanistic notions. 5 This research method seeks to discover, explore, identify or describe subjective human experiences using non-statistical methods and develops themes from the study participants’ stories. 5 Figure 4.1 depicts major features/ characteristics of qualitative research. It utilises exploratory open-ended questions and observations to search for patterns of meaning in collected data (e.g. observation, verbal/written narrative data, photographs, etc.) and uses inductive thinking (from specific observations to more general rules) to interpret meaning. 6 Participants’ voice is evident through quotations and description of the work. 6 The context/ setting of the study and the researcher’s reflexivity (i.e. “reflection on and awareness of their bias”, the effect of the researcher’s experience on the data and interpretations) are very important and described as part of data collection. 6 Analysis of collected data is complex, often involves inductive data analysis (exploration, contrasts, specific to general) and requires multiple coding and development of themes from participant stories. 6

Reflexivity- avoiding bias/Role of the qualitative researcher

Qualitative researchers generally begin their work with the recognition that their position (or worldview) has a significant impact on the overall research process. 7 Researcher worldview shapes the way the research is conducted, i.e., how the questions are formulated, methods are chosen, data are collected and analysed, and results are reported. Therefore, it is essential for qualitative researchers to acknowledge, articulate, reflect on and clarify their own underlying biases and assumptions before embarking on any research project. 7 Reflexivity helps to ensure that the researcher’s own experiences, values, and beliefs do not unintentionally bias the data collection, analysis, and interpretation. 7 It is the gold standard for establishing trustworthiness and has been established as one of the ways qualitative researchers should ensure rigour and quality in their work. 8 The following questions in Table 4.1 may help you begin the reflective process. 9

Table 4.1: Questions to aid the reflection process

Philosophical underpinnings to qualitative research

Qualitative research uses an inductive approach and stems from interpretivism or constructivism and assumes that realities are multiple, socially constructed, and holistic. 10 According to this philosophical viewpoint, humans build reality through their interactions with the world around them. 10 As a result, qualitative research aims to comprehend how individuals make sense of their experiences and build meaning in their lives. 10 Because reality is complex/nuanced and context-bound, participants constantly construct it depending on their understanding. Thus, the interactions between the researcher and the participants are considered necessary to offer a rich description of the concept and provide an in-depth understanding of the phenomenon under investigation. 11

An Introduction to Research Methods for Undergraduate Health Profession Students Copyright © 2023 by Faith Alele and Bunmi Malau-Aduli is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License , except where otherwise noted.

• UNC Libraries
• HSL Subject Research
• Qualitative Research Resources
• Finding Qualitative Studies

Qualitative Research Resources: Finding Qualitative Studies

Created by health science librarians.

• What is Qualitative Research?
• Qualitative Research Basics
• Special Topics
• Training Opportunities: UNC & Beyond
• Help at UNC
• Qualitative Software for Coding/Analysis
• Software for Audio, Video, Online Surveys

About this Page

Ready-built sets of search terms, database-specific search strategies, general qual search strategies, web resources.

• Assessing Qualitative Research
• Writing Up Your Research
• Integrating Qualitative Research into Systematic Reviews
• Publishing Qualitative Research
• Presenting Qualitative Research
• Qualitative & Libraries: a few gems
• Data Repositories

Why is this information important?

• Electronic databases for health science literature, such as PubMed or CINAHL, often do not index qualitative health studies very clearly.
• Authors also do not always identify their methods using the word "qualitative" in their titles or abstracts; in some cases they may use terminology for a specific qualitative method instead.
• Often, that means that it is hard to find qualitative studies in common health science databases like PubMed

On this page you'll find:

• articles that describe and evaluate search strategies for finding qualitative research
• articles that provide search strategies for specific databases
• web resources on search filters and finding qualitative articles in databases
• links to sets of search terms to use when searching for qualitative research
• Hedges: Evidence Based Health Informatics, McMaster University contains qualitative hedges for Medline, PsycInfo, and Embase
• ISSG Search Filters Resource: Qualitative Research Filters The InterTASC Information Specialists' Sub-Group Search Filter Resource is a collaborative venture to identify, assess and test search filters designed to retrieve research by study design or focus. The Search Filters Resource aims to provide easy access to published and unpublished search filters. It also provides information and guidance on how to critically appraise search filters, study design filters in progress and information on the development and use of search filters. Inclusion of a search filter is not an endorsement of its validity or a recommendation.
• PubMed Health Services Research Queries Using Research Methodology Filters

A Few Articles on Search Strategies for Specific Databases

Wilczynski NL, Marks S, Haynes RB.2007.  Search strategies for identifying qualitative studies in CINAHL.  Qualitative Health Research  17(5):705-10.

Walters LA, Wilczynski NL, Haynes RB; Hedges Team. 2006.  Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE.  Qualitative Health Research  16(1):162-8.

Wong SS, Wilczynski NL, Haynes RB, Hedges Team. 2004.  Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE.   Medinfo   11: 311-316.

McKibbon KA, Wilczynski NL, Haynes RB. 2006.  Developing optimal search strategies for retrieving qualitative studies in PsycINFO.  Evaluation and the Health Professions   29: 440-454.

CINAHL & PsycINFO :

Rosumeck S, Wagner M, Wallraf S, Euler U. A validation study revealed differences in design and performance of search filters for qualitative research in PsycINFO and CINAHL . J Clin Epidemiol. 2020 Dec;128:101-108. doi: 10.1016/j.jclinepi.2020.09.031. Epub 2020 Sep 26. PMID: 32987157.

MEDLINE, CINAHL, Social Science Citation Index (SSCI) :

DeJean D, Giacomini M, Simeonov D, Smith A. Finding Qualitative Research Evidence for Health Technology Assessment . Qual Health Res. 2016 Aug;26(10):1307-17. doi: 10.1177/1049732316644429. Epub 2016 Apr 26. PMID: 27117960.

MEDLINE, EMBASE, CINAHL, PsycINFO :

A Few Articles on General Search Strategies for Qualitative Literature

Booth, A. (2016). Searching for qualitative research for inclusion in systematic reviews: A structured methodological review . Systematic Reviews, 5 doi:http://dx.doi.org.libproxy.lib.unc.edu/10.1186/s13643-016-0249-x

Cook, A., D. Smith, and A. Booth. 2012. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qualitative Health Research 10: 1435-1443.

Evans, D. 2002.  Database searches for qualitative research .  Journal of the Medical Libraries Association , 90(3): 290-293.

Flemming K, Briggs M. 2007. Electronic searching to locate qualitative research: evaluation of three strategies. J Adv Nurs . 57(1):95-100

Gorecki CA, Brown JM, Briggs M, Nixon J. 2010. Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life . J Adv Nurs . 66(3):645-52

Grant MJ. 2004 How does your searching grow? A survey of search preferences and the use of optimal search strategies in the identification of qualitative research. Health Info Libr J . 21(1):21-32

Littleton, D, S Marsalis, D Z Bliss. 2004. Searching the literature by design . Western Journal of Nursing Research 26(8): 891-908.

Methley, A.M., S. Campbell, C. Chew-Graham, R. McNally, and S. Cheraghi-Sohi. 2014. PICO, PICOS, and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews . BMC Health Serv Res 14: 579.

Pearson, M., Moxham, T., & Ashton, K. 2011. Effectiveness of Search Strategies for Qualitative Research About Barriers and Facilitators of Program Delivery .  Evaluation & the Health Professions , 34(3), 297–308.  https://doi.org/10.1177/0163278710388029

Petticrew, Mark and Helen Roberts. 2008. Systematic Reviews in the Social Sciences: A Practical Guide. Chapter 4: How to Find the Studies: The Literature Search . Blackwell Publishing: Oxford, UK.

Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, et al. 2004. Finding qualitative research: an evaluation of search strategies . BMC Med Res Methodol 4:5

• Campbell Collaboration Information Retrieval Guide Campbell Collaboration is an organization that guides and publishes systematic reviews. This resource is their publication on searching strategies and finding articles; it is not specific to qualitative studies but offers useful hints.
• NYU Libraries: Locating Qualitative Research Includes a good starting list of qualitative keywords for a general keyword based search strategy that can be cut and pasted into any database plus database specific strategies for CINAHL, Medline (including PubMed), and PsycINFO (with APA Index Terms). Note that PsycINFO via OVID strategies would need to be translated for UNC's Ebsco version.
• University of Washington LibGuide: Finding Qualitative Research Articles This guide gives some basic general search strategies when looking for qualitative literature, as well as specific search strategies for specific databases (CINAHL, PubMed, PsycInfo), books, and grey literature.
• << Previous: Software for Audio, Video, Online Surveys
• Next: Assessing Qualitative Research >>
• Last Updated: Jul 28, 2024 4:11 PM
• URL: https://guides.lib.unc.edu/qual

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

A Guide to Writing a Qualitative Systematic Review Protocol to Enhance Evidence-Based Practice in Nursing and Health Care

Affiliations.

• 1 PhD candidate, School of Nursing and Midwifey, Monash University, and Clinical Nurse Specialist, Adult and Pediatric Intensive Care Unit, Monash Health, Melbourne, Victoria, Australia.
• 2 Lecturer, School of Nursing and Midwifery, Monash University, Melbourne, Victoria, Australia.
• 3 Senior Lecturer, School of Nursing and Midwifery, Monash University, Melbourne, Victoria, Australia.
• PMID: 26790142
• DOI: 10.1111/wvn.12134

Background: The qualitative systematic review is a rapidly developing area of nursing research. In order to present trustworthy, high-quality recommendations, such reviews should be based on a review protocol to minimize bias and enhance transparency and reproducibility. Although there are a number of resources available to guide researchers in developing a quantitative review protocol, very few resources exist for qualitative reviews.

Aims: To guide researchers through the process of developing a qualitative systematic review protocol, using an example review question.

Methodology: The key elements required in a systematic review protocol are discussed, with a focus on application to qualitative reviews: Development of a research question; formulation of key search terms and strategies; designing a multistage review process; critical appraisal of qualitative literature; development of data extraction techniques; and data synthesis. The paper highlights important considerations during the protocol development process, and uses a previously developed review question as a working example.

Implications for research: This paper will assist novice researchers in developing a qualitative systematic review protocol. By providing a worked example of a protocol, the paper encourages the development of review protocols, enhancing the trustworthiness and value of the completed qualitative systematic review findings.

Linking evidence to action: Qualitative systematic reviews should be based on well planned, peer reviewed protocols to enhance the trustworthiness of results and thus their usefulness in clinical practice. Protocols should outline, in detail, the processes which will be used to undertake the review, including key search terms, inclusion and exclusion criteria, and the methods used for critical appraisal, data extraction and data analysis to facilitate transparency of the review process. Additionally, journals should encourage and support the publication of review protocols, and should require reference to a protocol prior to publication of the review results.

Keywords: guidelines; meta synthesis; qualitative; systematic review protocol.

© 2016 Sigma Theta Tau International.

PubMed Disclaimer

Similar articles

• How has the impact of 'care pathway technologies' on service integration in stroke care been measured and what is the strength of the evidence to support their effectiveness in this respect? Allen D, Rixson L. Allen D, et al. Int J Evid Based Healthc. 2008 Mar;6(1):78-110. doi: 10.1111/j.1744-1609.2007.00098.x. Int J Evid Based Healthc. 2008. PMID: 21631815
• Procedures and methods of benefit assessments for medicines in Germany. Bekkering GE, Kleijnen J. Bekkering GE, et al. Eur J Health Econ. 2008 Nov;9 Suppl 1:5-29. doi: 10.1007/s10198-008-0122-5. Eur J Health Econ. 2008. PMID: 18987905
• [Procedures and methods of benefit assessments for medicines in Germany]. Bekkering GE, Kleijnen J. Bekkering GE, et al. Dtsch Med Wochenschr. 2008 Dec;133 Suppl 7:S225-46. doi: 10.1055/s-0028-1100954. Epub 2008 Nov 25. Dtsch Med Wochenschr. 2008. PMID: 19034813 German.
• Evidence-based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Manchikanti L. Manchikanti L. Pain Physician. 2008 Mar-Apr;11(2):161-86. Pain Physician. 2008. PMID: 18354710 Review.
• An example of the use of systematic reviews to answer an effectiveness question. Forbes DA. Forbes DA. West J Nurs Res. 2003 Mar;25(2):179-92. doi: 10.1177/0193945902250036. West J Nurs Res. 2003. PMID: 12666642 Review.
• Physical Activity Interventions in People with Diabetes: A Systematic Review of The Qualitative Evidence. Vilafranca-Cartagena M, Bonet-Augè A, Colillas-Malet E, Puiggrós-Binefa A, Tort-Nasarre G. Vilafranca-Cartagena M, et al. Healthcare (Basel). 2024 Jul 9;12(14):1373. doi: 10.3390/healthcare12141373. Healthcare (Basel). 2024. PMID: 39057516 Free PMC article. Review.
• Telemedicine in Advanced Kidney Disease and Kidney Transplant: A Qualitative Meta-Analysis of Studies of Patient Perspectives. Manko CD, Apple BJ, Chang AR, Romagnoli KM, Johannes BL. Manko CD, et al. Kidney Med. 2024 May 24;6(7):100849. doi: 10.1016/j.xkme.2024.100849. eCollection 2024 Jul. Kidney Med. 2024. PMID: 39040545 Free PMC article.
• Voices of Wisdom: Geriatric Interviews on Self-Management of Type 2 Diabetes in the United States-A Systematic Review and Metasynthesis. Lo DF, Gawash A, Shah KP, Emanuel J, Goodwin B, Shamilov DD, Kumar G, Jean N, White CP. Lo DF, et al. J Diabetes Res. 2024 Jul 13;2024:2673742. doi: 10.1155/2024/2673742. eCollection 2024. J Diabetes Res. 2024. PMID: 39035684 Free PMC article. Review.
• Factors affecting implementation of mindfulness in hospital settings: A qualitative meta-synthesis of healthcare professionals' experiences. Knudsen RK, Skovbjerg S, Pedersen EL, Nielsen CL, Storkholm MH, Timmermann C. Knudsen RK, et al. Int J Nurs Stud Adv. 2024 Mar 27;6:100192. doi: 10.1016/j.ijnsa.2024.100192. eCollection 2024 Jun. Int J Nurs Stud Adv. 2024. PMID: 38746813 Free PMC article. Review.
• Experiences of nurses and midwives in policy development in low- and middle-income countries: Qualitative systematic review. Etowa J, Vukic A, Aston M, Iduye D, Mckibbon S, George A, Nkwocha C, Thapa B, Abrha G, Dol J. Etowa J, et al. Int J Nurs Stud Adv. 2022 Dec 17;5:100116. doi: 10.1016/j.ijnsa.2022.100116. eCollection 2023 Dec. Int J Nurs Stud Adv. 2022. PMID: 38746557 Free PMC article. Review.
• Search in MeSH

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.

Other Literature Sources

• scite Smart Citations

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• Open access
• Published: 29 July 2024

Does randomised evidence alter clinical practise? The react qualitative study

• Samuel Lawday 1 , 2 , 3 , 4 ,
• Karen Mattick 2 &
• Rob Bethune 1 , 2  

BMC Health Services Research volume  24 , Article number:  862 ( 2024 ) Cite this article

Metrics details

In 2015, the results of the ‘Small bites versus large bites for closure of abdominal midline incisions (STITCH) Trial’ were published in The Lancet. This demonstrated the superiority of small bite laparotomy closure over mass closure for the reduction of incisional hernias; despite this most surgeons have not changed their practice. Previous research has shown the time taken for the implementation of evidenced based practise within medicine takes an average of 17 years. This study aims to understand the reasons why surgeons have and have not changed their practice with regards to closure of midline laparotomy.

Semi-structured interviews were completed with surgical consultants and registrars at a single institution in South West England. The interview topic guide was informed by a review of the published literature, which identified barriers to adopting evidence into surgical practice. Interview transcripts underwent thematic analysis with themes identified following discussions within the research team, exploring views on published data and clinical practise.

Nine interviews with general surgical and urological consultants as well as registrars in training were performed. Three themes were identified; ‘Trusting the Evidence & Critical Appraisal’, ‘Surgical Attitude to Risk’ and ‘Adopting Evidence in Practise’, that reflected barriers to the introduction of evidenced based practise to clinical work.

Identification of the themes highlights possible areas for intervention to decrease the adoption time for evidence, for example from randomised controlled trials. The continued updating of clinical practise allows clinicians to provide best evidenced based care for patients and improve their outcomes.

Peer Review reports

Surgical practise has mostly shifted from being based upon ‘comic opera’, surgical dogma and case series, to well conducted cohort studies and randomised controlled trials (RCTs) [ 1 ]. Over this time, surgeons have produced data to allow a more evidence-based approach to clinical practise. This evidence has highlighted alternative management strategies for disease and different surgical techniques, which can improve patient outcomes such as post-operative morbidity, mortality, oncological and quality of life.

There is an average 17 year delay for the implementation of evidenced based practise, delaying potential clinical benefits to patients [ 2 , 3 ]. This time from publication of RCTs to the widespread adoption is surprisingly long and estimates also show that only half of EBM is adopted into general clinical practice [ 4 , 5 ]. Issues with the adoption of EBM can be due to the research itself (e.g. inappropriate research questions, inappropriate methods, inaccessibility of a paper, biased findings or unusable reporting [ 6 ]) or due to other clinical factors (different patients population or variations in current clinical pathways).

In 2015, the results of the ‘Small bites versus large bites for closure of abdominal midline incisions (STITCH) Trial’ were published in The Lancet [ 7 ]. This was a multi-centre, double-blinded RCT investigating closure method following mid-line incisions for elective surgery. They compared small bite (SB) closure, the same technique used in a previous single centre RCT, against standard practice. Five hundred and forty five patients were included in the final analysis and the study demonstrated that, with SB closure, patients had a reduced incidence rate of incisional hernia at 1 year ( p  = 0.03). Since the publication of the STITCH trial, meta-analysis supported the finding of reduced incisional hernia rate with the use of SB closure [ 8 ]. The European Hernia Society (EHS) guidelines in 2015 and 2022 recommended SB technique for fascial closure [ 9 , 10 ]. A recent BJS editorial also supported the use of SB closure and highlighted its importance of closure as part of clinical care [ 11 ]. Despite this, few surgeons seem to have changed their clinical practise [ 7 , 12 , 13 ].

This study set out to understand the reasons why surgeons have and have not changed their practice with regards to closure of midline laparotomy; with the expectations that many of these findings may be transferable to other areas of surgical practice.

This study aimed to explore the perceptions of surgeons on the underlying reason behind the adoption or rejection of small bite closure into routine clinical practise.

Study design

This study explored clinician perceptions using qualitative semi-structured interviews. A scoping review of published work was completed to inform the interview topic guide. This identified knowledge of the evidence, belief in the evidence, resources and patient factors as possible barriers to implementation of evidenced based surgical practise. The topic guide was trialled and subsequently modified within the research group prior to use (Appendix 1). Closure technique was discussed for open surgery using midline laparotomies but also for midline extraction sites following laparoscopic surgery.

Participants and setting

The research setting was a hospital in the Southwest of England. The hospital was a large district general hospital with tertiary colorectal services and on-site urology.

Inclusion criteria of the study were:

Surgical consultant or registrars.

Surgical specialities including but not limited to general, vascular, gynaecological and urological surgeons.

Perform midline elective laparotomy closure as part of routine clinical duties.

Purposive sampling of surgeons was completed. Surgeons were contacted by e-mail following identification through the on-call rota with a participant information leaflet. Face to face interviews were conducted on site at a mutually convenient time following the acquisition of written consent.

Data collection

Interviews took place between the June and July 2019 in a private room and were audio-recorded. These underwent professional transcription, after which the audio recording was deleted. Basic demographic details were recorded at the start of the interview prior to discussion about the use of small bite closure and evidence use in clinical practise. Recognising that ‘thematic saturation’ is a contested concept, we undertook sufficient interviews to answer the research question in the single setting, and to start to explore transferability to other contexts, whilst recognising and accommodating the significant time pressures on the participants [ 14 ].

Data analysis

Transcripts underwent thematic analysis using NVIVO software [ 15 ]. An inductive approach was taken with no specific theory in mind; the scoping review had informed the design of the topic guide, however this was not taken into account during the analysis. Codes were produced from the analysis and grouped into themes; this underwent multiple iterative and sequential changes following further analysis of the original codes and discussions within the authorship group. The manuscript was written in accordance with Standards for reporting qualitative research (SRQR) reporting guidelines [ 16 ].

Team reflexivity

The interviewer was a junior surgical trainee and was well known to those who were interviewed having worked in the department for 2 years. This may have affected the information that surgeon participants were willing to give during the interview. The primary author prior to analysis undertook structured external qualitative training to ensure validity and quality of the analysis. The research team consisted of a surgical trainee, a surgical consultant with qualitative experience and an experienced qualitative researcher.

The surgical consultant had already changed their practice to SB closure before the STITCH trial was published and so the results fitted into to their prior beliefs. This attitude may have reflected in the attitude of the surgical team and may have affected how some surgeons who didn’t use SB closure viewed the research project.

Ethical approval

for this project was sought and obtained by the ethics committee of the NHS Health Research Authority (IRAS 255,295) & the University of Exeter (RG/CB/19/4/210). All participants gave their written consent.

Nine interviews were completed, with 11 different surgeons having been approached. One surgeon not interviewed did not meet inclusion criteria and the other a suitable time for the interview could not be made. Eight surgeons were colorectal surgeons and one was a Urologist. No upper gastro-intestinal surgeons at the site met the inclusion criteria. This represented greater than 50% of eligible general surgical consultants. Three surgeons were registrars who had completed a minimum of 6 years post graduate surgical training and one was a fellow who had completed their general surgical training abroad but was completing a year of higher level training. The remaining five were consultants with a variety of experience, with completion of surgical training varying from 2000 to 2016. One interviewee had an academic career but the other participants had no or minimal active academic involvement.

All participants completed the interview. Average duration of the interviews was 22:49 min (range 14:20 to 36.37). Demographics can be found in Table  1 . The codes from thematic analysis were reviewed at successive time-points. Following the 9th interview, the research team deemed further interviews within this cohort at a single site unlikely to identify any further themes to aid answering the research question. Further interviews were therefore stopped as theoretical sufficiency had been determined [ 14 ].

Three themes were identified following the thematic analysis of the interview transcripts. These were ‘Trusting the Evidence & Critical Appraisal’, ‘Surgical Attitude to Risk’ and ‘Adopting Evidence in Practise’ (Fig.  1 ). Quotes are reported with a unique identifier and surgical grade (C = consultant and R = registrar).

Themes The themes and grouped codes identified following thematic analysis

Theme 1: ‘Trusting the evidence & critical appraisal’

This theme covered different codes identified within the literature but these focused on the STITCH trial itself. Part of the issue of adoption into clinical practise was a lack of awareness of the data.

“I knew about the STITCH trial before but hadn’t really read it” (Interviewee 783 , R) .

Those who were aware of it had different perceptions of the trial data. Some surgeons thought this was a high quality trial and justified the use of SB closure as part of their clinical practise. However, others thought the wrong question had been asked and were therefore less willing to take on this new technique.

“What those original trials looked at was comparing a technique which I don’t use , with a technique which I don’t use.” (Interviewee 412 , C) . “the view amongst us is that it is not the small bite itself , but what makes a difference is the definition of the anterior sheath” (Interviewee 463 , R) .

The conflict within the literature and that consultants did not consistently use the same technique was a point of frustration for one registrar. They found this conflict difficult to then decide what should be done in their own practise.

“it’s really frustrating reading all the different papers and then speaking to all these different consultants who all know what they’re talking about .…… even though it’s completely at conflict with what someone else believes” (Interviewee 186 , R) .

Theme 2: attitude to risk

This theme reflected how different surgeons responded to the risk that is inherent in their clinical practise and therefore influenced the uptake of SB closure.

The varying characteristics of surgeons were reported to be a key influencer in the uptake of the new technique and this was recognised by the surgeons interviewed. There was also a recognition that surgeons who continued to modify their practise found this easier; surgeons who did not make regular changes to how they practised clinically found making adaptions much harder.

“I mean………. it comes down to personalities , as well …… There are definitely some people who are more likely to pioneer new things” Interviewee 543 , R) “I think I’m more cynical” (Interviewee 412 , C) . “I’m probably a bit of a luddite in terms of new things” (Interviewee 911 , C) . “I think once you feel that you are doing something the optimum way…. I think it’s more difficult to change” (Interviewee 704 , R) .

The role of authoritative bodies in managing the risk for the individual surgeons was also raised by those interviewed. Some surgeons felt that if a wider or expert group supported a new technique, they would be more likely to use it. This reflects the shift of risk for the individual surgeon to an authoritative body, if it supported their practise. It wasn’t just authoritative bodies that reflected a shift of risk. Surgical colleagues, both locally and nationally, utilising a new technique meant other surgeons left safer and more supported using such a technique:

“if there had been a diktat from the college or somewhere to say , thou most close laparotomies using a small bite closure , then you would have no choice” (Interviewee 911 , R) . “I say , for me , I’m probably being a bit of a chicken , if I was on my own , the only person doing a particular closure , let’s say , then that would worry me” (Interviewee 911 , C) . “in consideration with the department I’m working in. I wouldn’t do anything necessarily out on my own” (Interviewee 783 , R) .

The perceived risk associated with adopting a new technique was also linked to medico-legal issues by many of the interviewees:

“If you’re going to adopt a new technique you have to be able to justify it to yourself because if you can’t justify it to yourself , you certainly won’t be able to justify it to the patient or to the patient’s lawyer if something goes wrong. That’s key really” (Interviewee 412 , C) .

Theme 3: adopting evidence in clinical practice

The third and final theme was relevant to the barriers within clinical environment that meant surgeons were not able to alter their practise. Surgeons discussed the use of anecdotal evidence of the patients and results they saw on a day to day basis and how that had a substantial impact on the techniques they use.

“it is evidence-based as well as anecdotal evidence based on your own practice” (Interviewee 463 , R) . “it became more apparent that the way that seemed to be working best was to just take the anterior rectus sheath” (Interviewee 412 , C) . “we weigh our own personal experience against what we , I guess , know to be true , from all what we believe to be- whether or not we believe it to be true from the research” (Interviewee 543 , R) . “If I start doing small bite closure and then I get a wound dehiscence , I’ll be mortified.” (Interviewee 412 , C) .

There was also some concern that novel surgical techniques were not acceptable for patients. Surgical apprehension also centred on the new technique not being appropriate for all patients; the original trial had strict inclusion and exclusion criteria and therefore the evidence is not there for a large proportion of patients undergoing midline laparotomy.

“if you’ve maybe told a patient that you’ve tried something new and then they’ve had a problem with it. I think it feels very different” (Interviewee 543 , R) . “if you go to a patient and say , everybody does it this way , they’re automatically happier. I think selling evidence of patients if very difficult” (Interviewee 911 , C) .

There was also unease regarding the training surgeons undergo. In part, this was clinical. Surgeons felt they had not been taught about the new SB technique and therefore did not feel comfortable utilising it in clinical practise:

“I mean , I’m aware of the backdrop and the evidence-base to this project , but that doesn’t really come into my mind , because I don’t feel that anyone has , like , sat me down and taught me about it” (Interviewee 186 , R) . “So , I think that’s a technical aspect of it , but unless it’s specifically taught to you , or you specifically think about it” (Interviewee 543 , R) .

Surgeons also discussed that there was minimal education during surgical training pertaining to the translation of evidence into clinical practise. The use of journal clubs and exams mainly focused on critical appraisal of a single journal article and not the impact this has on clinical practise:

“In terms of how you then translate that to everyday practice , probably not an awful lot” (Interviewee 911 , C) . “translational stuff from research to clinical practice , I’ve not had specific training about” (Interviewee 186 , R) .

The local culture that surgeons worked in was also highlighted as a potential barrier to the implementation of the new technique:

“So , again , that kind of almost comes back to what you were saying earlier about culture , and if other people are changing , it makes it easier for you to change” (Interviewee 543 , R) . “I think it’s probably definitely a changing culture. I think some of the more senior consultants; it would take a lot to get them to change their practice. So , yes , I think things are changing” (Interviewee 704 , R) . “there were some dinosaurs there …… the old guards would not change their ways , regardless” (Interviewee 783 , R) .

This study set out to understand the reasons why surgeons have and have not changed their practice with regards to closure of midline laparotomy, as a model for the implementation of evidenced based practise into routine clinical practise. Semi-structured qualitative interviews underwent thematic analysis led to the identification of three barriers to the adoption of the new technique. ‘Trusting the Evidence and Critical Appraisal’ focused mainly on the individual interpretation of the results of STITCH trial. Issues with outcomes, length of follow up and the comparator meant take up of the new technique was varied. ‘Surgical Attitude to Risk’ included personality as being a significant determinant as to how early surgeons were willing to change their practise. The burden of complications and subsequent possible litigation following complications when a new technique had been used played a role. The final theme was ‘Adopting Evidence in Practise’. These were issues identified with local culture and the actual implementation within a hospital. The issues mentioned were the training individuals received, patient factors and the availability of specific surgical equipment.

Published evidence identifying barriers to the implementation of surgical evidenced based practise have been described elsewhere [ 17 , 18 , 19 , 20 ]. The scoping review completed as part of this project identified knowledge of the evidence, belief in the evidence, resources and patient factors as possible barriers to implementation of EBM. There is significant overlap with the outcomes of the qualitative analysis which further increases the validity of the work completed. ‘Surgical Attitude to Risk’ is a novel factor identified from the qualitative interviews. Individual surgeons sit on different parts on the adoption curve and so have different threshold for the adoption of new treatments in their clinical practise. There is no broad consensus on the evidence level required for the safest adoption of novel treatments into practise.

In terms of the implications for practice, this study demonstrates that the threshold for which new techniques are introduced appears to vary between surgeons. The STITCH was a multi-centre RCT, however it was the first of its kind. There were surgeons included in this study who changed their practise before the publication of this trial, as a result of the trial and there are those who still remain cynical of the technique and its results. One of the barriers to the uptake of novel techniques is that surgeons will want more evidence than a single study to change their practise. This requires time and significant investment. The role of authoritative bodes was discussed as a possible way to mitigate risk to an individual surgeon, despite EHS guidelines being published years before these interviews took place. Other bodies such as NICE (National Institue for Clinical Excellence) in the UK and ASERNIP-S (Australian Safety and Efficacy Register of New Interventional Procedures) complete reviews and highlight practise that is evidenced based, this is often produced after significant research has been produced and is targeted at the level of operations rather than intra-operative techniques. These different guidelines are mainly produced by surgeons with a significant academic interest and therefore this may mean their perceptions of novel techniques and research data may differ from other surgeons.

The introduction of new techniques is a controversial topic within surgery and the optimum method and timing of implementation of within clinical practise remains unclear. Improving the implementation of high-quality evidenced based work is of upmost importance if we are to reduce research waste and improve outcomes for patients. The IDEAL collaboration is working on the safe introduction of innovation within surgery has led to the creation of a research framework for surgical innovation, however this does not remedy the issue of introduction into routine clinical practise [ 21 ]. Rapid dissemination of data prior to publication through online webinars is currently being utilised by surgical research collaboratives and the increasing use of open access journals should also increasingly allow surgeons access to high quality data. There has also been a shift from funding bodies, with a requirement for an implementation plans as part of a submission. The impact of these on the implementation within clinical practice however remains unclear.

This study inevitably had strengths and limitations. This was a well-planned and conducted study with targeted recruitment of surgeons, reflecting over 50% of eligible general surgeons at the research site. There were differences of surgeons interviewed, ensuring that conflicting opinions were identified. There was variation in time of practise, with consultants having been appointed between 2000 and 2016 and surgeons still in training included. There was variation in involvement in academia, reflecting possible differences in literature awareness and accessibility, and belief in the data. The interviewer worked with the interviewees. This may have allowed surgeons to have more open discussions about their practise or alternatively the advocating for the technique from the surgical consultant may have impacted the ability for surgeons to be open. The clinical experience of the research team meant there was a greater understanding of clinical context of the evidence, and this may have led to greater detail within the interviews. A single site was used due to ethical, practical and financial constraints of the project. Surgeons coming from a single institution with the similar experiences, training and culture may have limited the themes identified. Surgical registrars move hospitals as regularly as every 6 months and so were able to provide experience of cultural variation between hospitals.

This study aimed to understand reasons surgeons have and had not changed their practise following the publication of the STITCH trial. This high quality qualitative study identified ‘Trusting the Evidence & Critical Appraisal’, ‘Surgical Attitude to Risk’ and ‘Adopting Evidence in Practise’ as reasons for non-adoption of small bite closure for elective midline laparotomies. These have relevance to other areas of clinical practise and so are areas for potential intervention to increase the uptake of evidenced based surgical practise. Future randomised controlled studies need to include implementation and dissemination plans for the adoption of treatment into routine clinical practise at the point of applying for funding. We know simply publishing a paper will not change practice.

Data availability

No datasets were generated or analysed during the current study.

Bagenal J, Lee N, Ademuyiwa AO, Nepogodiev D, Ramos-De la Medina A, Biccard B, et al. Surgical research—comic opera no more. The Lancet. 2023;402(10396):86 − 8.

Medlinskiene K, Tomlinson J, Marques I, Richardson S, Stirling K, Petty D. Barriers and facilitators to the uptake of new medicines into clinical practice: a systematic review. BMC Health Serv Res. 2021;21(1):1198.

Article   PubMed   PubMed Central   Google Scholar  

Morris ZS, Wooding S, Grant J. The answer is 17 years, what is the question: understanding time lags in translational research. J R Soc Med. 2011;104(12):510–20.

Balas EA, Boren SA. Managing clinical knowledge for Health Care Improvement. Yearb Med Inform. 2000;1:65–70.

Grant J, Green L, Mason B. Basic research and health: a reassessment of the scientific basis for the support of biomedical science. Res Evaluation. 2003;12(3):217–24.

Article   Google Scholar  

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet (London England). 2009;374(9683):86–9.

Article   PubMed   Google Scholar  

Deerenberg EB, Harlaar JJ, Steyerberg EW, Lont HE, van Doorn HC, Heisterkamp J, et al. Small bites versus large bites for closure of abdominal midline incisions (STITCH): a double-blind, multicentre, randomised controlled trial. Lancet. 2015;386(10000):1254–60.

Henriksen NA, Deerenberg EB, Venclauskas L, Fortelny RH, Miserez M, Muysoms FE. Meta-analysis on materials and techniques for Laparotomy Closure: the MATCH Review. World J Surg. 2018;42(6):1666–78.

Article   CAS   PubMed   Google Scholar  

Deerenberg EB, Henriksen NA, Antoniou GA, Antoniou SA, Bramer WM, Fischer JP, et al. Updated guideline for closure of abdominal wall incisions from the European and American Hernia societies. Br J Surg. 2022;109(12):1239–50.

Muysoms FE, Antoniou SA, Bury K, Campanelli G, Conze J, Cuccurullo D, et al. European Hernia Society guidelines on the closure of abdominal wall incisions. Hernia. 2015;19(1):1–24.

de Beaux AC. Abdominal wall closure. BJS (British J Surgery). 2019;106(3):163–4.

Millbourn D, Cengiz Y, Israelsson LA. Effect of stitch length on wound complications after closure of midline incisions: a randomized controlled trial. Archives Surg (Chicago Ill: 1960). 2009;144(11):1056–9.

Holubar S, Midline Lapaorotomy Closure Twitter Poll. 2019 [ https://twitter.com/HolubarStefan/status/1042121774861824000 .

Varpio L, Ajjawi R, Monrouxe LV, O’Brien BC, Rees CE. Shedding the cobra effect: problematising thematic emergence, triangulation, saturation and member checking. Med Educ. 2017;51(1):40–50.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3(2):77–101.

O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Medicine: J Association Am Med Colleges. 2014;89(9):1245–51.

Brooke BS, Finlayson SR. What surgeons can learn from the emerging science of implementation. JAMA Surg. 2015;150(10):1006–7.

Hull L, Athanasiou T, Russ S. Implementation science: a neglected opportunity to accelerate improvements in the Safety and Quality of Surgical Care. Ann Surg. 2017;265(6):1104–12.

Kitto S, Petrovic A, Gruen RL, Smith JA. Evidence-based medicine training and implementation in surgery: the role of surgical cultures. J Eval Clin Pract. 2011;17(4):819–26.

Stienen MN, Scholtes F, Samuel R, Weil A, Weyerbrock A, Surbeck W. Different but similar: personality traits of​ surgeons and internists—results of a cross-sectional observational study. BMJ Open. 2018;8(7):e021310.

McCulloch P, Cook JA, Altman DG, Heneghan C, Diener MK. IDEAL framework for surgical innovation 1: the idea and development stages. BMJ: Br Med J. 2013;346:f3012.

Download references

Acknowledgements

No conflict of interest. Samuel Lawday (Doctoral Research Fellow, NIHR303276) is funded by NIHR. The views expressed in this publication are those of the author and not necessarily those of the NIHR, NHS or the UK Department of Health and Social Care. The Royal Devon and Exeter NHS Foundation Trust provided a small research grant to fund transcription but were not involved in the data collection or analysis. Poster presentation at ACPGBI 2022.

The Royal Devon and Exeter NHS Foundation Trust provided a small research grant to fund transcription but were not involved in the data collection or analysis. Samuel Lawday (Doctoral Research Fellow, NIHR303276) is funded by NIHR. The views expressed in this publication are those of the author and not necessarily those of the NIHR, NHS or the UK Department of Health and Social Care.

Author information

Authors and affiliations.

HeSRU, Royal Devon & Exeter Hospital, Exeter, UK

Samuel Lawday & Rob Bethune

Department of Health and Community Sciences, Faculty of Health and Life Sciences, University of Exeter, Exeter, UK

Samuel Lawday, Karen Mattick & Rob Bethune

Centre for Surgical Research, Population Health Science, Bristol Medical School, University of Bristol, Bristol, UK

Samuel Lawday

North Bristol NHS Trust, Bristol, UK

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to conception and design. SL completed data collection and analysis, with input into analysis from RB and KM. SL wrote the manuscript with substantial revisions from RB and KM.

Corresponding author

Correspondence to Rob Bethune .

Ethics declarations

Ethics approval and consent to participate.

Ethical approval for this project was sought and obtained by the ethics committee of the NHS Health Research Authority (IRAS 255295) & the University of Exeter (RG/CB/19/4/210). All participants gave their written informed consent to participate in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Lawday, S., Mattick, K. & Bethune, R. Does randomised evidence alter clinical practise? The react qualitative study. BMC Health Serv Res 24 , 862 (2024). https://doi.org/10.1186/s12913-024-11305-4

Download citation

Received : 01 May 2024

Accepted : 10 July 2024

Published : 29 July 2024

DOI : https://doi.org/10.1186/s12913-024-11305-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 22 July 2024

The breastfeeding experience of women with multiple pregnancies: a meta-synthesis of qualitative studies

• Ruxue Bai 1 ,
• Yifan Cheng 1 ,
• Siyu Shan 1 ,
• Xinmiao Zhao 1 ,
• Jun Wei 1 &
• Chunling Xia 1  

BMC Pregnancy and Childbirth volume  24 , Article number:  492 ( 2024 ) Cite this article

200 Accesses

1 Altmetric

Metrics details

The experiences and challenges associated with breastfeeding multiple births can be considerably more complex than those of singletons. Multiple births refer to the delivery of more than one offspring in a single birth event. Emphasizing the needs and experiences of mothers with multiple births during breastfeeding can enable healthcare providers to design targeted interventions that enhance breastfeeding rates. However, existing breastfeeding and health education resources and practices do not fully meet the needs of women who breastfeed multiples. This review aimed to review and synthesize qualitative studies on the breastfeeding experiences of women with multiple births.

A systematic search was conducted in 10 electronic databases for papers published from the inception of the database to March 2024. The Joanna Briggs Institute Critical Appraisal Checklist for Qualitative Research was utilized to evaluate the methodological quality of the studies included. The thematic synthesis method of Thomas and Harden was employed to integrate and analyze the included literature to derive new categories and conclusions.

Eight studies met the inclusion criteria and quality assessment criteria for this study. Through the integration of their results, four themes were identified: the choice and willingness to breastfeed multiple births; the challenges of breastfeeding multiple births; stage management and individualised adaptation of breastfeeding; and the experience of support.

Throughout the feeding process from pregnancy to the postpartum period, mothers with multiple births often have predominantly negative experiences with breastfeeding. Consequently, hospitals should create a multidisciplinary follow-up team comprising obstetrics, neonatology, psychology, and community services to offer specialized and personalized support to these women at various stages.

Systematic review registration

[ https://www.crd.york.ac.uk/PROSPERO/ ], identifier [PROSPERO 2024 CRD42024520348].

Peer Review reports

Introduction

Over the past three decades, the global incidence of multiple pregnancies has significantly increased due to delayed childbearing, advancements in assisted reproductive technologies, and the use of ovulation-inducing drugs [ 1 ]. In Italy, the rate of twin pregnancy increased from 1.28% in 2015 to 2.48% in 2020 [ 2 ], and in the United States, it surged by 70% from 1980 to 2019 [ 3 , 4 ]. Similarly, in China, the incidence of multiple pregnancies grew by 60% between 2009 and 2019 [ 1 ]. Multiple pregnancies are categorized as high risk, with 60% of twins being born prematurely before 37 weeks and 75% of triplet pregnancies born before 35 weeks [ 5 ], significantly increasing the risk of adverse neonatal outcomes.

Exclusive breastfeeding is deemed the optimal feeding method for newborns, particularly beneficial for the development of premature infants [ 6 ]. The American Academy of Pediatrics strongly recommends exclusive breastfeeding for the first six months, highlighting its effectiveness in reducing the risk of various health issues in premature infants, including sepsis, necrotizing enterocolitis, severe retinopathy of prematurity, metabolic syndrome, long-term growth retardation, and neurodevelopmental disorders [ 7 ]. However, international studies from countries such as Japan [ 8 , 9 ], Italy [ 2 ], Indonesia [ 10 ], South Korea [ 11 ], Sweden [ 12 ], China [ 13 ], and those with high breastfeeding rates such as Ghana [ 14 , 15 ] revealed that exclusive breastfeeding rates among women with multiple births consistently lag behind those of women with single births at all measured intervals. Specifically, exclusive breastfeeding rates at six months were less than 17%, significantly below the World Health Organization’s target of 50% [ 16 ]. Therefore, enhancing the breastfeeding rates among women with multiple births is crucial.

Research indicates that the low breastfeeding rates among women with multiple births are attributable to factors related to mothers, newborns, and support systems. Breastfeeding self-efficacy is a critical determinant of exclusive breastfeeding rates in this group [ 10 , 17 ]. High self-efficacy signifies that mothers possess confidence and proficiency in breastfeeding multiple infants and are proactive in addressing challenges during the feeding process. The perception of insufficient milk supply is the predominant reason for cessation of breastfeeding among these women, with many doubting their ability to produce adequate milk for multiple infants [ 2 , 15 , 18 ]. In contrast, studies have shown that oxytocin levels in mothers of twins are double those in mothers of singletons, leading to doubled milk production. By six months postpartum, these mothers can produce between one to two kilograms of milk daily, sufficient for the needs of two infants [ 19 ]. The primary complication associated with multiple births is preterm delivery, which often results in the separation of mother and infant. This separation complicates the initiation of critical early breastfeeding practices, including immediate sucking, skin-to-skin contact, and early milk production. Furthermore, the persistent immature feeding behaviors of preterm infants—characterized by weak sucking, lethargy, and poor tolerance—continue to challenge breastfeeding efforts post-discharge, contributing to a lower rate of exclusive breastfeeding among mothers of multiples [ 20 ]. Comprehensive medical [ 21 ], family [ 2 ], and social support [ 15 ] are essential to enhance breastfeeding rates among women with multiple births.

As the number of infants increases, the complexities and challenges of infant care, particularly breastfeeding, intensify with multiple births. It requires greater physical strength, belief in breastfeeding and support systems from the mother [ 2 ]. After encountering numerous challenges, mothers are more likely to cease breastfeeding [ 20 ]. A deep understanding of the breastfeeding experiences and needs of women with multiple births can enable healthcare professionals and breastfeeding specialists to identify issues early and offer personalized support services, thereby enhancing breastfeeding rates and duration [ 22 ]. There are already qualitative studies on the real experiences of breastfeeding among women with multiple births, but a single qualitative study cannot fully reflect all experiences of this group. There is still a lack of integration of relevant research results. This study adopts a meta-synthesis approach to consolidate qualitative research in this area, aiming to provide a more comprehensive and profound insight into the real experiences of breastfeeding among women with multiple births. This approach serves as the foundation for developing targeted interventions, relevant social support, and psychological care in the future.

This systematic review was registered prospectively with the International Prospective Register of Systematic Reviews (PROSPERO) with registration number CRD42024520348. The Enhancing Transparency in Reporting the Synthesis of Qualitative Research (ENTREQ) checklist (Supplementary Table 1) was used to report the process and results of synthesis and enhance transparency [ 23 ].

Search strategy

We conducted comprehensive searches across ten widely-used Chinese and English databases, including the China Biomedical Database (CBM), China National Knowledge Infrastructure (CNKI), Wanfang Database, VIP Database, PubMed, Web of Science, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL), the Joanna Briggs Institute (JBI) Evidence-Based Healthcare Center Database, and PsycINFO, to collate qualitative studies on breastfeeding among mothers with multiple births. The search covered the period from the inception of each database to March 2024. We employed both MeSH descriptors and free-text terms, and systematically reviewed the reference lists of the included studies to ensure a thorough literature search. The search terms included: “pregnancy” “twins” “multiple births” “higher order pregnanc*” “breastfeeding” “breastfeed*” “breast-feed*” “experience” “need*” “feeling” “qualitative research” “qualitative study” “ground theory” “interview*” “focus groups” “phenomenon” “ethnography”. The search strategy for PubMed is detailed in Table  1 .

Inclusion and exclusion criteria

We applied the PICOS framework, as recommended by the Joanna Briggs Institute (JBI) Evidence-Based Healthcare Center [ 24 ], to establish the inclusion and exclusion criteria for the studies. The inclusion criteria were as follows: (1) Population: women with multiple gestations, including twins, triplets, and higher-order multiples; (2) Interest of phenomena: the experiences, feelings, and needs of these women concerning breastfeeding multiple infants; (3) Context: the specific settings such as hospital wards, outpatient departments, or the homes of the pregnant women; (4) Study design: qualitative research, including various qualitative research literature using descriptive analysis, phenomenology, grounded theory, ethnography, the qualitative part of mixed research and so on. The exclusion criteria were as follows: (1) literature not published in Chinese or English; (2) literature for which the full text is not available; (3) literature that has been published multiple times. (4) the literature quality evaluation grade is C.

Literature selection and data extraction

All search records were imported into the reference management software NoteExpress, and duplicate records were removed by the principal investigator (BRX). Two researchers (BRX and SSY), who had undergone systematic training in evidence-based practice methods, independently conducted literature searches using the specified terms. They then rigorously applied the inclusion and exclusion criteria to screen the literature. This involved a meticulous review of the titles and abstracts for preliminary screening to discard irrelevant studies. Subsequently, we conducted a full-text review to finalize the selection of included studies. Any disagreements were resolved through discussion or by consulting a third researcher (CYF).

The data extraction process included the first author’s name, publication date, country, qualitative research method, research subject, phenomenon of interest, and key findings.

Quality appraisal

Two researchers (BRX and SSY) worked independently using the Joanna Briggs Critical Assessment tool for Methodological Quality Assessment [ 24 ], which is widely applicable to the evaluation of qualitative research. The evaluation content consisted of 10 items, each of which is evaluated as “yes”, “no”, “unclear”, or “not applicable”. The quality of the literature is divided into A, B, and C levels. If all the standards are fully met, the possibility of bias is the smallest, and the quality level is “A”. If some of the above standards are met, the possibility of bias is moderate, and the quality level is “B”. If none of the above standards are met, the possibility of bias is the greatest, and the quality level is “C”. In the event of any disagreements, a third researcher (CYF) will be responsible for making a joint decision.

Translation

During the data analysis process, two researchers translated the results and discussion sections of the Chinese literature into English, compared their translations, and negotiated the most appropriate version. After that, the two researchers then translated the English back into Chinese, compared the back-translated content with the original Chinese text, and discussed and resolved any disagreements. Based on the discussion results, the English translation was revised. Where cultural concepts could not be translated, native English speakers were consulted for translation advice, but this was not encountered during the translation of this paper. All researchers were staff in the field of obstetrics and were proficient in English.

Data synthesis

The data was synthesized using Thomas and Harden’s thematic synthesis methods [ 25 ]. A thematic synthesis was used to interpret multiple findings and develop analytical themes enhancing understanding [ 26 ]. the development of analytical themes that enhance understanding. Initially, two researchers (BRX and SSY) imported all relevant content from the “Results” and “Conclusions” sections of the included studies into NVivo11. They then coded the findings line by line based on their meaning and content. These codes were systematically analyzed and organized to identify similarities and differences, which were grouped into related areas to form descriptive themes. Analytical themes were subsequently developed inductively by synthesizing the study findings and elucidating their meanings. Discrepancies were deliberated upon by two researchers and, where necessary, a third researcher (ZXM) was consulted to resolve any contradictions.

Literature search

We first obtained a preliminary total of 2,330 studies through database search and reference literature review. Then, using NoteExpress software, 248 duplicate studies were excluded. By reviewing the titles and abstracts, we then preliminarily selected 24 articles for full-text analysis. Following this, based on the inclusion and exclusion criteria, we further narrowed down the selection to 8 articles for quality evaluation. The process and results of the literature screening are depicted in Fig.  1 .

Quality assessment

Table 2 summarizes the quality assessment results of the eight identified studies. The majority of these studies described the phenomenon of interest, the typical participants, and the methods of data collection and analysis. Only two studies explicitly described the cultural or theoretical background of the researchers. Four studies addressed the researchers’ influence on the research process, such as through quality control measures, but failed to discuss the impact of the research on the researchers themselves. Only two studies did not explicitly state ethical approval or list the ethical approval number. Ultimately, two studies were assigned a grade A, six received a grade B, and all eight studies were included in the analysis.

Flow diagram of the search strategy and results

Characteristics of included studies

Eight studies were included after article screening and quality assessment. The studies were conducted in 6 countries; Canada ( n  = 2), Turkey ( n  = 2), New Zealand, Iceland, Ghana, and China. The methodologies used in the studies were 3 phenomenological studies [ 22 , 27 , 28 ], 1 ethnography study [ 29 ], 2 mixed studies [ 15 , 30 ], and 2 qualitative studies with unspecified methodologies [ 31 , 32 ]. A total of 122 multiparous pregnant women were included in the studies and the main method of data collection was interviews, both semi-structured and narrative; one study used observational methods. Data collection was postpartum in seven studies, and only one was collected antenatally [ 29 ]. Table  3  provides a detailed description of the included studies.

The breastfeeding experiences of mothers of multiples

After repeated reading, analysis, and comparison by the researchers, a total of 67 quotes were extracted, similar quotes were classified and combined into 11 categories, and further integrated into 4 integrated results.

Synthesized finding 1: The drive for breastfeeding

Breastfeeding is widely regarded as the best way to feed a newborn, providing numerous benefits for both the physical and psychological development of the infant and the mother [ 33 , 34 ]. Breastfeeding is conventionally regarded as integral to ‘good mother‘ [ 29 ]. Several mothers of multiple births in the study noted that breastfeeding provided them with a sense of wellbeing and fulfillment, fostering intimate connection and enhancing their sense of maternal identity [ 27 ]. “I think breastfeeding provides such a bond. Breastfeeding reinforces bonding.” [ 27 ] Moreover, some mothers described breastfeeding as a ‘miracle’ and viewed it as a privileged experience [ 27 ]. “I believe that breastfeeding is a sacred thing. It is something really miraculous. As soon as the baby is born, s/he is looking for it, s/he finds it and sucks it.” [ 27 ] Consequently, these mothers often purchased breastfeeding equipment early on, illustrating their anticipation of breastfeeding [ 28 ]. “I bought a twin breastfeeding pillow and tried simultaneous breastfeeding but I couldn’t succeed.the babies were too small.” [ 28 ] The significance that multiple birth mothers attribute to breastfeeding helps sustain their determination to breastfeed despite challenges. “I think being a mother of twins is really a great privilege. It is also very difficult! This means we are very strong. what a blessing!” [ 27 ].

Synthesized finding 2: The challenges of breastfeeding multiple births

The second theme of the study examines the complex array of challenges encountered by multiparous women in the process of breastfeeding. These challenges encompass a range of physical and psychological obstacles, conflicts arising from balancing work responsibilities with breastfeeding commitments, and concerns related to the care of the newborn. These challenges affect the duration of breastfeeding and may even lead mothers to decide to stop breastfeeding.

Physical burden

The feeding duration for multiple births is significantly longer than that for single births. Prolonged feeding and caring for multiple births impose substantial physical strain, including symptoms such as lower back pain, insomnia, fatigue, and breast and nipple pain [ 22 , 27 , 28 , 30 ]. “I was in bed, and I had a catheter and could not walk or talk. I was sort of out of it for about two days after birth, and I could not see my children ” [ 30 ]“My nipples were cracked” [ 28 ]. In addition, mothers of multiple births with older children have to juggle older children and multiple newborns, making breastfeeding multiple births overwhelming. “After the cases where I broke in tears, I got up immediately and told myself that I have to be strong, I have to do it, I have to accomplish it. In the end, they’re my children.” [ 27 ].

Thus, for women with multiple births, the physical exhaustion following a caesarean section conflicts with the physical and energetic demands of breastfeeding multiples, leading to significant maternal stress. Additionally, the frequency of suckling sessions for multiple breastfeeding infants increases substantially compared to single births, elevating the risk of breast and nipple pain.

Psychological burden

Some interviewees also talked about their psychological pressure, such as overwhelm, anxiety, guilt, disappointment and depression [ 22 , 27 , 28 , 30 ]. Some mothers feel confused and overwhelmed about breastfeeding multiple babies after the birth of newborns or when hospitalized newborns return home because of the complexity of breastfeeding and the physical challenge of feeding multiple babies [ 30 ]. Even some women gave up breastfeeding: “Although I know breastfeeding is good, when I really face two children, I feel that my mind is empty, and I always feel afraid that they will be hungry, so I just feed milk powder. ” [ 22 ]. However, parturients with low milk supply regretted that they could not extablish exclusive breastfeeding [ 30 ]. However, mothers with experience in breastfeeding consider successful breastfeeding experience to be very important for subsequent breastfeeding of twins [ 32 ].

Therefore, some women find breastfeeding burdensome and want to return to their normal routine and family relationships. “Then, I thought, ‘This is starting to be a burden’ it cannot overshadow the time spent with the family and other things I want to do. ” [ 30 ].

There are also some mothers who consider their own mother’s identity, so they can remind themselves to be strong in the face of poor physical and psychological conditions [ 27 , 31 ]. “After the cases where I broke in tears, I got up immediately and told myself that I have to be strong, I have to do it, I have to accomplish it. In the end, they’re my children. ” [ 27 ].

In most studies, multiple birth mothers have negative emotional experiences of breastfeeding, with some women viewing breastfeeding as a “burden” and longing to return to normal life, while others choose to continue breastfeeding and view it as a “problem” to be overcome.

The burden of immature feeding patterns

The incidence of premature birth in multiple births is higher than that in single births, and newborns who can be discharged with the mother are mostly late preterm or early term infants, with weak sucking ability and prone to sleepiness. The mother believes that breastfeeding is time-consuming and difficult, requiring more patience. “You know they sucked a little and then just went to sleep; they were so tired.” [ 30 ]; “The oldest is less than 4 catties and will not breastfeed after discharge” [ 22 ]. As a result of the infants’ immature feeding pattern, breastfeeding becomes complicated, leading to a loss of confidence in breastfeeding among multiple birth mothers.

Multiple births are fed more frequently than single births, and unfortunately the immature feeding patterns of newborns make them more difficult to feed, and the challenge of “making it harder” adds to the negative feelings of multiple births.

The burden of work-breastfeeding conflict

Similar to single-child mothers, the mothers’ jobs are also important factors affecting the choice of multiple-child mothers to continue breastfeeding. Balancing work and breastfeeding is a challenging issue for mothers. “Because I have to go to work, I really can’t help but give up.” [ 22 ]. “My work is so involving and I need to make money hence my inability to let them suckle much.” [ 15 ].

Following a series of physical and psychological challenges, the end of maternity leave, and conflicts between work and breastfeeding can make work an ‘opportunity’ or ‘turning point’ for weaning.

Synthesized finding 3: Stage management and individualised adaptation of breastfeeding

The third theme was stage management and individualised adaptation of breastfeeding, and this study demonstrated that mothers with multiples have different coping strategies in the early and mature stages of the breastfeeding process.

Coping strategies in the early stages of feeding

In the early stages of breastfeeding, the management of breastfeeding in mothers of multiples is characterized by “prudent” feeding and efforts to increase lactation [ 30 ]. Mothers of multiples with separated babies or preterm infants also frequently use equipment to establish lactation or actively go to the hospital to deliver milk [ 22 , 30 ]. “Prudent feeding” involves strict adherence to standardized intervals between feeds, precise measurements of milk volume, monitoring weight gain and loss, tracking baby excretions, and scheduling nappy changes. “We rented a scale that we placed on this table and weighed them. We wrote down how much they weighed and, of course, calculated how much they needed in addition to breastfeeding. After four days, we felt this was insane.” [ 30 ]. Mothers of multiples may often manage their milk supply tightly in the early stages [ 27 , 29 , 30 ], using a breastpump to empty the breasts after breastfeeding and managing inadequate milk supply by changing breasts or dividing the milk equally [ 28 , 29 , 30 ]. “After every feeding for many weeks after birth, I used the breast pump to get more milk.” [ 30 ] “There was more milk on the right side.the fatty baby (3,000 g) was sucking that side; the other one was 2,650 g. I thought that was unjust.thus, I interchanged the breasts.” [ 28 ] Mother-infant separation or immature feeding patterns increase the likelihood of using breast pumps, hand milking, or nipple shields to stimulate milk production.

Multiple birth mothers face particular challenges in the early stages of feeding, and their commitment to breastfeeding their babies is further reflected in their thorough feeding programmes and use of equipment to maintain lactation.

Adaptation shifts during feeding maturation

After mastering the early and complex breastfeeding management and the maturation of infant feeding patterns, some mothers became skilled in breastfeeding techniques and developed individualized feeding routines based on each infant’s needs and feeding bond [ 30 ]. However, some mothers experienced fatigue with breastfeeding, gradually stopped pumping, and began accepting formula feeds after breastfeeding. This eventually led to the cessation of breastfeeding due to decreased milk production [ 30 ]. “During the second month, I quit pumping. I just had enough and found it easier to just breast- and bottle feed. Pumping interrupted my sleep at night and just everything. We had three weeks where it was going fine, but suddenly, there was no milk.” [ 30 ].

Most mothers can find an individual way to breastfeed during the breastfeeding maturation period, although some may experience a decrease in milk supply and eventually stop breastfeeding.

Synthesized finding 4: The experience of supports

Multiple birth mothers receive support for breastfeeding from family, friends, socia, medical, and information, but these supports don’t exactly promote breastfeeding for multiple births. The following section discusses women’s experiences of receiving this support.

Family support

As the number of babies increases, the time spent on breastfeeding and other infant care tasks also increases. Therefore, women with twins or triplets often express the need for more family support to help them complete breastfeeding successfully [ 27 , 30 , 32 ]. Spouses or family members assisting in preparing formula, taking care of other infants, and providing emotional support during feeding allow mothers to spend more time on extended breastfeeding and find it meaningful. However, as the husband returns to work, this support diminishes, and the demanding task of caring for multiple infants leaves the mother fatigued and unable to continue breastfeeding [ 30 ]. “We could not have done this without my parents. My mother sat by their [the twins] side while they slept so we could take a nap” [ 30 ]. Differences in childcare beliefs between mothers and their family members can create discord within the family unit [ 27 ]. “. It would be better if the helper were closer to my mind set. I got angry because my mother-in-law was very involved. I asked her to stay out of it.because I want to make my own decisions” [ 27 ].

Some mothers of multiples also seek support and help from friends and experienced “elders” in the local area [ 15 , 30 ]. “My friend who lives close by sometimes dropped by during lunchtime. She, of course, got coffee, but she also fed them [solids] and gave them a bottle [laughs]. It somehow turned into this: If someone comes, then he or she is handed the bottle.” [ 30 ].

Mothers of multiple births require direct or indirect breastfeeding support from their social networks, including family and friends, to ensure they have the time and energy to breastfeed.

Social support

A New Zealand study mentions that the government provides 240 h of non-means-tested family support services over 12 months for families with children under five and then twins [ 32 ]. And these family support services are domestic services such as cleaning, cooking and laundry [ 35 ]. Additionally, twin mothers can benefit from the establishment of relevant social service groups or peer support groups, which can offer emotional and other forms of breastfeeding support. Mothers of multiples believe that the situation of feeding multiple children is different from that of single-child mothers, so they are eager to make friends with other multiple-child mothers to obtain peer support [ 29 ]. “And I’ve had some of them say, ‘Look, I have a girlfriend who had twins, this is her phone number’.” [ 29 ].

Social support from government organizations, social service groups, and peer support groups is essential for helping mothers of multiple births maintain breastfeeding.

Medical support

Professional medical support as a part of social support is very important for the success and duration of breastfeeding for mothers of multiple children. All participants in the study believed that receiving prenatal training from medical professionals was helpful for achieving breastfeeding. Additionally, they required sufficient medical support following discharge [ 28 ]. Mothers who are separated from their babies often face difficulties and doubts in breastfeeding due to lack of experience with multiple infants and not receiving professional breastfeeding guidance. As a result, they require more continuous medical support [ 31 ]. “Two babies were born prematurely and transferred to pediatrics. After returning home, the baby did not eat (breast milk) at all, but the eldest learned it as soon as possible, so only one was breastfeeding.” [ 31 ]. In two studies, respondents had a contradictory psychological state of worrying about disturbing medical staff but also desiring continuous medical services [ 28 , 31 ]. “They just answered me as I questioned.they didn’t give information on their own.” [ 28 ]. However, the unprofessional feeding support and unencouraging feeding attitude of medical staff can also indirectly lead to multiple women not being able to achieve exclusive breastfeeding [ 27 ].

In terms of medical support for breastfeeding, multiple birth mothers expect professional antenatal education, continuity of medical support, proactive medical care and positive attitudes towards breastfeeding.

Information support

Most studies have mentioned the need for information support during various stages of breastfeeding, as well as the problems encountered in seeking information support [ 15 , 22 , 29 , 30 , 31 ]. This study will be presented from four perspectives: the source, characteristics, content, and dissemination form of information.

Regarding the source of information, compared to feeding guidance from community workers, mothers trust the feeding knowledge provided by hospital medical staff more. “After returning home, I had engorged breasts, so I hired a lactation consultant, but the effect was not good. When community workers visited, I was more willing to believe in your guidance.” [ 22 ]. However, some mothers did not receive information about breastfeeding multiple children, so they turned to the internet, friends or “elders” around them for help [ 30 ].

In terms of the characteristics of information, some mothers of multiple children believe that their feeding experiences are different from those of single-child mothers and that each mother and baby is unique. However, the feeding information provided in prenatal classes and by medical staff is not targeted at multiple births, and medical staff do not provide personalized guidance for each mother. “I said, ‘I know that probably everyone, everyone else was expecting one but’ I said, ‘can you cover twins a little bit for me? ’” [ 29 ].

In terms of the content of information, mothers of multiple children usually hope to get a detailed explanation of a range of topics, including the benefits of breastfeeding, the possibility of breastfeeding multiple children, how to breastfeed two infants at the same time, how to balance rest with breastfeeding, how family members can support mothers of multiple children in breastfeeding, and how to increase milk production [ 29 , 30 ]. “I’m worried about one getting up, and then you just nurse them and feed them and then you fall asleep and then the other one’s getting up, and then it’s like you’d be so tired.” [ 29 ].

One study mentioned that mothers of multiple children prefer to receive information through written promotional brochures as a medium. “Instructions should start during pregnancy. an illustrated booklet would be very good” [ 28 ].

Mothers of multiple births who sought information about breastfeeding rated their experiences with various information sources, personalisation, content, and media support. They expressed a preference for personalised information from health professionals regarding breastfeeding multiples. Additionally, they found an easily accessible illustrated booklet to be an effective medium for information support.

This study comprehensively describes the real experiences and diverse needs of women with multiple births concerning breastfeeding throughout the perinatal period and at all stages of the future implementation of feeding, including four aspects: willingness to breastfeed, multiple challenges in the implementation of breastfeeding multiple births, stage management and individualised adaptation of breastfeeding, and the evaluation of support received.

Breastfeeding intention reflects the attitudes and beliefs of multiparous mothers toward breastfeeding. A positive intention facilitates their active pursuit of breastfeeding information and the establishment of related behaviors, thereby enhancing both the duration and quality of breastfeeding. A study has shown that antenatal intention to breastfeed is a predictor of six-month breastfeeding rates in pregnancies with twins [ 14 ]. Our study revealed that while the majority of multiparous mothers demonstrated a strong inclination to breastfeed, a subset remained ambivalent. The findings suggest that professional prenatal education significantly boosts breastfeeding intentions [ 36 ]. Accordingly, medical staff should not only encourage a positive breastfeeding disposition but also respect individual preferences, providing tailored information to those without an initial intention to breastfeed [ 37 ]. The Queensland Organization Breastfeeding Guidelines 2023 [ 37 ] recommend that healthcare providers assess barriers to breastfeeding at delivery, offer detailed guidance on infant feeding practices at subsequent deliveries, and collaborate with mothers to formulate and document comprehensive breastfeeding plans. Additionally, multiparous mothers are consulted about their previous breastfeeding experiences; if negative, strategies to address these issues are discussed [ 37 ].

Women with multiple births, as evidenced in five studies, frequently report fatigue and strain during breastfeeding [ 22 , 27 , 28 , 30 , 31 ]. Postpartum, woman’s recovery will be affected by physical and psychological factors, such as a weakened postpartum state, frequent breastfeeding and sleep deprivation. Common complaints among women with multiple births include lower back pain, fatigue, sleep deprivation, nipple soreness, anxiety, and increased incidences of crying, leading to adverse attitudes toward breastfeeding. Furthermore, research indicates that multiple births correlate with diminished postnatal mental health compared to single births [ 5 ], and negative emotional states can further decrease lactation and milk supply [ 38 ]. To mitigate these challenges, medical and mental health professionals should prioritize multiparous mothers for follow-up visits. These should focus on educating them about the stress-relieving benefits of oxytocin released during breastfeeding, acknowledging and addressing their physical and psychological hardships, and providing strategies to prevent and manage physical discomfort. This approach aims to bolster their confidence and promote successful breastfeeding [ 38 , 39 ].

Compared with full-term infants, infants from multiple pregnancies often experience developmental challenges such as inadequate sucking, lethargy, and poor feeding tolerance, which complicate breastfeeding [ 40 ]. These challenges can diminish the self-confidence of multiparous women and exacerbate anxiety and other negative emotions [ 22 ]. Furthermore, current antenatal breastfeeding promotion primarily addresses term and singleton births, lacking detailed guidance for multiple births. A meta-analysis demonstrated that breastfeeding interventions grounded in the Theory of Planned Behavior or the Theory of Breastfeeding Self-Efficacy significantly enhance breastfeeding rates and durations [ 41 ]. Medical professionals should therefore implement evidence based grounded in psychological theory programs, offering tailored breastfeeding guidance that addresses the specific behaviors of the newborn and the physiological condition of the mother. This guidance should include techniques for assessing effective suckling, measuring actual breastfeeding amounts, scheduling feedings, and incorporating breastfeeding enhancements such as breastmilk fortifiers. For complex cases, prompt referral to breastfeeding specialists or certified lactation consultants is crucial.

Research analyzing mothers with multiple births who breastfed beyond 12 months indicates that those with partner and family support experienced longer breastfeeding durations compared to their unsupported counterparts [ 42 ]. Thus, robust family and social support plays a critical role in promoting successful breastfeeding among women with multiple births. Emotional, parenting, and household support from partners or family members significantly enhance the ability of mothers with multiple births to allocate more time to breastfeeding. Such support also promotes adequate rest and relaxation, which facilitates oxytocin secretion and reduces the risk of insufficient milk production. The Breastfeeding Co-Parenting Theoretical Framework [ 43 ] underscores the importance of partner or co-parent involvement in breastfeeding success and provides guidance on effective support strategies. The framework comprises five components, providing support ranging from the emotional to the practical dimensions. In the emotional dimension, examples include co-parents’ willingness to participate in breastfeeding, offering encouragement and praise to mothers of multiples, and addressing challenges faced during childcare and feeding in a positive manner. In the practical dimension, support includes providing breastfeeding-related information, necessary feeding equipment, and ensuring a fair distribution of daily childcare and household chores. This equitable distribution allows mothers of multiples to have the time and energy needed for breastfeeding. Consequently, healthcare professionals should discuss parenting styles with mothers of multiples and their families during the antenatal and postnatal periods, actively coordinating breastfeeding and emotional support based on this framework. This approach highlights how critical co-parenting support is to the longevity and success of breastfeeding. Currently, healthcare workers lack adequate knowledge about breastfeeding multiples, revealing a significant gap between medical practice and optimal multiple breastfeeding support. This gap hinders healthcare workers from effectively delivering personalized breastfeeding assistance to mothers of multiples. Addressing this gap requires enhancing the training and education of medical personnel to improve their competence and professionalism in breastfeeding support.

The New Zealand Government’s non-means-tested domestic support for families with multiple births can help families with multiple births to focus more on breastfeeding and parenting, and increase breastfeeding rates [ 32 ]. Currently, non-profit organizations such as The Multiple Births Foundation, Twins Trust, and Twins & Multiple Births Association offer some free social services to these mothers. Healthcare professionals should provide specific information and counseling about these resources when assisting families with multiple, thereby enhancing their access to social support.

Excellent medical support for breastfeeding is multifaceted. Healthcare workers must possess a comprehensive understanding of multiple birth breastfeeding to effectively assist mothers in selecting and initiating appropriate infant feeding practices, as well as in addressing subsequent issues [ 44 ]. It is essential for healthcare professionals to exhibit commendable personal qualities, respect the feeding choices of women with multiples, affirm their decisions confidently, and offer guidance on suitable methods [ 37 ]. Moreover, clinical staff require advanced communication skills to engage with mothers using acceptable and effective methods [ 37 ]. Additionally, it is crucial for healthcare workers to develop and implement feasible, evidence-based breastfeeding intervention programs specifically tailored for multiple births [ 45 ]. Such programs should leverage distance learning [ 46 ], peer education [ 47 ], and visual aids like graphic booklets or videos [ 48 ] to enhance breastfeeding rates among mothers of multiples.

In the UK, health visitors offer support advice and guidance to families with children under five. However, families with multiple births often find that the generalized support from health visitors fails to address the complexities of breastfeeding multiples [ 49 ]. The Elizabeth Bryan Centre for Multiple Births advocates for providing families with health visitors who have specialized training in multiple births [ 50 ]. The health visitor service, which facilitates the transition from hospital to home, is crucial. In countries lacking such services, hospitals should establish multidisciplinary follow-up teams to ensure continuity of care for families with multiple and support breastfeeding mothers. Therefore, both health visitors and medical follow-up team members need to pursue systematic education on multiple breastfeeding, such as studying specialized guidelines [ 51 ], to offer scientifically sound, authoritative, and customized support for managing breastfeeding in multiple births and assisting mothers through all stages of breastfeeding.

Strengths and limitations

This study constitutes the first systematic review of qualitative research synthesis concerning breastfeeding among women with multiple births, offering significant clinical relevance. We meticulously recorded the outcomes of the original studies, and through the aggregation and analysis of codes, were able to effectively mitigate content bias. In addition, all members of this study had undergone systematic evidence-based training, which enabled them to professionally review and assess the quality of the literature. Moreover, during the process of the study, the research members kept in touch with each other to discuss and analyse the results, which effectively improved the credibility of the whole study. The literature included in this study involved countries with both high and low rates of breastfeeding, so the findings may have a wider application. We have included suggestions for improving care in Table  4 .

However, the study also has several limitations. The results of the meta-synthesis were influenced by the limited number of qualitative studies on the subject of breastfeeding in multiple births—specifically, eight studies—of which six were rated Grade B and only two were Grade A for high quality. In addition, this study did not include qualitative articles with the theme of real-life experiences of parenting in women with multiple births, which also affected the comprehensiveness of the results to a certain extent, considering that descriptions of breastfeeding might be mentioned in their studies.

This study employs a meta-synthesis approach to examine the experiences and needs of women breastfeeding twins or higher-order multiples, aiming to offer targeted recommendations. The findings suggest that these women generally report negative experiences and encounter numerous challenges. Consequently, it is advised that hospitals establish multidisciplinary follow-up teams involving obstetrics, neonatology, and community care for these women. These teams should develop interventions based on theoretical frameworks to support breastfeeding in multiple-birth scenarios. Furthermore, families with multiple births should be treated as educational units to ensure continuous, personalized care that includes promoting feeding information before delivery, providing guidance after delivery, and monitoring breastfeeding post-discharge.

Future relevant interventional trials and multicentre cohort studies on breastfeeding multiples are needed to assess the contribution of different interventions to multiple breastfeeding rates and to inform subsequent guideline development by specialist in breastfeeding for multiple births.

Data availability

No datasets were generated or analysed during the current study.

Tang W, Zou L. Trends and characteristics of multiple births in Baoan Shenzhen: a retrospective study over a decade. Front Public Health. 2022;10:1025867.

Article   PubMed   PubMed Central   Google Scholar  

Quitadamo PA, Comegna L, Palumbo G, Copetti M, Lurdo P, Zambianco F et al. Feeding Twins with Human Milk and Factors Associated with its duration: a qualitative and quantitative study in Southern Italy. Nutrients. 2021; 13(9).

Martin JA, Hamilton BE, Osterman M, Driscoll AK, Births. Final data for 2019. Natl Vital Stat Rep. 2021;70(2):1–51.

PubMed   Google Scholar  

Multifetal Gestations. Twin, Triplet, and higher-Order Multifetal pregnancies: ACOG Practice Bulletin, Number 231. Obstet Gynecol. 2021;137(6):e145–62.

Article   Google Scholar  

Fenwick N, Chorley A. Twin and higher-order pregnancy – patient Voice. In: Khalil L, Lopriore, editor. Twin and higher-order pregnancies. Cham: Springer International Publishing; 2021. pp. 425–35.

Chapter   Google Scholar  

Merter OS, Altay N. Effect of feeding fresh or frozen breast milk on the gut microbiota of premature infants: a prospective observational study. Biol Res Nurs. 2024;26(1):78–90.

Article   PubMed   CAS   Google Scholar  

Section OB, Eidelman AI, Schanler RJ, Johnston M, Landers S, Noble L, et al. Breastfeeding and the use of human milk. Pediatrics. 2012;129(3):e827–41.

Yokoyama Y, Wada S, Sugimoto M, Katayama M, Saito M, Sono J. Breastfeeding rates among singletons, twins and triplets in Japan: a population-based study. Twin Res Hum Genet. 2006;9(2):298–302.

Article   PubMed   Google Scholar  

Ooki S. The effect of an increase in the rate of multiple births on low-birth-weight and preterm deliveries during 1975–2008. J Epidemiol. 2010;20(6):480–8.

Anjarwati N, Waluyanti FT, Rachmawati IN. Exclusive breastfeeding for Twin babies and its influencing factors: a study in East Java, Indonesia. Compr Child Adolesc Nurs. 2019;42(sup1):261–6.

Shim JI, Kang SJ. Impact of Breastfeeding Knowledge, attitude, and barriers on breastfeeding practice among Twin mothers. Korean J Women Healt. 2017;23 2:89–98.

Ostlund A, Nordstrom M, Dykes F, Flacking R. Breastfeeding in preterm and term twins–maternal factors associated with early cessation: a population-based study. J Hum Lact. 2010;26(3):235–41.

Wang S, Li M, Xiang X, Guo X, Peng C, Wang D, et al. Analysis on the current situation of twin breastfeeding and its influencing factors. Med. 2023;102(38):e35161.

Odei JA. Factors Associated with Exclusive Breastfeeding of Ghanaian Twins. Ghana: University of Ghana; 2013.

Google Scholar  

Tahiru R, Agbozo F, Garti H, Abubakari A. Exclusive Breastfeeding and Associated Factors among Mothers with Twins in the Tamale Metropolis. Int J Pediatr. 2020; 2020:5605437.

World Health Organization. Breastfeeding. https://www.who.int/health-topics/breastfeeding#tab=tab_3

Ceylan S, Şahin S. Comparison of breastfeeding self-efficacy and breastfeeding success of obese and normal-weight mothers in the early period. Afr Health Sci. 2020;20:2022–31.

Cinar N, Kose D, Alvur M, Dogu O. Mothers’ attitudes toward Feeding Twin babies in the First Six months of life: a sample from Sakarya, Turkey. Iran J Pediatr. 2016;26(5):e5413.

Bartick M, Reinhold A. The burden of suboptimal breastfeeding in the United States: a pediatric cost analysis. Pediatrics. 2010;125(5):e1048–56.

Mikami F, Francisco R, Rodrigues A, Hernandez WR, Zugaib M, de Lourdes BM. Breastfeeding twins: factors related to Weaning. J Hum Lact. 2018;34(4):749–59.

Huaijie Y, Tao W, Xiaoyun L, Wenfei Z, Xiaoyu Y. [The current situation investigation and analysis about breastfeeding of twins]. Hebei Med. 2019;41(3):463–5.

Huaijie Y, Qiong D, Wenhua L, Aihua C, Lu H, Liqiong Z, et al. [A qualitative study of the breastfeeding experiences and needs of twin mothers]. Chin Nurs Manag. 2015;15(09):1051–4.

Tong A, Flemming K, Mcinnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12:181.

Lockwood C, Munn Z, Porritt K. Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. JBI Evid Implement. 2015; 13(3).

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45.

Foyston Z, Higgins L, Smith DM, Wittkowski A. Parents’ experiences of life after medicalised conception: a thematic meta-synthesis of the qualitative literature. Bmc Pregnancy Child. 2023;23(1):520.

Article   CAS   Google Scholar  

Kocabey Z, Karakoç A. Breastfeeding experience of mothers with multiple babies: a phenomenological study. Clin Exp Health Sci. 2022;12(1):18–25.

Cinar ND, Alvur TM, Kose D, Nemut T. Breastfeeding twins: a qualitative study. J Health Popul Nutr. 2013;31(4):504–9.

PubMed   PubMed Central   Google Scholar  

Mckenzie PJ. The seeking of baby-feeding information by Canadian women pregnant with twins. Midwifery. 2006;22(3):218–27.

Jonsdottir RB, Flacking R, Jonsdottir H. Breastfeeding initiation, duration, and experiences of mothers of late preterm twins: a mixed-methods study. Int Breastfeed J. 2022;17(1):68.

Leonard LG. Breastfeeding higher order multiples: enhancing support during the postpartum hospitalization period. J Hum Lact: Official J Int Lactation Consultant Association. 2002;18(4):386–92.

Mcgovern T. The challenges of breastfeeding twins. Kai Tiaki Nurs New Z. 2014;20(11):26–44.

Li R, Ware J, Chen A, Nelson JM, Kmet JM, Parks SE et al. Breastfeeding and post-perinatal infant deaths in the United States, a national prospective cohort analysis. Lancet Reg Health Am. 2022; 5.

Yuen M, Hall OJ, Masters GA, Nephew BC, Carr C, Leung K, et al. The effects of breastfeeding on maternal Mental Health: a systematic review. J Womens Health (Larchmt). 2022;31(6):787–807.

Government Assistance. Multiples NZ. https://multiples.org.nz/government-assistance/

Menekse D, Çinar N. The Effect of Breastfeeding Education Provided to pregnant women who expect twins on their breastfeeding intention. Online Türk Sağlık Bilimleri Dergisi. 2022;7:4.

Queensland Clinical Guidelines. Establishing breastfeeding. Queensland Health. 2021. http://www.health.qld.gov.au/qcg

Nagel EM, Howland MA, Pando C, Stang J, Mason SM, Fields DA, et al. Maternal psychological distress and lactation and breastfeeding outcomes: a narrative review. Clin Ther. 2022;44(2):215–27.

Henshaw EJ. Breastfeeding and Postpartum Depression: A Review of relationships and potential mechanisms. Curr Psychiatry Rep. 2023;25(12):803–8.

Russell S, Russell N. Breastfeeding Twins and multiples. In: Khalil L, Lopriore, editor. Twin and higher-order pregnancies. Cham: Springer International Publishing; 2021. pp. 355–62.

Chipojola R, Chiu HY, Huda MH, Lin YM, Kuo SY. Effectiveness of theory-based educational interventions on breastfeeding self-efficacy and exclusive breastfeeding: a systematic review and meta-analysis. Int J Nurs Stud. 2020;109:103675.

Monvillers S, Tchaconas A, Li R, Adesman A, Keim SA. Characteristics of and sources of support for women who Breastfed multiples for more than 12 months. Breastfeed Med. 2020;15(4):213–23.

Abbass-Dick J, Dennis CL. Breast-feeding Coparenting Framework: a New Framework to improve breast-feeding duration and exclusivity. Fam Community Health. 2017;40(1):28–31.

Wallenborn JT, Lu J, Perera RA, Wheeler DC, Masho SW. The impact of the Professional qualifications of the prenatal care provider on Breastfeeding Duration. Breastfeed Med. 2018;13(2):106–11.

Whitford HM, Wallis SK, Dowswell T, West HM, Renfrew MJ. Breastfeeding education and support for women with twins or higher order multiples. Cochrane Database Syst Rev. 2017;2(2):CD012003.

Moukarzel S, Caduff A, Rehm M, Del FM, Perez-Escamilla R, Daly AJ. Breastfeeding Communication Strategies, challenges and opportunities in the Twitter-Verse: perspectives of influencers and Social Network Analysis. Int J Environ Res Public Health. 2021; 18(12).

Rodriguez-Gallego I, Leon-Larios F, Corrales-Gutierrez I, Gonzalez-Sanz JD. Impact and Effectiveness of Group Strategies for Supporting Breastfeeding after birth: a systematic review. Int J Environ Res Public Health. 2021; 18(5).

Novianty N, Syarif S, Ahmad M. Influence of breast milk education media on increasing knowledge about breast milk: literature review. Gac Sanit. 2021;35(Suppl 2):S268–70.

Turville N, Alamad L, Denton J, Cook R, Harvey M. Supporting multiple birth families: perceptions and experiences of health visitors. Public Health Nurs. 2022;39(1):229–37.

Turville N, Alamad L, Denton J, Harvey M. Supporting multiple birth families; establishing an evidence base to inform Health Visitor Practice. Twin Res Hum Genet. 2021;24(6):392.

Foundation MB. Guidance for health professionals on feeding twins, triplets and higher order multiples. Multiple Births Foundation; 2015.

Download references

Acknowledgements

We would like to thank the Ministry of Science and Technology of the People’s Republic of China for the funding support provided to this research.

This study was supported by the scientific research project of the Ministry of Science and Technology of the People’s Republic of China (2023YFC2705903). The funder had no role in the study design, themes analysis and extraction process, results interpretation, or manuscript preparation.

Author information

Authors and affiliations.

Department of Obstetrics and Gynecology, Shengjing Hospital of China Medical University, Shenyang City, 110004, Liaoning Province, China

Ruxue Bai, Yifan Cheng, Siyu Shan, Xinmiao Zhao, Jun Wei & Chunling Xia

You can also search for this author in PubMed   Google Scholar

Contributions

BRX: Conceptualization, Resources, Methodology, Software, Investigation, Data curation, Formal analysis, Writing – original draft, Writing – review & editing; CYF: Methodology, Software, Investigation, Data curation, Formal analysis, Validation; SSY: Methodology, Software, Investigation, Data curation, Formal analysis, Writing – review & editing; ZXM: Software; Formal analysis, Writing – review & editing; WJ: Writing – review & editing, Funding acquisition; XCL: Supervision, Writing – review & editing.

Corresponding authors

Correspondence to Jun Wei or Chunling Xia .

Ethics declarations

Ethics approval and consent to participate.

Not applicable. This article does not require ethics committee approval as it is a systematic review of already published and accessible literature.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Bai, R., Cheng, Y., Shan, S. et al. The breastfeeding experience of women with multiple pregnancies: a meta-synthesis of qualitative studies. BMC Pregnancy Childbirth 24 , 492 (2024). https://doi.org/10.1186/s12884-024-06697-4

Download citation

Received : 27 April 2024

Accepted : 15 July 2024

Published : 22 July 2024

DOI : https://doi.org/10.1186/s12884-024-06697-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Breastfeeding
• Multiple birth
• Qualitative research
• Systematic review
• Meta-synthesis

BMC Pregnancy and Childbirth

ISSN: 1471-2393

• Submission enquiries: [email protected]
• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 30.7.2024 in Vol 26 (2024)

This is a member publication of University of Oxford (Jisc)

The Acceptability, Engagement, and Feasibility of Mental Health Apps for Marginalized and Underserved Young People: Systematic Review and Qualitative Study

Authors of this article:

• Holly Alice Bear 1 , DPhil   ; 
• Lara Ayala Nunes 1 , DPhil   ; 
• Giovanni Ramos 2 , PhD   ; 
• Tanya Manchanda 1 , MEd   ; 
• Blossom Fernandes 1 , PhD   ; 
• Sophia Chabursky 3 , MSc   ; 
• Sabine Walper 3 , DPhil   ; 
• Edward Watkins 4 , DPhil   ; 
• Mina Fazel 1 , DM  

1 Department of Psychiatry, University of Oxford, Oxford, United Kingdom

2 Department of Psychological Science, University of California, Irvine, CA, United States

3 German Youth Institute, Munich, Germany

4 School of Psychology, University of Exeter, Exeter, United Kingdom

Corresponding Author:

Holly Alice Bear, DPhil

Department of Psychiatry

University of Oxford

Warneford Hospital

Warneford Lane

Oxford, OX3 7JX

United Kingdom

Phone: 44 01865 6182

Email: [email protected]

Background: Smartphone apps may provide an opportunity to deliver mental health resources and interventions in a scalable and cost-effective manner. However, young people from marginalized and underserved groups face numerous and unique challenges to accessing, engaging with, and benefiting from these apps.

Objective: This study aims to better understand the acceptability (ie, perceived usefulness and satisfaction with an app) and feasibility (ie, the extent to which an app was successfully used) of mental health apps for underserved young people. A secondary aim was to establish whether adaptations can be made to increase the accessibility and inclusivity of apps for these groups.

Methods: We conducted 2 sequential studies, consisting of a systematic literature review of mental health apps for underserved populations followed by a qualitative study with underserved young male participants (n=20; age: mean 19). Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, an electronic search of 5 databases was conducted in 2021. The search yielded 18,687 results, of which 14 articles met the eligibility criteria.

Results: The included studies comprised a range of groups, including those affected by homelessness, having physical health conditions, living in low- and middle-income countries, and those with sexual and gender minority identities. Establishing and maintaining user engagement was a pervasive challenge across mental health apps and populations, and dropout was a reported problem among nearly all the included studies. Positive subjective reports of usability, satisfaction, and acceptability were insufficient to determine users’ objective engagement.

Conclusions: Despite the significant amount of funding directed to the development of mental health apps, juxtaposed with only limited empirical evidence to support their effectiveness, few apps have been deliberately developed or adapted to meet the heterogeneous needs of marginalized and underserved young people. Before mental health apps are scaled up, a greater understanding is needed of the types of services that more at-risk young people and those in limited-resource settings prefer (eg, standard vs digital) followed by more rigorous and consistent demonstrations of acceptability, effectiveness, and cost-effectiveness. Adopting an iterative participatory approach by involving young people in the development and evaluation process is an essential step in enhancing the adoption of any intervention, including apps, in “real-world” settings and will support future implementation and sustainability efforts to ensure that marginalized and underserved groups are reached.

Trial Registration: PROSPERO CRD42021254241; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=254241

Introduction

Addressing health inequities is a key challenge for the mental health field, especially when trying to ensure that interventions and services are accessible and acceptable for all populations. Nearly 50% of lifelong mental health disorders begin by the age of 14 years, and by the age of 24 years, 75% of mental health disorders have begun [ 1 ]. Given the frequent onset of mental health problems during youth, here defined as the period between 15 and 24 years, special attention must be paid to older adolescents, including those from underserved and marginalized minority groups and socioeconomically deprived backgrounds [ 2 ]. In these groups, common barriers to accessing mental health services can be exacerbated (eg, poor mental health literacy, lack of knowledge about where to seek help, negative attitudes toward professional help seeking, embarrassment, preference for self-reliance, fear of stigma, and confidentiality concerns), and additional barriers exist (eg, reliance on informal supports, shame, lack of housing or money, and therapist factors, such as different race and level of experience), creating increased risk of untreated mental health problems and thus poorer mental health outcomes [ 3 - 7 ]. In this study, marginalized and underserved populations are defined as those with higher prevalence of mental health problems and lower rates of help seeking, such as racially and ethnically minoritized individuals, rural and remote communities, financially deprived groups, individuals experiencing homelessness, refugee and migrant populations, and sexual and gender minority groups [ 8 - 10 ] and those with lower inclusion in mental health intervention research than one would expect from population estimates [ 11 ], respectively. These groups are exposed to risk factors for poor mental health and experience disparities in mental health care, including lower access to care, poorer treatment quality, and limited engagement in treatment [ 5 , 10 , 12 ].

Smartphone apps could offer an opportunity to deliver mental health and well-being resources and interventions in a scalable, cost-effective, and potentially personalized manner, particularly for those who experience the greatest barriers to accessing health care [ 13 , 14 ]. Given that smartphone ownership is nearly ubiquitous among young people in high-income nations and increasingly across lower-resource settings, apps have the potential to address some of the accessibility issues in service provision for young people’s mental health [ 15 ]. Young people are more digitally connected (ie, they are more likely to own smartphones and spend more time on the web) and more likely to seek health information on the web than older generations, meaning that app-based interventions may be particularly well-suited for this population [ 16 , 17 ].

Not surprisingly, the number of mental health apps being developed, both commercially and in academic research programs, has expanded rapidly, outpacing scientific evaluations of their effectiveness [ 18 , 19 ]. Emerging evidence suggests that some apps may produce significant symptom improvement across multiple outcomes, compared with waitlist or control conditions [ 20 - 22 ]. Despite promise, empirical research often fails to translate into meaningful and sustained implementation in “real-world” settings [ 23 , 24 ]. Research has focused primarily on efficacy under ideal “laboratory” conditions rather than effectiveness in real-world settings [ 25 ]. Therefore, at present, most apps, especially those available to the public, lack strong empirical support [ 19 ]. The acceptability (ie, perception that a given technology is useful, agreeable, palatable, or satisfactory); accessibility (ie, the technology being easy to obtain or use); engagement (ie, initial adoption and sustained interactions with the technology, including the level of app use, intervention adherence, and premature dropout); and feasibility (ie, the actual fit, utility, or suitability and the extent to which the technology can be successfully used or conducted within a given context) of apps for marginalized and underserved groups remain poorly understood [ 25 - 28 ]. Although mental health apps may provide a possible solution, marginalized and underserved groups of young people face unique challenges to engage with and benefit from these interventions (eg, intervention cost, content that is not culturally attuned and lack of reliable access to the internet) and are typically underrepresented in intervention research [ 25 , 27 ]. Although increased access is often seen as a major benefit of digital mental health interventions, issues related to the “digital divide” describe the phenomena that technology is not equally available to all social groups due to economic, social, or cultural inequalities and is a potential ethical concern [ 29 ]. Furthermore, underrepresentation in the intervention development process potentially reinforces structural inequalities by limiting the availability of products that are culturally accessible, inclusive, and effective or by skewing the product features to attract young people from more advantaged backgrounds [ 9 , 14 ]. Therefore, considering diversity, equity, and inclusion issues at the outset of health care research as well as within app evaluation is essential to prevent the perpetuation of existing inequities [ 27 ].

To date, little has been published on the attempts to create new or adapt existing app interventions to meet the heterogeneous needs of diverse groups of young people [ 9 , 14 ]. Moving forward, careful consideration is needed to ensure optimal leveraging of all mental health intervention research, including that of mental health apps, to increase health equity while also ensuring that innovations do not inadvertently widen the digital divide and exacerbate health inequalities [ 14 ]. Although the efficacy of many mental health apps remains unclear, future attempts to translate findings for underserved populations will need to ensure that all apps are developed with enough flexibility to fit a wider range of user needs and preferences. To achieve this goal, research is needed to assess the acceptability and feasibility of mental health apps for underserved young people to ensure that they are not further excluded from research and to advance toward mental health provision that meets their needs.

This Research

We conducted two sequential studies: (1) a systematic literature review and (2) a qualitative study with a targeted sample of young people who often are underrepresented in research, with limited access to health care and socioeconomic deprivation. The overarching aim of these combined studies was to better understand whether mental health apps are feasible and acceptable to underserved young people. A secondary aim was to determine which adaptations might enable accessibility of and effective engagement with mental health apps for these groups.

The research questions of interest were as follows: (1) On the basis of the existing literature, are mental health apps acceptable, feasible, and engaging for marginalized and underserved young people and how have these constructs been measured? (2) On the basis of the qualitative study, what are young people’s experiences of using a mental well-being app, including its acceptability, feasibility, and level of engagement? (3) On the basis of both studies, are apps an acceptable, feasible, and engaging intervention approach to meet the specific needs of underserved young people? What adaptations can be made to ensure that mental health apps are accessible and inclusive for these groups?

To fully address our research questions, we adopted a 2-pronged approach. First, we conducted a systematic review of the literature to better understand the acceptability and feasibility of mental health apps for underserved young people. To explore the findings of the systematic review in greater depth and to provide further insights from multiple perspectives, we next conducted a qualitative study with young men not in education, employment, or training (NEET) in the United Kingdom and Spain and asylum seekers and refugees in Germany. The interviews were conducted between August 2021 and February 2022.

Systematic Review

Literature search and search strategy.

An electronic literature search was performed in English on the following databases from January 2009 to May 2021: Cochrane Library, Embase, MEDLINE, and PsycINFO. We used key search terms relating to (1) underserved young people; (2) mental health mobile apps; and (3) acceptability, feasibility, and engagement. The search strategy was guided by similar reviews exploring digital mental health interventions for young people [ 25 , 30 ], and the terms for apps were derived from Cochrane reviews [ 31 , 32 ]. An updated search was conducted in September 2023. The full search strategies are available in Multimedia Appendix 1 .

Inclusion and Exclusion Criteria

Screened articles were included if (1) the study targeted marginalized and underserved young people with a mean age of 15 to 25 years, including individuals who were NEET, apprentices, teenage parents, members of minoritized racial or ethnic groups, members of sexual and gender minoritized groups, residents of low- and middle-income countries (LMICs), experiencing homelessness, socioeconomic deprivation, refugees or asylum seekers, and migrants; individuals with substance use disorders; those under state or statutory care; people with physical disabilities; and individuals involved in the criminal justice system or incarcerated; (2) the intervention was a “native” mobile app (ie, not on a web browser), whose primary aim was to promote well-being, prevent mental health problems, or treat existing mental health problems, delivered as a stand-alone intervention or as an adjunct to therapist-assisted interventions; (3) the primary outcome was a measure of mental health or well-being; and (4) the study reported a measure of user acceptability or feasibility.

Articles were excluded if (1) the mean age of participants was outside of the 15 to 25 years range; (2) the intervention was not a mobile app, that is, other digital interventions, including teletherapy (eg, therapy delivered by phone, SMS text messages, video platforms, or PCs); and (3) there was no measure of acceptability or feasibility. Gray literature was not included in the search.

Study Selection

In accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 33 ], the flowchart presented in Figure 1 provides step-by-step details of the study selection procedure. The PRISMA checklist is provided in Multimedia Appendix 1 . The search strategy identified 11,539 citations after deduplication. After an initial screening of the titles, which resulted in the exclusion of 10,061 (87.19%) irrelevant entries, the abstracts of 1478 (12.81%) studies were screened by 4 members of the review team (LAN, HAB, BF, and TM). The identified 176 (11.91%) full texts were then screened by LAN. In this final stage, 11 (6.2%) studies, corresponding to 9 interventions, were identified for inclusion in the review, with 8 (73%) found through the electronic search and 3 (27%) through manual searches of the reference lists of relevant articles. The updated search identified a further 7148 citations after deduplication, of which 3 met inclusion criteria and were included in the review.

Data Extraction

Data were extracted by 1 reviewer (LAN, HAB, or BF) and reviewed for accuracy and completeness by a second reviewer. After verifying all the extracted data, discrepancies were resolved by discussion or adjudication by another author (MF). Extracted data included information on study characteristics (ie, authors, publication year, country, study design, and study population); intervention characteristics (ie, characteristics of the technology, app name, therapeutic modality, and intervention outcomes); and feasibility and acceptability.

Quality Assessment

We used the Mixed Methods Appraisal Tool (MMAT; version 2018) to assess the methodological quality of the included studies [ 34 ]. MMAT was developed by combining the core relevant methodological criteria found in different well-known and widely used qualitative and quantitative critical appraisal tools. The MMAT consists of 2 screening questions applicable to all types of study design and a further 5 questions applicable to specific study designs. Responses were rated on a categorical scale as “no,” “unclear,” or “yes” to any of the methodological quality criteria. Quality assessments were made by 1 reviewer (TM). We did not exclude any studies based on quality assessment scores.

Data Synthesis and Analysis

The extracted data were collated and summarized to produce a narrative summary of the study; sample characteristics; and acceptability, feasibility, and engagement outcomes. A codebook approach was used to code and synthesize implementation data from all available sources according to the outcome categories [ 35 ].

Qualitative Study

Study context.

The Emotional Competence for Well-Being (ECoWeB) cohort multiple randomized controlled trial involved a longitudinal prospective cohort to examine the well-being, mental health, and emotional competence in individuals aged 16 to 22 years across 12 months. The experimental arm was an emotional competence self-help app (ie, MyMoodCoach).

Intervention

MyMoodCoach was designed to test if an app could improve different processes affecting mental well-being, including, but not limited to, improving emotion regulation by reducing maladaptive strategies such as worry and rumination and replacing them with constructive alternatives and problem-solving and enhancing emotional knowledge and perception through psychoeducation and learning tasks. The app was designed for young people broadly and was not targeted at a specific population. Full details of the ECoWeB trial are reported in the study protocol [ 36 ].

Participant Recruitment

In parallel to the trial, we additionally recruited young people to understand the views and experiences of those underrepresented in the study sample (and most other app-based studies). As there was an overrecruitment of White, university-educated female participants in the study sample, we decided to only recruit male participants and to focus on 2 specific groups: NEET men and migrant populations (including both voluntary and forced migrants). In the United Kingdom and Spain, recruitment was conducted through a variety of channels, including Twitter, targeted adverts on social media (eg, Instagram and Facebook), newsletters sent to youth and practitioner networks, outreach to third sector organizations and mental health support groups, and advertisements placed on university and charity websites. In Germany, participants were recruited directly from refugee homes, integration courses, migration services, and youth centers.

Inclusion criteria required the participants to be (1) aged between 16 and 22 years; (2) able to speak and read English, Spanish, or German; (3) male; (4) NEET (United Kingdom), NEET or migrant (Spain), or asylum seeker or refugee (Germany); (5) having access to a smartphone with the minimal technological specifications necessary for the app (ie, iOS 9 or later or Android 8.0 or later); and (6) having access to the internet via mobile data or Wi-Fi. The exclusion criteria were having current suicidal ideation, psychosis, or bipolar disorder.

The research team was led by the principal investigator, MF, professor of Adolescent Psychiatry and Consultant in Children’s Psychological Medicine, with particular expertise in the mental health needs of refugee populations. EW, professor of Experimental and Applied Clinical Psychology, Chartered Clinical Psychologist, provided additional supervisory input and guidance as a leading expert in the field of child and adolescent mental health research. SW, professor of Education with a focus on youth research and director at German Youth Institute, provided supervisory oversight of the interviews in Germany. HAB and LAN were postdoctoral researchers at the time the research was conducted, both of whom have several years of experience conducting qualitative research with young people and expertise in analyzing qualitative data. SC, researcher at the German Youth Institute, conducted and analyzed the interviews in Germany and has experience conducting qualitative research with refugee populations. The authors had no relationship with any of the participants.

In the United Kingdom and Spain, participation involved (1) short questionnaires about mood and feelings and demographic questions such as age, gender, race, ethnicity, educational attainment, and country of origin; (2) downloading MyMoodCoach and using it for 4 weeks; and (3) completing a follow-up interview. Interviews were conducted by HAB and LAN via MS Teams (Microsoft, Corp) and lasted approximately 45 minutes. A similar procedure was followed in conducting the interviews with refugees in Germany, but given the likely language barriers in navigating the app, an additional, earlier interview was included 2 to 3 weeks after the initial instructions had been sent to explore if features of the app were understood and to clarify questions. After 2 further weeks, a second interview was conducted. One interview was conducted in person, while all the others were conducted over the telephone.

Interviews followed a semistructured schedule based on a taxonomy of implementation constructs [ 26 ]. Topics included self-reported app use, satisfaction and feedback about content, usability, and acceptability ( Multimedia Appendix 2 ). In Germany, the interview topic guide was translated and adapted to the target group. As not to stigmatize the young men, the wording “refugee” was avoided, and instead, when referring to the target group, the wording “young man such as yourself” was used.

Ethical Considerations

In the United Kingdom, ethical approval was granted by the University of Exeter Research Ethics Committee (eCLESPsy000048 v10.0); in Spain, by the Jaime I University Research Ethics Committee (CD/93/2021); and in Germany, by the Ethics Committee of the Faculty of Medicine at Ludwig-Maximilians University Munich (PNr 19-0468/19-0315). Cognizant of the ethical and practical implications of conducting research with underserved populations, we conducted the interviews in private, quiet spaces and in a friendly and reassuring manner to ensure that participants felt comfortable and safe.

With prior consent, all interviews were audio recorded. Participants were reimbursed with a shopping voucher of up to €50 or £50 (US $64) for their time and app use.

Analytic Strategy

Interviews were transcribed verbatim, and the transcripts were assigned a unique pseudonym to anonymize participants. The interviews were analyzed using a combination of theory- and data-driven analysis techniques, consisting primarily of deductive, theory-driven thematic analysis [ 37 ]. Analysis of the transcripts in the United Kingdom and Spain was conducted by HAB, using NVivo (version 11; Lumivero). Following a similar procedure, the interviews in Germany were transcribed and analyzed using the coding software MAXQDA (VERBI GmbH) by SC.

Initial familiarization with the data was achieved through the transcription process and iterative rereading of the interviews. Analysis was carried out through a recursive process of open coding, when concepts were named and their properties and dimensions identified, followed by axial coding, when links and associations were drawn between codes. Codes were based on language used by the young people and were applied to each new unit of meaning. Data extracts were multiply coded when appropriate, as were contradictory and minority features of the data. The data set was iteratively reviewed, and codes were systematically applied to the whole data set until a finalized coding manual was established. Codes were organized into potential themes using thematic maps and tables. The development of the coding manual was iteratively reviewed and refined through discussion with all authors throughout the analysis process to ensure the reliability and rigor of the process and results.

Approach to Inquiry

Analysis was conducted from a critical realist perspective to provide a more nuanced understanding and explanation of participants’ experiences [ 38 ]. This position assumes that although participants’ accounts provide important insights about the real world, these accounts are not objective and represent an interpretation of reality [ 39 ]. These data require interpretation and explanation by the researcher, who also has their own perspectives on the world, to better understand the underlying mechanisms and processes and, in turn, make recommendations for practice [ 40 ].

Study Characteristics

Characteristics of the 14 included studies, examining 12 interventions, are presented in Table 1 . The included studies were published in the United States (n=3), Australia (n=3), Switzerland (n=2), India (n=2), South Korea (n=1) and Germany (n=1). As for the type of intervention, 7 apps were stand-alone, and the rest (n=5) were delivered in combination with other forms of professional support. The interventions (n=12) were targeted at apprentices and the unemployed (n=3), homeless populations (n=2), those with physical health conditions (n=2), sexual and gender minoritized individuals (n=3), those residing in LMICs (n=1) and those with co-occurring autism spectrum disorders (n=1). Most of the interventions had been co-designed with young people (11/12, 92%). The study sample size ranged from 9 to 877, with 4 of the 14 studies having a sample size of >200 participants.

a CBT: cognitive behavioral therapy.

b CF: cystic fibrosis.

c JIA: juvenile idiopathic arthritis.

d T1D: type 1 diabetes.

e RCT: randomized controlled trial.

f iCBT: internet-based cognitive behavioral therapy.

g Mobile phones were preloaded with several apps designed to promote mental health wellness and provide real-time resources. Pocket Helper was 1 app specifically designed for this study.

h Studies identified in the updated search.

Study Quality

The included studies varied in their methodological quality ( Multimedia Appendix 3 ) [ 41 - 54 ]. Most (13/14, 93%) were judged to contain possible limitations in at least 1 criterion. All studies but 1 (13/14, 93%) were clear in their description of study participants or the process of recruiting a sample representative of the population of interest. All studies addressed the research question using collected data and reported in some way on feasibility and acceptability. Most studies (12/14, 86%) effectively used appropriate qualitative, quantitative, or mixed methods to answer their research question. However, many studies (6/14, 43%) did not have a sufficiently large sample to warrant definitive conclusions about the feasibility and acceptability of the intervention studied.

Acceptability and Feasibility

User acceptability was found to be high across all included studies, with participants rating the apps positively and reporting high satisfaction with the content of the interventions ( Table 2 ). In studies where participants were asked to indicate if they would recommend the study to someone else, the vast majority reported they would [ 42 , 44 - 46 , 51 ]. It is notable that despite many users across the studies reporting high satisfaction levels and being willing to recommend the apps, they themselves did not intend to continue using the apps (ie, low predicted engagement), as they did not find them useful or relevant for their own circumstances [ 41 , 42 , 51 ]. It is also important to note that many studies incentivized participation with payments, prize draws, and vouchers, and this may have influenced acceptability ratings and engagement [ 41 - 43 , 46 , 49 - 51 , 53 , 54 ]. Furthermore, in the only study that asked if participants would pay to use apps, most were unwilling to do so [ 42 ].

a Studies identified in the updated search.

Regarding co-design strategies used in these studies, early stakeholder consultation and service provider focus groups were conducted in the early development phase of POD Adventures, a gamified intervention for people with or at risk of anxiety, depression, and conduct difficulties in India; the results highlighted that “self-help” was not a culturally congruent concept for most Indian adolescents [ 47 ]. This early feedback was important as it revealed the norms around seeking or receiving direct instruction from parents, teachers, and other elders and that support from a counselor might be necessary to ensure acceptability, feasibility, and engagement [ 47 ]. The app was therefore designed to incorporate a combination of teaching methods, including direct instruction, modeling, and practice to accommodate different learning styles and to emphasize self-efficacy [ 47 ]. Furthermore, user testing also highlighted the need for more direct language, particularly around problem-solving concepts [ 47 ]. The iterative study methodology used in this study enabled the participants to guide the development and provide their inputs at each stage to increase acceptability and feasibility.

Although, overall, the apps were well received by young people, poor engagement (eg, not engaging at the recommended frequency or complete the full course of the intervention), measured through both self-report, intervention adherence, and data capture was a commonly reported issue. Many studies failed to achieve continued participation, with high rates of attrition [ 41 , 42 , 44 ]. In addition, app use often decreased markedly after the initial few weeks [ 41 , 42 , 44 ]. The results of some studies suggested that engagement (in the form of intervention completion) was related to the effectiveness the intervention [ 41 , 42 ]. Although engagement was problematic in many of the stand-alone interventions, engagement and study participation in a school setting seemed more promising [ 49 ]. For example, a proactive invitation for study participation in a school enabled 4 out of 5 eligible adolescents to participate in the “ready4life” life skills program [ 49 ]. This strategy consisted of individuals who were trained in the program to be delivered, giving arranged sessions lasting 30 minutes in participating vocational schools during regular school lessons reserved for health education. Within this session, the students were informed about and invited to participate in the study, including being informed about the study’s aims and assessments, reimbursement, and data protection.

Barriers to Engagement

Qualitative interviews and user feedback provided important insights about relevant barriers to engagement. The most frequently mentioned reasons for not using the app were that participants could not see the obvious benefits of using the app [ 41 ], lack of time or forgetting [ 42 ], and technical difficulties [ 41 , 50 , 51 ]. In a life skills training app for vocational students, participation was lower in male adolescents and among those reporting an immigrant background [ 49 ], although the reasons behind this poor engagement remained unclear.

What Do Young People Want From Apps?

There was some heterogeneity between studies in terms of the features and content that participants found acceptable and appropriate. For example, findings suggested that young people who experience homelessness tended to prefer both automated and self-help features compared with ones involving more direct human interaction [ 46 ]. However, participants in other studies valued both human interaction with professionals either via the app interface or through face-to-face contact and self-help features [ 47 ]. Human support was suggested as being helpful in offering both instruction and guidance as well as personalized support when needed. Numerous participants wanted opportunities to interact with peers [ 43 , 44 , 52 ] and even suggested connecting apps to social media [ 41 ]. Others also wanted the design of the apps to be more attractive (eg, improve the layout and create a more intuitive structure) and made suggestions about how gamifying apps could make them more interesting [ 41 , 43 , 47 ].

We interviewed 13 young men in the United Kingdom (age: mean 18.7, SD 2.5 y), 2 in Spain (age: mean 17, SD 0 y), and 5 in Germany (age: mean 20.2, SD 1.6 y). In the United Kingdom, 62% (8/13) of the participants self-reported as ethnically White, compared with 50% (1/2) in Spain and 20% (1/5) in Germany ( Table 3 ).

In terms of participants’ mental health and well-being ( Table 4 ), the mean Patient Health Questionnaire-9 score in the United Kingdom was 9.7 (SD 7.3) compared with 5 (SD 1.4) in Spain.

a WEBWBS: Warwick-Edinburgh Mental Well-Being Scale.

b Not available.

c PHQ-9: Patient Health Questionnaire-9.

d GAD-7: Generalized Anxiety Disorder Assessment.

A key finding was that despite best efforts and financial incentives, recruiting underserved young male participants, especially in Spain and Germany, was challenging. This might suggest that these young people may not deem such an emotional competence app as relevant or useful to them, making recruitment and engagement problematic. We also assessed if the app was deemed acceptable (ie, useful, agreeable, palatable, or satisfactory) and appropriate (ie, relevant, suitable, or compatible). Overall, the app was viewed by participants in the United Kingdom, Spain, and Germany as being appropriate and relevant for young people of different ages and walks of life, as they thought that all young people had a smartphone and were adept at using technology:

So, I was able to learn about my feelings, I was able to evaluate how I actually felt today, concerning my feelings, if I was angry or I was sad. I was actually able to write them down in detail. [Participant in Germany]

Several participants commented that the content of the app was best suited to university and school students. Another common view was that the app was better suited to those struggling with their mental health and that it was less relevant for those for whom things were going well. Many participants perceived the app to be aimed at improving mental health problems, as opposed to being a universal intervention intended to improve well-being, which represented a barrier to engagement. Of those who reported that the app was not relevant to them, they did see it as being of potential use to friends and family members who were stressed, anxious, or going through a difficult time:

There will be folks who maybe aren’t going through a good time in their lives, and they will need the app to feel... to understand themselves, mostly. And I think it’s relevant at any age, because I am lucky that I don’t think I need it as much as someone else who feels like that. [Participant in Spain]

Partly it was important, partly it was not. I’ll give an example again, for example if a refugee came to Germany from a war zone, it’s going to be difficult, very difficult to find a topic that would fit him, for the future I mean, so the version now is already okay if you want all persons to use this app. Partly it’s already relevant and partly it’s not. If someone has mental problems or bad experiences, you cannot find such a topic in the app. [Participant in Germany]

Although some participants reported using the app regularly during the 4-week study period, a consistent finding was that participants tended to use the app most when they first downloaded it, with a marked reduction in use over time:

Uh, I probably used it about three times in the first week. And then not really that much at all I’m afraid. [Participant in the United Kingdom]

I don’t know, I just dropped off using a little bit after a couple of weeks, but I’ve been trying to keep on top of doing that like the daily rating things and everything.... I kind of lost my motivation to use it. [Participant in the United Kingdom]

We identified several barriers that hindered participants’ engagement and use of the app. These included the following: (1) repetitive and time-consuming app contents, (2) a paucity of new content and personalized or interactive tools (eg, matching mood to tools), (3) unclear instructions, (4) a lack of rationale for the app, (5) perceiving the app as not being relevant, (6) a lack of motivation, and (7) privacy concerns:

Yes, for example, I would not like to write in this diary, because I do not know if it would be one hundred percent anonymous and if others might read it. And maybe I have more privacy if I do not write it. [Participant in Germany]

I think by now I would slowly stop using the app. It was nice up to this point, but I think for me I might need a step further now. To really deal with my personal problems and I don’t know how much an app like this can help and that rather an expert and therapy is needed. [Participant in Germany]

For the asylum seekers and refugees in Germany, the language and content of the app was not suited to their needs. The participants would have preferred the app in their native language as some had to use translation programs to help access the content. Furthermore, specific topics of relevance to refugees were missing, such as dealing with asylum uncertainty, whereabouts of family members, and their living situation.

Finally, underserved young people, including asylum seekers and refugees, migrants, and those NEET, are more likely to experience financial deprivations and therefore less likely to pay directly for apps, especially for those that do not address their primary difficulties:

If it came to the point that I had to pay for it, I would look for free options. [Participant in Germany]

Summary of Findings

The use of mobile apps in mental health care continues to attract interest and investment; however, research geared toward understanding the needs of marginalized and underserved populations is still nascent. This study, focusing on the implementation of mental health apps in underserved young people, highlighted that little research exists to support the widespread adoption of these apps as a mental health intervention for marginalized and underserved groups. Findings from both our systematic review and qualitative study were largely consistent: markers of acceptability and usability were positive; however, engagement for underserved young people was low, which is notable given the widespread ownership of smartphones [ 55 , 56 ]. To date, research has focused primarily on efficacy studies rather than effectiveness and implementation in “real-world” settings and may have overestimated users’ “natural tendency” to adopt smartphone apps for their mental health and well-being [ 57 ]. Our findings suggest that despite the rapid proliferation of mobile mental health technology, the uptake and engagement of mental health apps among marginalized young people are low and remain a key implementation challenge.

Our data suggest that establishing and maintaining user commitment and engagement in the content of the intervention as intended is a pervasive challenge across mental health apps and marginalized populations, and premature dropout was prominent in nearly all the included studies. This is consistent with the literature that suggests that the majority of those offered these app-based interventions do not engage at the recommended frequency or complete the full course of treatment [ 58 , 59 ]. In this study, various app components were associated with engagement level, with the most engaging interventions providing young people with some form of associated real-human interaction and those having a more interactive interface. This aligned with other findings that the feedback of personalized information to participants is an especially important aspect of creating engaging and impactful digital tools [ 60 ]. Young people tend to quickly disengage if there are technical difficulties or if the app does not specifically target their perceived needs [ 41 , 50 , 51 ]. Furthermore, recruitment of marginalized groups to app-based studies is difficult. For instance, in this study, the use of advertisements, financial incentives, vouchers, and prize draw incentives seemed to be insufficient to recruit a significant number of participants in Spain and Germany.

Measuring engagement is a challenge that has likely contributed to our lack of knowledge on app components that effectively increase user engagement. Reporting engagement with mental health apps in intervention trials is highly variable, and a number of basic metrics of intervention engagement, such as rate of intervention uptake, weekly use patterns, and number of intervention completers, are available, yet not routinely reported [ 58 , 59 ]. The results of this study highlight the importance of objective engagement measures and that relying on positive subjective self-reports of usability, satisfaction, acceptability, and feasibility is insufficient to determine actual engagement. Furthermore, the findings suggest that apps involving human interactions with a professional (eg, therapist or counselor) or that are completed in a supervised setting tend to be more acceptable and effective and have higher engagement rates [ 47 , 48 ]. Our research suggests that similar to traditional face-to-face mental health services, app-based programs still face numerous barriers to reach marginalized youth, especially since the mental health apps available to the public do not seem to consider the unique developmental needs of these groups, participants do not seem to perceive an obvious benefit from using them, and some potential users prefer to interact with a professional face to face. Thus, it is also possible that the digital mental health field might be inadvertently contributing to mental health inequities among this population by not engaging marginalized groups sufficiently at the outset of research to ensure that the designed app meets their needs. However, for the studies included in this study that did engage these groups in the co-design of the apps, there was no notable improvement in engagement. Thus, we hope these findings encourage researchers and clinicians to think more critically of the role that mental health apps can truly have in addressing mental health equities among underserved groups.

As in other areas of mental health research, young people from LMICs were underrepresented in these studies, which typically originated from high-income settings, including the United States, Australia, and Canada. There are relatively few app-based interventions that were designed or adapted for young people in LMICs that have been rigorously evaluated or are even available in local languages [ 47 , 48 ]. Many living in LMIC regions, for example, adults in Asian countries, are often faced with apps that are not culturally relevant or in the right language [ 61 ]. These inequities are surprising given the high rates of smartphone use in Asia, even in rural regions [ 62 ]. Yet, it is still likely that youth in this region faced barriers related to data availability and more limited phone access, which will likely inhibit the broad implementation of apps beyond research studies [ 16 ]. Considerable work is required to ensure the availability of mental health apps that fit a wide range of user needs and preferences. It is important to ensure that the acceptability and feasibility of mental health apps for young people residing in LMICs are prioritized so that they are not further excluded from relevant mental health research.

Finally, a significant challenge is the lack of diversity in mental health app research participation, which limits our understanding of “real-world” efficacy and implementation for underserved and marginalized groups. While undoubtedly invaluable, and indeed deemed gold standard when evaluating efficacy of interventions, randomized controlled trial of mental health apps are not without flaws [ 63 , 64 ]. Trial recruitment is often highly selective due to stringent inclusion and exclusion criteria resulting in lower inclusion in research than one would expect from population estimates [ 65 ]. In the United Kingdom, the National Institute for Health and Care Research data have revealed that geographies with the highest burden of disease also have the lowest number of patients taking part in research [ 66 ]. The postcodes in which research recruitment is low also aligns closely to areas where earnings are the lowest and indexes of deprivation are the highest [ 66 ]. There are many reasons why some groups are underrepresented in research: language barriers, culturally inappropriate explanations, poor health literacy and the use of jargon, communication not being suitable for people with special learning needs, requirement to complete many administrative forms, negative financial impact in participating, lack of effective incentives for participation, or lack of clarity around incentives, and specific cultural and religious beliefs [ 66 ]. Failing to include a broad range of participants is problematic in that results may not be generalizable to a broad population.

Limitations

Although this research was carefully executed and used a robust methodological approach with an exhaustive search strategy, it is not without limitations. Foremost, although the systematic review attempted to identify and include as many articles as possible, some papers may have been missed because of the inconsistencies in how feasibility and acceptability outcomes are recorded and reported. It was also difficult to ensure that all apps for this age group were identified because those aged between 15 and 25 years are harder to differentiate in adolescent and adult studies, meaning we might have missed some relevant studies where data could not be disaggregated by age. The exclusion of gray literature (eg, institutional reports and websites) may have also made us overlook potentially relevant apps, albeit lacking the quality assurance of peer-reviewed research. It is also likely that commercial organizations, including app companies, collect rich user demographic and engagement data but do not share it publicly, thus limiting our ability to conduct empirical analyses about the “real-world” acceptability, engagement, and implementation for specific populations. We did not analyze the extent to which publication bias may have influenced the results of our search, and, therefore, there may be a much higher number of mental health apps that have been developed with an underserved sample of young people, but due to their lack of efficacy or acceptability, these studies have not been submitted or accepted for publication. The sample sizes of many of the included studies were relatively low, which potentially limits their generalizability. However, we included all study designs so as to ensure that our learning from existing research was maximized. Furthermore, many of the studies included in the systematic review, as well as our qualitative study, had some form of language competency as an inclusion criterion (eg, English speaking), which likely excludes important perspectives from the results. For the qualitative study, we were only able to gather data from those who had used the app at least once and who were therefore somewhat engaged in the app. Despite our best efforts, we were unable to recruit participants who, following consent, had never then downloaded or used the app and so could not explore barriers to engagement for the least engaged young people or understand why the app was not appealing to those who chose not to proceed or take part. Those who did participate in this research were financially incentivized to do so and often highlighted the importance of this incentive in keeping them engaged. Therefore, we were unable to draw conclusions about the naturalistic engagement, feasibility, and acceptability of the app, if it were to be made available without payment in schools, universities, and health services or to be made commercially available on the app marketplace. It is also possible that social desirability bias (ie, a tendency to present reality to align with what is perceived to be socially acceptable) occurred during the interviews, whereby participants responded to the interview questions in a manner that they believed would be more acceptable to the study team, concealing their true opinions or experiences [ 67 , 68 ]. As previously noted by others, results may be subject to further bias in that findings could be led by more articulate young people, while it is more difficult to hear the voices of those who are less articulate or digitally literate [ 69 ]. Finally, it is also possible that the positionality of the research team, including our own experiences, backgrounds, and biases, impacted what information participants disclosed to the research team as well as the interpretation of the qualitative data in this study.

Recommendations of Adaptations to Increase Acceptability, Feasibility, and Engagement

To overcome this complex engagement and implementation challenge, we have taken together our findings with relevant previous literature to generate 3 key suggestions about how to improve the feasibility and potential utility of apps for young people from marginalized and underserved populations.

Increasing Participant Diversity in Mental Health Intervention Research

Studies should aim to prioritize the inclusion of marginalized groups in trials testing the effectiveness of digital interventions by intentionally planning recruitment efforts aimed to reach these communities [ 70 ]. First, steps can be taken to build trust, connections, and credibility between the research team and these communities. NHS England [ 66 ] suggests involving representatives from those groups during the inception and implementation of recruitment efforts. This approach ensures that the intervention is relevant to the target group by meeting their preferences and needs, incorporating culturally salient factors relevant for recruitment efforts, addressing concerns about community mistrust and participant resource constraints, and establishing partnerships with key community stakeholders that can be gatekeepers in the community [ 14 , 71 ]. These strategies are likely to improve research accessibility, recruitment, and retention. Research teams need to ensure that the findings and any actionable takeaways from the research conducted with the participants are shared with them by asking participants how they would like to receive this information (eg, verbal, written, or via a trusted advocate). Equally important is to explain that the research process can be slow. These steps help create a positive legacy for the research project and build trust between individuals and public institutions, helping future health researchers to further address underrepresentation of marginalized groups in digital research.

Identifying and Addressing Needs and Preferences of Underserved and Marginalized Groups by Using Human-Centered Design Principles

A comprehensive understanding of the needs, challenges, and life circumstances of the target population is a key implementation driver for designing relevant, engaging, and effective mental health apps. This knowledge is particularly important when the app is a stand-alone intervention received during daily life outside of traditional psychotherapy or human support [ 50 ]. This goal can be best achieved through a participatory approach, which reflects a growing recognition among intervention researchers and developers that end users need to be involved in the creation of interventions and their future iterations [ 47 , 72 ]. This process may involve a series of stages, including (1) person-centered co-design to ensure that tools are developed to be acceptable to the underserved or marginalized populations as well as meet their specific needs, life circumstances, and cultural norms [ 47 ]; (2) iterative testing that incorporates users’ feedback on a rolling basis to ensure the relevance of the intervention [ 43 , 47 , 72 ]; and (3) changes and adaptations needed to meet users’ needs in “real-world” settings including consideration of economic viability and implementation [ 27 ].

Especially relevant for the underserved and marginalized groups is the need (or lack thereof) to culturally adapt app interventions for specific racial, ethnic, or cultural groups through this person-centered design. In traditional face-to-face interventions, some have argued that all treatments need to be culturally adapted to ensure their validity, relevance, and effectiveness since these interventions are often developed with individuals who can be substantially different from some marginalized populations [ 73 ]. Similar to culturally adapted face-to-face interventions [ 74 - 76 ], culturally adapted digital mental health interventions seem to be effective [ 77 , 78 ]. However, there is no evidence that these culturally adapted interventions outperform the original programs [ 79 , 80 ]. Given that culturally adapting digital interventions is a time-consuming and resource-intensive process, this approach may not be sustainable and limit the dissemination and implementation impact of app programs [ 28 ]. In lieu of culturally adapting digital interventions without careful consideration, Ramos and Chavira [ 28 ] recommend using information gathered through person-centered approaches to integrate culture into the use of already available digital interventions (including apps), using an idiographic, flexible, and personalized approach. This strategy may have a broader implementation and dissemination potential, given that few researchers and clinicians are in a position to develop new apps.

Embedding Apps Within Existing Care Structures

Several systematic reviews and meta-analyses have demonstrated that app-based mental health interventions with a human-support component are more effective and more acceptable than stand-alone, fully automatized, or self-administered apps [ 13 , 25 , 81 ]. Young people seem to want practical skills and usable tools to apply to their current daily life stressors to improve their well-being and functioning. Intervention engagement is enhanced if the intervention serves an obvious purpose, is relevant, and has a clear rationale and instructions, and embedding these interventions within the systems and structures that are already working with users (eg, clinical services, schools, universities, and community agencies) will likely improve implementation. Considering the broad and highly varied nature of intervention formats and modalities, it may be useful for future research to focus on identifying core components of app-based interventions (ie, active ingredients of interventions associated with uptake, adherence, and clinical outcomes) that will allow such integration of app interventions into the varied context of care for marginalized youth.

Conclusions

Despite the enthusiasm that has surrounded the potential of digital technologies to revolutionize mental health and health care service delivery, little evidence yet supports the use of mental health apps for marginalized and underserved young people. Despite the substantial financial and human investment directed to the development of mental health apps over several years, only a small proportion have empirical evidence to support their effectiveness, and there have been few attempts to develop or adapt interventions to meet some of the more unique and heterogeneous needs of diverse groups of young people. Although acceptability seems to be good, engagement is poor and attrition is high, particularly if not supported by in-person elements. Given that most interventions are implemented in high-income countries, very little is known about the generalizability of the findings to LMICs and to a range of adolescents and young people with different socioeconomic, cultural, and racial backgrounds. In this paper, we have drawn several insights about the feasibility and acceptability of mental health apps for underserved young people that may be useful to future app-based mental health promotion and treatment projects. However, before the widespread adoption and scaling-up of digital mental health interventions progresses further, especially for more vulnerable and underserved populations and in settings with limited resources, a greater understanding is needed on the unique barriers faced by these groups in accessing treatment and the types of services young people themselves prefer (eg, standard vs digital) followed by more rigorous and consistent demonstrations of feasibility, effectiveness, and cost-effectiveness.

Acknowledgments

This project received funding from the European Union’s Horizon 2020 research and innovation program (grant agreement number 754657).

The authors are grateful to the young people who took the time to participate in this research and who shared their insights with us. The authors would also like to thank those who supported this research including professional youth advisor Emily Bampton, research assistant Catherine Reeve, and researchers Dr Alexandra Langmeyer and Simon Weiser. Finally, the authors would like to thank the ECoWeB (Emotional Competence for Well-Being) Consortium for their support and feedback throughout the duration of this research, including, but not limited to, Dr Lexy Newbold, Dr Azucena Garcia Palacios, and Dr Guadalupe Molinari.

Data Availability

The data extracted to support the findings of the systematic review are available from the corresponding author upon reasonable request. Due to the confidential and sensitive nature of the interview transcripts, qualitative data will not be made available.

Authors' Contributions

HAB, LAN, and MF designed the systematic review including the research questions and methods. LAN carried out the database search. HAB, LAN, TM, and BF conducted the study screening and data extraction. TM did the study quality assessments, and HAB did the data synthesis and analysis. MF, SW, EW, and HAB were involved in the conception of the qualitative study. HAB, LAN, and SC conducted the quality study including conducting the qualitative interviews and analysis. HAB wrote the first draft and HAB, LAN, MF, and GR contributed substantially to manuscript drafting. All authors contributed to the manuscript and approved the submitted version.

Conflicts of Interest

None declared.

Search strategy.

Topic guide.

Study quality assessment.

• Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. Jun 2005;62(6):593-602. [ CrossRef ] [ Medline ]
• Solmi M, Radua J, Olivola M, Croce E, Soardo L, Salazar de Pablo G, et al. Age at onset of mental disorders worldwide: large-scale meta-analysis of 192 epidemiological studies. Mol Psychiatry. Jan 2022;27(1):281-295. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brown A, Rice SM, Rickwood DJ, Parker AG. Systematic review of barriers and facilitators to accessing and engaging with mental health care among at-risk young people. Asia Pac Psychiatry. Mar 03, 2016;8(1):3-22. [ CrossRef ] [ Medline ]
• Marrast L, Himmelstein DU, Woolhandler S. Racial and ethnic disparities in mental health care for children and young adults: a national study. Int J Health Serv. Oct 20, 2016;46(4):810-824. [ CrossRef ] [ Medline ]
• Cook BL, Trinh NH, Li Z, Hou SS, Progovac AM. Trends in racial-ethnic disparities in access to mental health care, 2004-2012. Psychiatr Serv. Jan 01, 2017;68(1):9-16. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rimes KA, Shivakumar S, Ussher G, Baker D, Rahman Q, West E. Psychosocial factors associated with suicide attempts, ideation, and future risk in lesbian, gay, and bisexual youth. Crisis. Mar 2019;40(2):83-92. [ CrossRef ] [ Medline ]
• Reiss F. Socioeconomic inequalities and mental health problems in children and adolescents: a systematic review. Soc Sci Med. Aug 2013;90:24-31. [ CrossRef ] [ Medline ]
• Warnecke RB, Oh A, Breen N, Gehlert S, Paskett E, Tucker KL, et al. Approaching health disparities from a population perspective: the National Institutes of Health Centers for Population Health and Health Disparities. Am J Public Health. Sep 2008;98(9):1608-1615. [ CrossRef ]
• Schueller SM, Hunter JF, Figueroa C, Aguilera A. Use of digital mental health for marginalized and underserved populations. Curr Treat Options Psych. Jul 5, 2019;6(3):243-255. [ CrossRef ]
• Introduction: NIH minority health and health disparities strategic plan 2021-2025. National Institutes of Health National Institute on Minority Health and Health Disparities. URL: https://www.nimhd.nih.gov/about/strategic-plan/nih-strategic-plan-directors-foreword.html [accessed 2024-06-29]
• Improving inclusion of under-served groups in clinical research: Guidance from INCLUDE project. National Institute for Health and Care Excellence. Aug 7, 2020. URL: https:/​/www.​nihr.ac.uk/​documents/​improving-inclusion-of-under-served-groups-in-clinical-research-guidance-from-include-project/​25435 [accessed 2024-06-29]
• Evans-Lacko S, Aguilar-Gaxiola S, Al-Hamzawi A, Alonso J, Benjet C, Bruffaerts R, et al. Socio-economic variations in the mental health treatment gap for people with anxiety, mood, and substance use disorders: results from the WHO World Mental Health (WMH) surveys. Psychol Med. Nov 27, 2017;48(9):1560-1571. [ CrossRef ]
• Hollis C, Falconer CJ, Martin JL, Whittington C, Stockton S, Glazebrook C, et al. Annual research review: digital health interventions for children and young people with mental health problems - a systematic and meta-review. J Child Psychol Psychiatry. Apr 10, 2017;58(4):474-503. [ CrossRef ] [ Medline ]
• Friis-Healy EA, Nagy GA, Kollins SH. It is time to REACT: opportunities for digital mental health apps to reduce mental health disparities in racially and ethnically minoritized groups. JMIR Ment Health. Jan 26, 2021;8(1):e25456. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mobile fact sheet. Pew Research Center. Jan 31, 2024. URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2024-07-12]
• Silver L. 2. In emerging economies, smartphone adoption has grown more quickly among younger generations. Pew Research Center. Feb 5, 2019. URL: https:/​/www.​pewresearch.org/​global/​2019/​02/​05/​in-emerging-economies-smartphone-adoption-has-grown-more-quickly-among-younger-generations/​ [accessed 2024-07-12]
• Pretorius C, Chambers D, Coyle D. Young people's online help-seeking and mental health difficulties: systematic narrative review. J Med Internet Res. Nov 19, 2019;21(11):e13873. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grist R, Porter J, Stallard P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J Med Internet Res. May 25, 2017;19(5):e176. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lehtimaki S, Martic J, Wahl B, Foster KT, Schwalbe N. Evidence on digital mental health interventions for adolescents and young people: systematic overview. JMIR Ment Health. Apr 29, 2021;8(4):e25847. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Leech T, Dorstyn D, Taylor A, Li W. Mental health apps for adolescents and young adults: a systematic review of randomised controlled trials. Child Youth Serv Rev. Aug 2021;127:106073. [ CrossRef ]
• Buttazzoni A, Brar K, Minaker L. Smartphone-based interventions and internalizing disorders in youth: systematic review and meta-analysis. J Med Internet Res. Jan 11, 2021;23(1):e16490. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eisenstadt M, Liverpool S, Infanti E, Ciuvat RM, Carlsson C. Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: systematic review and meta-analysis. JMIR Ment Health. Nov 08, 2021;8(11):e31170. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lipschitz J, Hogan TP, Bauer MS, Mohr DC. Closing the research-to-practice gap in digital psychiatry: the need to integrate implementation science. J Clin Psychiatry. May 14, 2019;80(3):18com12659. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hollis C. Youth mental health: risks and opportunities in the digital world. World Psychiatry. Feb 11, 2022;21(1):81-82. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bear HA, Ayala Nunes L, DeJesus J, Liverpool S, Moltrecht B, Neelakantan L, et al. Determination of markers of successful implementation of mental health apps for young people: systematic review. J Med Internet Res. Nov 09, 2022;24(11):e40347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. Mar 19, 2011;38(2):65-76. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ramos G, Ponting C, Labao JP, Sobowale K. Considerations of diversity, equity, and inclusion in mental health apps: a scoping review of evaluation frameworks. Behav Res Ther. Dec 2021;147:103990. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ramos G, Chavira DA. Use of technology to provide mental health care for racial and ethnic minorities: evidence, promise, and challenges. Cognit Behav Pract. Feb 2022;29(1):15-40. [ CrossRef ]
• Wies B, Landers C, Ienca M. Digital mental health for young people: a scoping review of ethical promises and challenges. Front Digit Health. Sep 6, 2021;3:697072. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Liverpool S, Mota CP, Sales CM, Čuš A, Carletto S, Hancheva C, et al. Engaging children and young people in digital mental health interventions: systematic review of modes of delivery, facilitators, and barriers. J Med Internet Res. Jun 23, 2020;22(6):e16317. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marcano Belisario JS, Huckvale K, Greenfield G, Car J, Gunn LH. Smartphone and tablet self management apps for asthma. Cochrane Database Syst Rev. Nov 27, 2013;2013(11):CD010013. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marcano Belisario JS, Jamsek J, Huckvale K, O'Donoghue J, Morrison CP, Car J. Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev. Jul 27, 2015;2015(7):MR000042. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. Mar 29, 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hong QN, Fàbregues S, Bartlett G, Boardman F, Cargo MP, Dagenais P, et al. The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Educ Inf. Dec 18, 2018;34(4):285-291. [ CrossRef ]
• Crabtree BF, Miller WL. A template approach to text analysis: developing and using codebooks. In: Crabtree BF, Miller WL, editors. Doing Qualitative Research. Thousand Oaks, CA. SAGE Publications, Inc; 1992:93-109.
• Newbold A, Warren FC, Taylor RS, Hulme C, Burnett S, Aas B, et al. Promotion of mental health in young adults via mobile phone app: study protocol of the ECoWeB (emotional competence for well-being in young adults) cohort multiple randomised trials. BMC Psychiatry. Sep 22, 2020;20(1):458. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. Jan 2006;3(2):77-101. [ CrossRef ]
• Ellaway RH, Kehoe A, Illing J. Critical realism and realist inquiry in medical education. Acad Med. Jul 2020;95(7):984-988. [ CrossRef ] [ Medline ]
• Willig C. Perspectives on the epistemological bases for qualitative research. In: Cooper H, Camic PM, Long DL, Panter AT, Rindskopf D, Sher KJ, editors. APA Handbook of Research Methods in Psychology, Vol. 1. Foundations, Planning, Measures, and Psychometrics. Washington, DC. American Psychological Association; 2012.
• Fletcher AJ. Applying critical realism in qualitative research: methodology meets method. Int J Soc Res Methodol. Feb 29, 2016;20(2):181-194. [ CrossRef ]
• Bohleber L, Crameri A, Eich-Stierli B, Telesko R, von Wyl A. Can we foster a culture of peer support and promote mental health in adolescence using a web-based app? A control group study. JMIR Ment Health. Sep 23, 2016;3(3):e45. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Deady M, Glozier N, Collins D, Einboden R, Lavender I, Wray A, et al. The utility of a mental health app in apprentice workers: a pilot study. Front Public Health. Sep 4, 2020;8:389. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fleming JB, Hill YN, Burns MN. Usability of a culturally informed mHealth intervention for symptoms of anxiety and depression: feedback from young sexual minority men. JMIR Hum Factors. Aug 25, 2017;4(3):e22. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Francis J, Cross D, Schultz A, Armstrong D, Nguyen R, Branch-Smith C. Developing a smartphone application to support social connectedness and wellbeing in young people with cystic fibrosis. J Cyst Fibros. Mar 2020;19(2):277-283. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Geirhos A, Domhardt M, Lunkenheimer F, Temming S, Holl RW, Minden K, et al. Feasibility and potential efficacy of a guided internet- and mobile-based CBT for adolescents and young adults with chronic medical conditions and comorbid depression or anxiety symptoms (youthCOACH): a randomized controlled pilot trial. BMC Pediatr. Jan 29, 2022;22(1):69. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Glover AC, Schueller SM, Winiarski DA, Smith DL, Karnik NS, Zalta AK. Automated mobile phone-based mental health resource for homeless youth: pilot study assessing feasibility and acceptability. JMIR Ment Health. Oct 11, 2019;6(10):e15144. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gonsalves PP, Hodgson ES, Kumar A, Aurora T, Chandak Y, Sharma R, et al. Design and development of the "POD adventures" smartphone game: a blended problem-solving intervention for adolescent mental health in India. Front Public Health. Aug 23, 2019;7:238. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gonsalves PP, Hodgson ES, Bhat B, Sharma R, Jambhale A, Michelson D, et al. App-based guided problem-solving intervention for adolescent mental health: a pilot cohort study in Indian schools. Evid Based Ment Health. Feb 18, 2021;24(1):11-18. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haug S, Paz Castro R, Meyer C, Filler A, Kowatsch T, Schaub MP. A mobile phone-based life skills training program for substance use prevention among adolescents: pre-post study on the acceptance and potential effectiveness of the program, Ready4life. JMIR Mhealth Uhealth. Oct 04, 2017;5(10):e143. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Leonard NR, Casarjian B, Fletcher RR, Praia C, Sherpa D, Kelemen A, et al. Theoretically-based emotion regulation strategies using a mobile app and wearable sensor among homeless adolescent mothers: acceptability and feasibility study. JMIR Pediatr Parent. Mar 01, 2018;1(1):e1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schueller SM, Glover AC, Rufa AK, Dowdle CL, Gross GD, Karnik NS, et al. A mobile phone-based intervention to improve mental health among homeless young adults: pilot feasibility trial. JMIR Mhealth Uhealth. Jul 02, 2019;7(7):e12347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Escobar-Viera CG, Porta G, Coulter RW, Martina J, Goldbach J, Rollman BL. A chatbot-delivered intervention for optimizing social media use and reducing perceived isolation among rural-living LGBTQ+ youth: development, acceptability, usability, satisfaction, and utility. Internet Interv. Dec 2023;34:100668. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Torok M, Han J, McGillivray L, Wong Q, Werner-Seidler A, O'Dea B, et al. The effect of a therapeutic smartphone application on suicidal ideation in young adults: findings from a randomized controlled trial in Australia. PLoS Med. May 2022;19(5):e1003978. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yang YJ, Chung KM. Pilot randomized control trial of an app-based CBT program for reducing anxiety in individuals with ASD without intellectual disability. J Autism Dev Disord. Apr 2023;53(4):1331-1346. [ CrossRef ] [ Medline ]
• Children and parents: media use and attitudes report 2018. Ofcom. Mar 5, 2019. URL: https:/​/www.​ofcom.org.uk/​media-use-and-attitudes/​media-habits-children/​children-and-parents-media-use-and-attitudes-report-2018/​ [accessed 2024-07-12]
• Silver L. Smartphone ownership is growing rapidly around the world, but not always equally. Pew Research Center. Feb 5, 2019. URL: https:/​/www.​pewresearch.org/​global/​2019/​02/​05/​smartphone-ownership-is-growing-rapidly-around-the-world-but-not-always-equally/​ [accessed 2024-07-12]
• Chan A, Kow R, Cheng JK. Adolescents’ perceptions on smartphone applications (apps) for health management. JournalMTM. Aug 2017;6(2):47-55. [ CrossRef ]
• Lipschitz JM, Van Boxtel R, Torous J, Firth J, Lebovitz JG, Burdick KE, et al. Digital mental health interventions for depression: scoping review of user engagement. J Med Internet Res. Oct 14, 2022;24(10):e39204. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ng MM, Firth J, Minen M, Torous J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv. Jul 01, 2019;70(7):538-544. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Oakley-Girvan I, Yunis R, Longmire M, Ouillon JS. What works best to engage participants in mobile app interventions and e-health: a scoping review. Telemed J E Health. Jun 2022;28(6):768-780. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brian RM, Ben-Zeev D. Mobile health (mHealth) for mental health in Asia: objectives, strategies, and limitations. Asian J Psychiatr. Aug 2014;10:96-100. [ CrossRef ] [ Medline ]
• Li H, Lewis C, Chi H, Singleton G, Williams N. Mobile health applications for mental illnesses: an Asian context. Asian J Psychiatr. Dec 2020;54:102209. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Spielmans GI, Gatlin ET, McFall JP. The efficacy of evidence-based psychotherapies versus usual care for youths: controlling confounds in a meta-reanalysis. Psychother Res. Mar 23, 2010;20(2):234-246. [ CrossRef ] [ Medline ]
• Rapport F, Smith J, Hutchinson K, Clay-Williams R, Churruca K, Bierbaum M, et al. Too much theory and not enough practice? The challenge of implementation science application in healthcare practice. J Eval Clin Pract. Dec 15, 2022;28(6):991-1002. [ CrossRef ] [ Medline ]
• Jensen SA, Corralejo SM. Measurement issues: large effect sizes do not mean most people get better - clinical significance and the importance of individual results. Child Adolesc Ment Health. Sep 2017;22(3):163-166. [ CrossRef ] [ Medline ]
• Increasing diversity in research participation: a good practice guide for engaging with underrepresented groups. National Health Service England. Feb 14, 2023. URL: https://www.england.nhs.uk/aac/publication/increasing-diversity-in-research-participation/ [accessed 2023-03-16]
• Bergen N, Labonté R. "Everything is perfect, and we have no problems": detecting and limiting social desirability bias in qualitative research. Qual Health Res. Apr 2020;30(5):783-792. [ CrossRef ] [ Medline ]
• Vogt WP, Johnson RB. Dictionary of Statistics & Methodology: A Nontechnical Guide for the Social Sciences. Thousand Oaks, CA. SAGE Publications; 2011.
• Midgley N, Parkinson S, Holmes J, Stapley E, Eatough V, Target M. "Did I bring it on myself?" An exploratory study of the beliefs that adolescents referred to mental health services have about the causes of their depression. Eur Child Adolesc Psychiatry. Jan 20, 2017;26(1):25-34. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Willis A, Isaacs T, Khunti K. Improving diversity in research and trial participation: the challenges of language. Lancet Public Health. Jul 2021;6(7):e445-e446. [ CrossRef ]
• Ellard-Gray A, Jeffrey NK, Choubak M, Crann SE. Finding the hidden participant: solutions for recruiting hidden, hard-to-reach, and vulnerable populations. Int J Qual Method. Dec 17, 2015;14(5). [ CrossRef ]
• Moltrecht B, Patalay P, Bear HA, Deighton J, Edbrooke-Childs J. A transdiagnostic, emotion regulation app (Eda) for children: design, development, and lessons learned. JMIR Form Res. Jan 19, 2022;6(1):e28300. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bernal G, Jiménez-Chafey MI, Domenech Rodríguez MM. Cultural adaptation of treatments: a resource for considering culture in evidence-based practice. Prof Psychol Res Pract. Aug 2009;40(4):361-368. [ CrossRef ]
• Benish SG, Quintana S, Wampold BE. Culturally adapted psychotherapy and the legitimacy of myth: a direct-comparison meta-analysis. J Couns Psychol. Jul 2011;58(3):279-289. [ CrossRef ] [ Medline ]
• Hall GC, Ibaraki AY, Huang ER, Marti CN, Stice E. A meta-analysis of cultural adaptations of psychological interventions. Behav Ther. Nov 2016;47(6):993-1014. [ CrossRef ] [ Medline ]
• Huey SJJ, Polo AJ. Evidence-based psychosocial treatments for ethnic minority youth. J Clin Child Adolesc Psychol. Jan 03, 2008;37(1):262-301. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ellis DM, Draheim AA, Anderson PL. Culturally adapted digital mental health interventions for ethnic/racial minorities: a systematic review and meta-analysis. J Consult Clin Psychol. Oct 2022;90(10):717-733. [ CrossRef ] [ Medline ]
• Harper Shehadeh M, Heim E, Chowdhary N, Maercker A, Albanese E. Cultural adaptation of minimally guided interventions for common mental disorders: a systematic review and meta-analysis. JMIR Ment Health. Sep 26, 2016;3(3):e44. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Balci S, Spanhel K, Sander LB, Baumeister H. Culturally adapting internet- and mobile-based health promotion interventions might not be worth the effort: a systematic review and meta-analysis. NPJ Digit Med. Mar 23, 2022;5(1):34. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Spanhel K, Balci S, Feldhahn F, Bengel J, Baumeister H, Sander LB. Cultural adaptation of internet- and mobile-based interventions for mental disorders: a systematic review. NPJ Digit Med. Aug 25, 2021;4(1):128. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grist R, Croker A, Denne M, Stallard P. Technology delivered interventions for depression and anxiety in children and adolescents: a systematic review and meta-analysis. Clin Child Fam Psychol Rev. Jun 18, 2019;22(2):147-171. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by T de Azevedo Cardoso, S Ma; submitted 13.05.23; peer-reviewed by P Whelan, I Vainieri, H Bao; comments to author 13.09.23; revised version received 26.09.23; accepted 10.06.24; published 30.07.24.

©Holly Alice Bear, Lara Ayala Nunes, Giovanni Ramos, Tanya Manchanda, Blossom Fernandes, Sophia Chabursky, Sabine Walper, Edward Watkins, Mina Fazel. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 30.07.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.9(1); 2015 Feb

Qualitative systematic reviews: their importance for our understanding of research relevant to pain

This article outlines what a qualitative systematic review is and explores what it can contribute to our understanding of pain. Many of us use evidence of effectiveness for various interventions when working with people in pain. A good systematic review can be invaluable in bringing together research evidence to help inform our practice and help us understand what works. In addition to evidence of effectiveness, understanding how people with pain experience both their pain and their care can help us when we are working with them to provide care that meets their needs. A rigorous qualitative systematic review can also uncover new understandings, often helping illuminate ‘why’ and can help build theory. Such a review can answer the question ‘What is it like to have chronic pain?’ This article presents the different stages of meta-ethnography, which is the most common methodology used for qualitative systematic reviews. It presents evidence from four meta-ethnographies relevant to pain to illustrate the types of findings that can emerge from this approach. It shows how new understandings may emerge and gives an example of chronic musculoskeletal pain being experienced as ‘an adversarial struggle’ across many aspects of the person’s life. This article concludes that evidence from qualitative systematic reviews has its place alongside or integrated with evidence from more quantitative approaches.

Many of us use evidence of effectiveness for various interventions when working with people in pain. A good systematic review can be invaluable in bringing together research evidence to help inform our practice and help us understand what works. In addition to evidence of effectiveness, understanding how people with pain experience both their pain and their care can help us when we are working with them to provide care that meets their needs. A high-quality qualitative systematic review can also uncover new understandings, often helping illuminate ‘why’ and can help build theory. A qualitative systematic review could answer the question ‘What is it like to have chronic non-malignant pain?’

The purpose of this article is to outline what a qualitative systematic review is and explore what it can contribute to our understanding of pain. A qualitative systematic review brings together research on a topic, systematically searching for research evidence from primary qualitative studies and drawing the findings together. There is a debate over whether the search needs to be exhaustive. 1 , 2 Methods for systematic reviews of quantitative research are well established and explicit and have been pioneered through the Cochrane Collaboration. Methods for qualitative systematic reviews have been developed more recently and are still evolving. The Cochrane Collaboration now has a Qualitative and Implementation Methods Group, including a register of protocols, illustrating the recognition of the importance of qualitative research within the Cochrane Collaboration. In November 2013, an editorial described the Cochrane Collaboration’s first publication of a qualitative systematic review as ‘a new milestone’ for Cochrane. 3 Other editorials have raised awareness of qualitative systematic reviews in health. 4

Noblit and Hare 5 were pioneers in the area of synthesising qualitative data. They describe such reviews as aggregated or as interpretative. The aggregated review summarises the data, and Hannes and Pearson 6 provide a worked example of an aggregation approach. Interpretative approaches, as the name suggests, interpret the data, and from that interpretation, new understandings can develop that may lead to development of a theory that helps us to understand or predict behaviour. Types of interpretative qualitative systematic reviews include meta-ethnography, critical interpretative synthesis, realist synthesis and narrative synthesis. More details about these and other approaches can be found in other papers and books. 1 , 5 , 7 – 11 This article will describe one approach, meta-ethnography, as it was identified as the most frequently used approach, 1 and there are some examples using meta-ethnography that focus on pain. A meta-ethnographic approach can be used with a variety of qualitative methodologies, not only ethnography. The data for a meta-ethnography are the concepts or themes described by the authors of the primary studies.

Noblit and Hare 5 outlined the seven steps of a meta-ethnography: (1) getting started, (2) deciding what is relevant, (3) reading the studies, (4) determining how studies are related to each other, (5) translating studies into each other, (6) synthesising translations and (7) expressing the synthesis.

The first three might seem relatively straightforward, although Lee et al. 12 emphasised both the importance and nuances of the reading stage, and Toye et al. 13 discuss the complexities of making quality assessments of qualitative papers and searching for this type of study. You need to understand what data to extract from the papers and how you are going to do this.

You have to first identify what is a concept and what is purely descriptive. Toye et al. 2 describe a process for collaboratively identifying concepts. In determining how studies are related to each other and translating them into each other, the meta-ethnographer compares the concepts found in each study with each other and then groups similar concepts into conceptual themes. Translating studies into each other involves looking at where concepts between studies agree (reciprocal synthesis) and where they do not agree (refutational synthesis). Developing conceptual categories can be challenging as you need to judge the extent to which a concept from one study adequately reflects concepts from other studies and choose one that seems to fit best. This is discussed in more detail in Toye et al. 2 , 13

To synthesise the translation, a line of argument is then developed from the conceptual categories. How the concepts group and relate to each other are developed. This provides an overall interpretation of the findings, ensuring this is grounded in the data from the primary studies. You are aiming to explain, and new concepts and understandings may emerge, which can then go on to underpin development of theory. For example, a qualitative systematic review that explored medicine taking found that ‘resistance’ was a new concept, revealed through meta-ethnography, and this helped understanding of lay responses to medicine taking. 1 Hannes and Macaitis, 14 in a review of published papers, reported that over time, authors have become more transparent about searching and critical appraisal, but that the synthesis element of reviews is often not well described. Being transparent about decisions that are interpretative has its own challenges. Working collaboratively to challenge interpretations and assumptions can be helpful. 2 , 12 The next section will use examples of qualitative systematic reviews from the pain field to illuminate what this type of review can contribute to our understanding of pain.

What can a qualitative systematic review contribute to the field of pain – some examples

Toye et al. 2 , 15 undertook a meta-ethnography to look at patients’ experiences of chronic non-malignant musculoskeletal pain. At the time of this research, no other qualitative systematic reviews had been published in this area. Their review included 77 papers reporting 60 individual studies, resulting from searches of six electronic bibliographic databases (MEDLINE, EMBASE, CINAHL, PsycINFO, AMED and HMIC) from inception until February 2012 and hand-searching key journals from 2001 to 2012.

They developed a new concept which they identified as an ‘adversarial struggle’. This struggle took place across five main dimensions: (1) there was a struggle to affirm themselves, where there was a tension between the ‘real me’ (without pain) and ‘not real me’ (me with pain). (2) The present and future were often unpredictable, and construction of time was altered and they struggled to reconstruct themselves in time. (3) People struggled to find an acceptable explanation for their pain and suffering. (4) There was a struggle to negotiate the healthcare system and (5) a struggle for pain to be seen as legitimate, including the need to be believed, and a struggle to know whether to show or hide their pain. Some people were able to move forward with pain. They saw their body as more integrated, they re-defined what was normal, they told people about their pain, they were part of a community of people with pain and they felt more expert on how their pain affected them and what they could do about it.

So, this meta-ethnography highlighted the adversarial nature of having chronic musculoskeletal pain and how this struggle pervaded many different areas of their life. It also illustrated how by showing patients their pain is understood and being alongside the person in pain, they can start to move forward. A short film based on the 77 papers in this meta-ethnography has been made and is available on YouTube. 16 This film was made as an attempt to disseminate the findings of a meta-ethnography in a way that is accessible to a range of people.

Snelgrove and Liossi 17 undertook a meta-ethnography of qualitative research in chronic low back pain (CLBP) using meta-ethnography. They included 33 papers of 28 studies published between 2000 and 2012. They identified three overarching themes of (1) the impact of CLBP on self, (2) relationships with others (health professionals and family and friends) and (3) coping with CLBP. They found that very few successful coping strategies were reported. Like Toye et al., 2 , 15 they also reported disruption to self, distancing their valued self from their painful self, legitimising pain, the struggle to manage daily living and the importance of social relationships alongside negotiation of their care in the health system.

MacNeela et al. 18 also undertook a meta-ethnography of experiences of CLBP. They included 38 articles published between 1994 and 2012 representing 28 studies. They identified four themes: (1) the undermining influence of pain, (2) the disempowering impact on all levels, (3) unsatisfying relationships with healthcare professionals and (4) learning to live with the pain. They reported the findings being dominated by ‘wide-ranging distress and loss’. They discussed the disempowering consequences of pain and a search for help. However, they also highlighted self-determination and resilience and suggested these could offer ‘pathways to endurance’. They emphasised self-management and adaptation, which resonates with the moving forward category reported by Toye et al. 2 , 15

Froud et al. 19 looked at the impact of low back pain on people’s lives. They describe their approach as meta-ethnographic and meta-narrative. They included 49 papers of about 42 studies from inception of databases searched until July 2011. They described five themes: activities, relationships, work, stigma and changing outlook, which they derived from ‘participant-level data’. They described their findings as showing patients wanted to be believed. They highlighted the importance of social factors when developing relevant outcome measures. There are other examples of qualitative systematic reviews relevant to pain. 20 – 23

Different qualitative systematic reviews on a similar subject may come up with overlapping but also some different findings. This could be, for example, because different search periods or different inclusion criteria are used, so different primary studies may be included in different reviews. In addition, undertaking a qualitative systematic review requires researchers to interpret concepts. This interpretation does not need to be a limitation. For example, to ensure rigour and transparency, Toye et al. 24 report a process of collaborative interpretation of concepts among a team of experienced qualitative researchers to ensure individual interpretations were challenged and remained grounded in the original studies. They also published a detailed audit trail of the processes and decisions made. 2 Campbell et al. 1 argue ‘Meta-ethnography is a highly interpretative method requiring considerable immersion in the individual studies to achieve a synthesis. It places substantial demands upon the synthesiser and requires a high degree of qualitative research skill’. It is important to be able to think conceptually when undertaking a meta-ethnography, and it can be a time-consuming process. However, the ability of a meta-ethnography to synthesise a large number of primary research studies, generate new conceptual understandings and thus increase our understanding of patients’ experiences of pain makes it a very useful resource for our evidence-based practice.

The way forward

A register of qualitative systematic reviews would be useful for researchers and clinicians, so there was a clear way of identifying existing qualitative reviews or reviews that are planned or underway. The Cochrane Collaboration does now have a register for protocols of qualitative systematic reviews being undertaken under the aegis of the Cochrane Qualitative and Implementation Methods Group. It would help those wanting to undertake qualitative systematic reviews if reviews that were underway were registered and described more clearly to prevent duplication of effort, for example, using ‘qualitative systematic review’ and the methodological approach used (such as meta-ethnography) in the title and/or abstract. The Toye et al. 2 protocol 25 was accessible on the National Institutes for Health website from 2010. The Snelgrove and Liossi 17 study was done without external funding, so it would be difficult to pick up that it was underway. The MacNeela et al. 18 study was listed on the Irish Research Council for the Humanities and Social Sciences under their Research Development Initiative 2008–2009, but was described as ‘Motivation and Beliefs among People Experiencing Chronic Low Back Pain’, so it was not clearly identified at that stage as a qualitative systematic review. Finally, the Froud et al. 19 award details 26 do not mention qualitative systematic reviews or meta-ethnography. This highlights the difficulty of finding some of these reviews and the importance of a register of both completed and ongoing reviews.

This article has argued that qualitative systematic reviews have their place alongside or integrated with more quantitative approaches. There is an increasing body of evidence from qualitative systematic reviews. They can synthesise primary research, and this can be helpful for the busy practitioner. The methods for these approaches are still developing, and attention to rigour at each stage is crucial. It is important that each stage of the synthesis is reported transparently and that the researchers’ stance is clearly reported. 27 Meta-ethnographies published over the last year 2 , 15 , 17 – 19 have drawn together a wide range of primary studies and shown that people’s lives can be markedly changed by their pain across multiple dimensions of their life.

Declaration of Conflicting Interests: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The authors received no financial support for the research, authorship, and/or publication of this article.