methodology discussion paper

How to Write the Discussion Section of a Research Paper

The discussion section of a research paper analyzes and interprets the findings, provides context, compares them with previous studies, identifies limitations, and suggests future research directions.

Updated on September 15, 2023

Structure your discussion section right, and you’ll be cited more often while doing a greater service to the scientific community. So, what actually goes into the discussion section? And how do you write it?

The discussion section of your research paper is where you let the reader know how your study is positioned in the literature, what to take away from your paper, and how your work helps them. It can also include your conclusions and suggestions for future studies.

First, we’ll define all the parts of your discussion paper, and then look into how to write a strong, effective discussion section for your paper or manuscript.

Discussion section: what is it, what it does

The discussion section comes later in your paper, following the introduction, methods, and results. The discussion sets up your study’s conclusions. Its main goals are to present, interpret, and provide a context for your results.

What is it?

The discussion section provides an analysis and interpretation of the findings, compares them with previous studies, identifies limitations, and suggests future directions for research.

This section combines information from the preceding parts of your paper into a coherent story. By this point, the reader already knows why you did your study (introduction), how you did it (methods), and what happened (results). In the discussion, you’ll help the reader connect the ideas from these sections.

Why is it necessary?

The discussion provides context and interpretations for the results. It also answers the questions posed in the introduction. While the results section describes your findings, the discussion explains what they say. This is also where you can describe the impact or implications of your research.

Adds context for your results

Most research studies aim to answer a question, replicate a finding, or address limitations in the literature. These goals are first described in the introduction. However, in the discussion section, the author can refer back to them to explain how the study's objective was achieved. 

Shows what your results actually mean and real-world implications

The discussion can also describe the effect of your findings on research or practice. How are your results significant for readers, other researchers, or policymakers?

What to include in your discussion (in the correct order)

A complete and effective discussion section should at least touch on the points described below.

Summary of key findings

The discussion should begin with a brief factual summary of the results. Concisely overview the main results you obtained.

Begin with key findings with supporting evidence

Your results section described a list of findings, but what message do they send when you look at them all together?

Your findings were detailed in the results section, so there’s no need to repeat them here, but do provide at least a few highlights. This will help refresh the reader’s memory and help them focus on the big picture.

Read the first paragraph of the discussion section in this article (PDF) for an example of how to start this part of your paper. Notice how the authors break down their results and follow each description sentence with an explanation of why each finding is relevant. 

State clearly and concisely

Following a clear and direct writing style is especially important in the discussion section. After all, this is where you will make some of the most impactful points in your paper. While the results section often contains technical vocabulary, such as statistical terms, the discussion section lets you describe your findings more clearly. 

Interpretation of results

Once you’ve given your reader an overview of your results, you need to interpret those results. In other words, what do your results mean? Discuss the findings’ implications and significance in relation to your research question or hypothesis.

Analyze and interpret your findings

Look into your findings and explore what’s behind them or what may have caused them. If your introduction cited theories or studies that could explain your findings, use these sources as a basis to discuss your results.

For example, look at the second paragraph in the discussion section of this article on waggling honey bees. Here, the authors explore their results based on information from the literature.

Unexpected or contradictory results

Sometimes, your findings are not what you expect. Here’s where you describe this and try to find a reason for it. Could it be because of the method you used? Does it have something to do with the variables analyzed? Comparing your methods with those of other similar studies can help with this task.

Context and comparison with previous work

Refer to related studies to place your research in a larger context and the literature. Compare and contrast your findings with existing literature, highlighting similarities, differences, and/or contradictions.

How your work compares or contrasts with previous work

Studies with similar findings to yours can be cited to show the strength of your findings. Information from these studies can also be used to help explain your results. Differences between your findings and others in the literature can also be discussed here. 

How to divide this section into subsections

If you have more than one objective in your study or many key findings, you can dedicate a separate section to each of these. Here’s an example of this approach. You can see that the discussion section is divided into topics and even has a separate heading for each of them. 

Limitations

Many journals require you to include the limitations of your study in the discussion. Even if they don’t, there are good reasons to mention these in your paper.

Why limitations don’t have a negative connotation

A study’s limitations are points to be improved upon in future research. While some of these may be flaws in your method, many may be due to factors you couldn’t predict.

Examples include time constraints or small sample sizes. Pointing this out will help future researchers avoid or address these issues. This part of the discussion can also include any attempts you have made to reduce the impact of these limitations, as in this study .

How limitations add to a researcher's credibility

Pointing out the limitations of your study demonstrates transparency. It also shows that you know your methods well and can conduct a critical assessment of them.  

Implications and significance

The final paragraph of the discussion section should contain the take-home messages for your study. It can also cite the “strong points” of your study, to contrast with the limitations section.

Restate your hypothesis

Remind the reader what your hypothesis was before you conducted the study. 

How was it proven or disproven?

Identify your main findings and describe how they relate to your hypothesis.

How your results contribute to the literature

Were you able to answer your research question? Or address a gap in the literature?

Future implications of your research

Describe the impact that your results may have on the topic of study. Your results may show, for instance, that there are still limitations in the literature for future studies to address. There may be a need for studies that extend your findings in a specific way. You also may need additional research to corroborate your findings. 

Sample discussion section

This fictitious example covers all the aspects discussed above. Your actual discussion section will probably be much longer, but you can read this to get an idea of everything your discussion should cover.

Our results showed that the presence of cats in a household is associated with higher levels of perceived happiness by its human occupants. These findings support our hypothesis and demonstrate the association between pet ownership and well-being. 

The present findings align with those of Bao and Schreer (2016) and Hardie et al. (2023), who observed greater life satisfaction in pet owners relative to non-owners. Although the present study did not directly evaluate life satisfaction, this factor may explain the association between happiness and cat ownership observed in our sample.

Our findings must be interpreted in light of some limitations, such as the focus on cat ownership only rather than pets as a whole. This may limit the generalizability of our results.

Nevertheless, this study had several strengths. These include its strict exclusion criteria and use of a standardized assessment instrument to investigate the relationships between pets and owners. These attributes bolster the accuracy of our results and reduce the influence of confounding factors, increasing the strength of our conclusions. Future studies may examine the factors that mediate the association between pet ownership and happiness to better comprehend this phenomenon.

This brief discussion begins with a quick summary of the results and hypothesis. The next paragraph cites previous research and compares its findings to those of this study. Information from previous studies is also used to help interpret the findings. After discussing the results of the study, some limitations are pointed out. The paper also explains why these limitations may influence the interpretation of results. Then, final conclusions are drawn based on the study, and directions for future research are suggested.

How to make your discussion flow naturally

If you find writing in scientific English challenging, the discussion and conclusions are often the hardest parts of the paper to write. That’s because you’re not just listing up studies, methods, and outcomes. You’re actually expressing your thoughts and interpretations in words.

• How formal should it be?
• What words should you use, or not use?
• How do you meet strict word limits, or make it longer and more informative?

Always give it your best, but sometimes a helping hand can, well, help. Getting a professional edit can help clarify your work’s importance while improving the English used to explain it. When readers know the value of your work, they’ll cite it. We’ll assign your study to an expert editor knowledgeable in your area of research. Their work will clarify your discussion, helping it to tell your story. Find out more about AJE Editing.

Adam Goulston, PsyD, MS, MBA, MISD, ELS

Science Marketing Consultant

See our "Privacy Policy"

Ensure your structure and ideas are consistent and clearly communicated

Pair your Premium Editing with our add-on service Presubmission Review for an overall assessment of your manuscript.

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 8. The Discussion
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Applying Critical Thinking
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The purpose of the discussion section is to interpret and describe the significance of your findings in relation to what was already known about the research problem being investigated and to explain any new understanding or insights that emerged as a result of your research. The discussion will always connect to the introduction by way of the research questions or hypotheses you posed and the literature you reviewed, but the discussion does not simply repeat or rearrange the first parts of your paper; the discussion clearly explains how your study advanced the reader's understanding of the research problem from where you left them at the end of your review of prior research.

Annesley, Thomas M. “The Discussion Section: Your Closing Argument.” Clinical Chemistry 56 (November 2010): 1671-1674; Peacock, Matthew. “Communicative Moves in the Discussion Section of Research Articles.” System 30 (December 2002): 479-497.

Importance of a Good Discussion

The discussion section is often considered the most important part of your research paper because it:

• Most effectively demonstrates your ability as a researcher to think critically about an issue, to develop creative solutions to problems based upon a logical synthesis of the findings, and to formulate a deeper, more profound understanding of the research problem under investigation;
• Presents the underlying meaning of your research, notes possible implications in other areas of study, and explores possible improvements that can be made in order to further develop the concerns of your research;
• Highlights the importance of your study and how it can contribute to understanding the research problem within the field of study;
• Presents how the findings from your study revealed and helped fill gaps in the literature that had not been previously exposed or adequately described; and,
• Engages the reader in thinking critically about issues based on an evidence-based interpretation of findings; it is not governed strictly by objective reporting of information.

Annesley Thomas M. “The Discussion Section: Your Closing Argument.” Clinical Chemistry 56 (November 2010): 1671-1674; Bitchener, John and Helen Basturkmen. “Perceptions of the Difficulties of Postgraduate L2 Thesis Students Writing the Discussion Section.” Journal of English for Academic Purposes 5 (January 2006): 4-18; Kretchmer, Paul. Fourteen Steps to Writing an Effective Discussion Section. San Francisco Edit, 2003-2008.

Structure and Writing Style

I.  General Rules

These are the general rules you should adopt when composing your discussion of the results :

• Do not be verbose or repetitive; be concise and make your points clearly
• Avoid the use of jargon or undefined technical language
• Follow a logical stream of thought; in general, interpret and discuss the significance of your findings in the same sequence you described them in your results section [a notable exception is to begin by highlighting an unexpected result or a finding that can grab the reader's attention]
• Use the present verb tense, especially for established facts; however, refer to specific works or prior studies in the past tense
• If needed, use subheadings to help organize your discussion or to categorize your interpretations into themes

II.  The Content

The content of the discussion section of your paper most often includes :

• Explanation of results : Comment on whether or not the results were expected for each set of findings; go into greater depth to explain findings that were unexpected or especially profound. If appropriate, note any unusual or unanticipated patterns or trends that emerged from your results and explain their meaning in relation to the research problem.
• References to previous research : Either compare your results with the findings from other studies or use the studies to support a claim. This can include re-visiting key sources already cited in your literature review section, or, save them to cite later in the discussion section if they are more important to compare with your results instead of being a part of the general literature review of prior research used to provide context and background information. Note that you can make this decision to highlight specific studies after you have begun writing the discussion section.
• Deduction : A claim for how the results can be applied more generally. For example, describing lessons learned, proposing recommendations that can help improve a situation, or highlighting best practices.
• Hypothesis : A more general claim or possible conclusion arising from the results [which may be proved or disproved in subsequent research]. This can be framed as new research questions that emerged as a consequence of your analysis.

III.  Organization and Structure

Keep the following sequential points in mind as you organize and write the discussion section of your paper:

• Think of your discussion as an inverted pyramid. Organize the discussion from the general to the specific, linking your findings to the literature, then to theory, then to practice [if appropriate].
• Use the same key terms, narrative style, and verb tense [present] that you used when describing the research problem in your introduction.
• Begin by briefly re-stating the research problem you were investigating and answer all of the research questions underpinning the problem that you posed in the introduction.
• Describe the patterns, principles, and relationships shown by each major findings and place them in proper perspective. The sequence of this information is important; first state the answer, then the relevant results, then cite the work of others. If appropriate, refer the reader to a figure or table to help enhance the interpretation of the data [either within the text or as an appendix].
• Regardless of where it's mentioned, a good discussion section includes analysis of any unexpected findings. This part of the discussion should begin with a description of the unanticipated finding, followed by a brief interpretation as to why you believe it appeared and, if necessary, its possible significance in relation to the overall study. If more than one unexpected finding emerged during the study, describe each of them in the order they appeared as you gathered or analyzed the data. As noted, the exception to discussing findings in the same order you described them in the results section would be to begin by highlighting the implications of a particularly unexpected or significant finding that emerged from the study, followed by a discussion of the remaining findings.
• Before concluding the discussion, identify potential limitations and weaknesses if you do not plan to do so in the conclusion of the paper. Comment on their relative importance in relation to your overall interpretation of the results and, if necessary, note how they may affect the validity of your findings. Avoid using an apologetic tone; however, be honest and self-critical [e.g., in retrospect, had you included a particular question in a survey instrument, additional data could have been revealed].
• The discussion section should end with a concise summary of the principal implications of the findings regardless of their significance. Give a brief explanation about why you believe the findings and conclusions of your study are important and how they support broader knowledge or understanding of the research problem. This can be followed by any recommendations for further research. However, do not offer recommendations which could have been easily addressed within the study. This would demonstrate to the reader that you have inadequately examined and interpreted the data.

IV.  Overall Objectives

The objectives of your discussion section should include the following: I.  Reiterate the Research Problem/State the Major Findings

Briefly reiterate the research problem or problems you are investigating and the methods you used to investigate them, then move quickly to describe the major findings of the study. You should write a direct, declarative, and succinct proclamation of the study results, usually in one paragraph.

II.  Explain the Meaning of the Findings and Why They are Important

No one has thought as long and hard about your study as you have. Systematically explain the underlying meaning of your findings and state why you believe they are significant. After reading the discussion section, you want the reader to think critically about the results and why they are important. You don’t want to force the reader to go through the paper multiple times to figure out what it all means. If applicable, begin this part of the section by repeating what you consider to be your most significant or unanticipated finding first, then systematically review each finding. Otherwise, follow the general order you reported the findings presented in the results section.

III.  Relate the Findings to Similar Studies

No study in the social sciences is so novel or possesses such a restricted focus that it has absolutely no relation to previously published research. The discussion section should relate your results to those found in other studies, particularly if questions raised from prior studies served as the motivation for your research. This is important because comparing and contrasting the findings of other studies helps to support the overall importance of your results and it highlights how and in what ways your study differs from other research about the topic. Note that any significant or unanticipated finding is often because there was no prior research to indicate the finding could occur. If there is prior research to indicate this, you need to explain why it was significant or unanticipated. IV.  Consider Alternative Explanations of the Findings

It is important to remember that the purpose of research in the social sciences is to discover and not to prove . When writing the discussion section, you should carefully consider all possible explanations for the study results, rather than just those that fit your hypothesis or prior assumptions and biases. This is especially important when describing the discovery of significant or unanticipated findings.

V.  Acknowledge the Study’s Limitations

It is far better for you to identify and acknowledge your study’s limitations than to have them pointed out by your professor! Note any unanswered questions or issues your study could not address and describe the generalizability of your results to other situations. If a limitation is applicable to the method chosen to gather information, then describe in detail the problems you encountered and why. VI.  Make Suggestions for Further Research

You may choose to conclude the discussion section by making suggestions for further research [as opposed to offering suggestions in the conclusion of your paper]. Although your study can offer important insights about the research problem, this is where you can address other questions related to the problem that remain unanswered or highlight hidden issues that were revealed as a result of conducting your research. You should frame your suggestions by linking the need for further research to the limitations of your study [e.g., in future studies, the survey instrument should include more questions that ask..."] or linking to critical issues revealed from the data that were not considered initially in your research.

NOTE: Besides the literature review section, the preponderance of references to sources is usually found in the discussion section . A few historical references may be helpful for perspective, but most of the references should be relatively recent and included to aid in the interpretation of your results, to support the significance of a finding, and/or to place a finding within a particular context. If a study that you cited does not support your findings, don't ignore it--clearly explain why your research findings differ from theirs.

V.  Problems to Avoid

• Do not waste time restating your results . Should you need to remind the reader of a finding to be discussed, use "bridge sentences" that relate the result to the interpretation. An example would be: “In the case of determining available housing to single women with children in rural areas of Texas, the findings suggest that access to good schools is important...," then move on to further explaining this finding and its implications.
• As noted, recommendations for further research can be included in either the discussion or conclusion of your paper, but do not repeat your recommendations in the both sections. Think about the overall narrative flow of your paper to determine where best to locate this information. However, if your findings raise a lot of new questions or issues, consider including suggestions for further research in the discussion section.
• Do not introduce new results in the discussion section. Be wary of mistaking the reiteration of a specific finding for an interpretation because it may confuse the reader. The description of findings [results section] and the interpretation of their significance [discussion section] should be distinct parts of your paper. If you choose to combine the results section and the discussion section into a single narrative, you must be clear in how you report the information discovered and your own interpretation of each finding. This approach is not recommended if you lack experience writing college-level research papers.
• Use of the first person pronoun is generally acceptable. Using first person singular pronouns can help emphasize a point or illustrate a contrasting finding. However, keep in mind that too much use of the first person can actually distract the reader from the main points [i.e., I know you're telling me this--just tell me!].

Analyzing vs. Summarizing. Department of English Writing Guide. George Mason University; Discussion. The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College; Hess, Dean R. "How to Write an Effective Discussion." Respiratory Care 49 (October 2004); Kretchmer, Paul. Fourteen Steps to Writing to Writing an Effective Discussion Section. San Francisco Edit, 2003-2008; The Lab Report. University College Writing Centre. University of Toronto; Sauaia, A. et al. "The Anatomy of an Article: The Discussion Section: "How Does the Article I Read Today Change What I Will Recommend to my Patients Tomorrow?” The Journal of Trauma and Acute Care Surgery 74 (June 2013): 1599-1602; Research Limitations & Future Research . Lund Research Ltd., 2012; Summary: Using it Wisely. The Writing Center. University of North Carolina; Schafer, Mickey S. Writing the Discussion. Writing in Psychology course syllabus. University of Florida; Yellin, Linda L. A Sociology Writer's Guide . Boston, MA: Allyn and Bacon, 2009.

Writing Tip

Don’t Over-Interpret the Results!

Interpretation is a subjective exercise. As such, you should always approach the selection and interpretation of your findings introspectively and to think critically about the possibility of judgmental biases unintentionally entering into discussions about the significance of your work. With this in mind, be careful that you do not read more into the findings than can be supported by the evidence you have gathered. Remember that the data are the data: nothing more, nothing less.

MacCoun, Robert J. "Biases in the Interpretation and Use of Research Results." Annual Review of Psychology 49 (February 1998): 259-287; Ward, Paulet al, editors. The Oxford Handbook of Expertise . Oxford, UK: Oxford University Press, 2018.

Another Writing Tip

Don't Write Two Results Sections!

One of the most common mistakes that you can make when discussing the results of your study is to present a superficial interpretation of the findings that more or less re-states the results section of your paper. Obviously, you must refer to your results when discussing them, but focus on the interpretation of those results and their significance in relation to the research problem, not the data itself.

Azar, Beth. "Discussing Your Findings."  American Psychological Association gradPSYCH Magazine (January 2006).

Yet Another Writing Tip

Avoid Unwarranted Speculation!

The discussion section should remain focused on the findings of your study. For example, if the purpose of your research was to measure the impact of foreign aid on increasing access to education among disadvantaged children in Bangladesh, it would not be appropriate to speculate about how your findings might apply to populations in other countries without drawing from existing studies to support your claim or if analysis of other countries was not a part of your original research design. If you feel compelled to speculate, do so in the form of describing possible implications or explaining possible impacts. Be certain that you clearly identify your comments as speculation or as a suggestion for where further research is needed. Sometimes your professor will encourage you to expand your discussion of the results in this way, while others don’t care what your opinion is beyond your effort to interpret the data in relation to the research problem.

• << Previous: Using Non-Textual Elements
• Next: Limitations of the Study >>
• Last Updated: Sep 4, 2024 9:40 AM
• URL: https://libguides.usc.edu/writingguide

• Langson Library
• Science Library
• Grunigen Medical Library
• Law Library
• Connect From Off-Campus
• Accessibility
• Gateway Study Center

Email this link

Writing a scientific paper.

• Writing a lab report
• INTRODUCTION

Writing a "good" discussion section

"discussion and conclusions checklist" from: how to write a good scientific paper. chris a. mack. spie. 2018., peer review.

• LITERATURE CITED
• Bibliography of guides to scientific writing and presenting
• Presentations
• Lab Report Writing Guides on the Web

This is is usually the hardest section to write. You are trying to bring out the true meaning of your data without being too long. Do not use words to conceal your facts or reasoning. Also do not repeat your results, this is a discussion.

• Present principles, relationships and generalizations shown by the results
• Point out exceptions or lack of correlations. Define why you think this is so.
• Show how your results agree or disagree with previously published works
• Discuss the theoretical implications of your work as well as practical applications
• State your conclusions clearly. Summarize your evidence for each conclusion.
• Discuss the significance of the results
•  Evidence does not explain itself; the results must be presented and then explained.
• Typical stages in the discussion: summarizing the results, discussing whether results are expected or unexpected, comparing these results to previous work, interpreting and explaining the results (often by comparison to a theory or model), and hypothesizing about their generality.
• Discuss any problems or shortcomings encountered during the course of the work.
• Discuss possible alternate explanations for the results.
• Avoid: presenting results that are never discussed; presenting discussion that does not relate to any of the results; presenting results and discussion in chronological order rather than logical order; ignoring results that do not support the conclusions; drawing conclusions from results without logical arguments to back them up. 

CONCLUSIONS

• Provide a very brief summary of the Results and Discussion.
• Emphasize the implications of the findings, explaining how the work is significant and providing the key message(s) the author wishes to convey.
• Provide the most general claims that can be supported by the evidence.
• Provide a future perspective on the work.
• Avoid: repeating the abstract; repeating background information from the Introduction; introducing new evidence or new arguments not found in the Results and Discussion; repeating the arguments made in the Results and Discussion; failing to address all of the research questions set out in the Introduction. 

WHAT HAPPENS AFTER I COMPLETE MY PAPER?

 The peer review process is the quality control step in the publication of ideas.  Papers that are submitted to a journal for publication are sent out to several scientists (peers) who look carefully at the paper to see if it is "good science".  These reviewers then recommend to the editor of a journal whether or not a paper should be published. Most journals have publication guidelines. Ask for them and follow them exactly.    Peer reviewers examine the soundness of the materials and methods section.  Are the materials and methods used written clearly enough for another scientist to reproduce the experiment?  Other areas they look at are: originality of research, significance of research question studied, soundness of the discussion and interpretation, correct spelling and use of technical terms, and length of the article.

• << Previous: RESULTS
• Next: LITERATURE CITED >>
• Last Updated: Aug 4, 2023 9:33 AM
• URL: https://guides.lib.uci.edu/scientificwriting

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

• Affiliate Program

• UNITED STATES
• 台灣 (TAIWAN)
• TÜRKIYE (TURKEY)
• Academic Editing Services
• - Research Paper
• - Journal Manuscript
• - Dissertation
• - College & University Assignments
• Admissions Editing Services
• - Application Essay
• - Personal Statement
• - Recommendation Letter
• - Cover Letter
• - CV/Resume
• Business Editing Services
• - Business Documents
• - Report & Brochure
• - Website & Blog
• Writer Editing Services
• - Script & Screenplay
• Our Editors
• Client Reviews
• Editing & Proofreading Prices
• Wordvice Points
• Partner Discount
• Plagiarism Checker
• APA Citation Generator
• MLA Citation Generator
• Chicago Citation Generator
• Vancouver Citation Generator
• - APA Style
• - MLA Style
• - Chicago Style
• - Vancouver Style
• Writing & Editing Guide
• Academic Resources
• Admissions Resources

How to Write a Discussion Section for a Research Paper

We’ve talked about several useful writing tips that authors should consider while drafting or editing their research papers. In particular, we’ve focused on  figures and legends , as well as the Introduction ,  Methods , and  Results . Now that we’ve addressed the more technical portions of your journal manuscript, let’s turn to the analytical segments of your research article. In this article, we’ll provide tips on how to write a strong Discussion section that best portrays the significance of your research contributions.

What is the Discussion section of a research paper?

In a nutshell,  your Discussion fulfills the promise you made to readers in your Introduction . At the beginning of your paper, you tell us why we should care about your research. You then guide us through a series of intricate images and graphs that capture all the relevant data you collected during your research. We may be dazzled and impressed at first, but none of that matters if you deliver an anti-climactic conclusion in the Discussion section!

Are you feeling pressured? Don’t worry. To be honest, you will edit the Discussion section of your manuscript numerous times. After all, in as little as one to two paragraphs ( Nature ‘s suggestion  based on their 3,000-word main body text limit), you have to explain how your research moves us from point A (issues you raise in the Introduction) to point B (our new understanding of these matters). You must also recommend how we might get to point C (i.e., identify what you think is the next direction for research in this field). That’s a lot to say in two paragraphs!

So, how do you do that? Let’s take a closer look.

What should I include in the Discussion section?

As we stated above, the goal of your Discussion section is to  answer the questions you raise in your Introduction by using the results you collected during your research . The content you include in the Discussions segment should include the following information:

• Remind us why we should be interested in this research project.
• Describe the nature of the knowledge gap you were trying to fill using the results of your study.
• Don’t repeat your Introduction. Instead, focus on why  this  particular study was needed to fill the gap you noticed and why that gap needed filling in the first place.
• Mainly, you want to remind us of how your research will increase our knowledge base and inspire others to conduct further research.
• Clearly tell us what that piece of missing knowledge was.
• Answer each of the questions you asked in your Introduction and explain how your results support those conclusions.
• Make sure to factor in all results relevant to the questions (even if those results were not statistically significant).
• Focus on the significance of the most noteworthy results.
• If conflicting inferences can be drawn from your results, evaluate the merits of all of them.
• Don’t rehash what you said earlier in the Results section. Rather, discuss your findings in the context of answering your hypothesis. Instead of making statements like “[The first result] was this…,” say, “[The first result] suggests [conclusion].”
• Do your conclusions line up with existing literature?
• Discuss whether your findings agree with current knowledge and expectations.
• Keep in mind good persuasive argument skills, such as explaining the strengths of your arguments and highlighting the weaknesses of contrary opinions.
• If you discovered something unexpected, offer reasons. If your conclusions aren’t aligned with current literature, explain.
• Address any limitations of your study and how relevant they are to interpreting your results and validating your findings.
• Make sure to acknowledge any weaknesses in your conclusions and suggest room for further research concerning that aspect of your analysis.
• Make sure your suggestions aren’t ones that should have been conducted during your research! Doing so might raise questions about your initial research design and protocols.
• Similarly, maintain a critical but unapologetic tone. You want to instill confidence in your readers that you have thoroughly examined your results and have objectively assessed them in a way that would benefit the scientific community’s desire to expand our knowledge base.
• Recommend next steps.
• Your suggestions should inspire other researchers to conduct follow-up studies to build upon the knowledge you have shared with them.
• Keep the list short (no more than two).

How to Write the Discussion Section

The above list of what to include in the Discussion section gives an overall idea of what you need to focus on throughout the section. Below are some tips and general suggestions about the technical aspects of writing and organization that you might find useful as you draft or revise the contents we’ve outlined above.

Technical writing elements

• Embrace active voice because it eliminates the awkward phrasing and wordiness that accompanies passive voice.
• Use the present tense, which should also be employed in the Introduction.
• Sprinkle with first person pronouns if needed, but generally, avoid it. We want to focus on your findings.
• Maintain an objective and analytical tone.

Discussion section organization

• Keep the same flow across the Results, Methods, and Discussion sections.
• We develop a rhythm as we read and parallel structures facilitate our comprehension. When you organize information the same way in each of these related parts of your journal manuscript, we can quickly see how a certain result was interpreted and quickly verify the particular methods used to produce that result.
• Notice how using parallel structure will eliminate extra narration in the Discussion part since we can anticipate the flow of your ideas based on what we read in the Results segment. Reducing wordiness is important when you only have a few paragraphs to devote to the Discussion section!
• Within each subpart of a Discussion, the information should flow as follows: (A) conclusion first, (B) relevant results and how they relate to that conclusion and (C) relevant literature.
• End with a concise summary explaining the big-picture impact of your study on our understanding of the subject matter. At the beginning of your Discussion section, you stated why  this  particular study was needed to fill the gap you noticed and why that gap needed filling in the first place. Now, it is time to end with “how your research filled that gap.”

Discussion Part 1: Summarizing Key Findings

Begin the Discussion section by restating your  statement of the problem  and briefly summarizing the major results. Do not simply repeat your findings. Rather, try to create a concise statement of the main results that directly answer the central research question that you stated in the Introduction section . This content should not be longer than one paragraph in length.

Many researchers struggle with understanding the precise differences between a Discussion section and a Results section . The most important thing to remember here is that your Discussion section should subjectively evaluate the findings presented in the Results section, and in relatively the same order. Keep these sections distinct by making sure that you do not repeat the findings without providing an interpretation.

Phrase examples: Summarizing the results

• The findings indicate that …
• These results suggest a correlation between A and B …
• The data present here suggest that …
• An interpretation of the findings reveals a connection between…

Discussion Part 2: Interpreting the Findings

What do the results mean? It may seem obvious to you, but simply looking at the figures in the Results section will not necessarily convey to readers the importance of the findings in answering your research questions.

The exact structure of interpretations depends on the type of research being conducted. Here are some common approaches to interpreting data:

• Identifying correlations and relationships in the findings
• Explaining whether the results confirm or undermine your research hypothesis
• Giving the findings context within the history of similar research studies
• Discussing unexpected results and analyzing their significance to your study or general research
• Offering alternative explanations and arguing for your position

Organize the Discussion section around key arguments, themes, hypotheses, or research questions or problems. Again, make sure to follow the same order as you did in the Results section.

Discussion Part 3: Discussing the Implications

In addition to providing your own interpretations, show how your results fit into the wider scholarly literature you surveyed in the  literature review section. This section is called the implications of the study . Show where and how these results fit into existing knowledge, what additional insights they contribute, and any possible consequences that might arise from this knowledge, both in the specific research topic and in the wider scientific domain.

Questions to ask yourself when dealing with potential implications:

• Do your findings fall in line with existing theories, or do they challenge these theories or findings? What new information do they contribute to the literature, if any? How exactly do these findings impact or conflict with existing theories or models?
• What are the practical implications on actual subjects or demographics?
• What are the methodological implications for similar studies conducted either in the past or future?

Your purpose in giving the implications is to spell out exactly what your study has contributed and why researchers and other readers should be interested.

Phrase examples: Discussing the implications of the research

• These results confirm the existing evidence in X studies…
• The results are not in line with the foregoing theory that…
• This experiment provides new insights into the connection between…
• These findings present a more nuanced understanding of…
• While previous studies have focused on X, these results demonstrate that Y.

Step 4: Acknowledging the limitations

All research has study limitations of one sort or another. Acknowledging limitations in methodology or approach helps strengthen your credibility as a researcher. Study limitations are not simply a list of mistakes made in the study. Rather, limitations help provide a more detailed picture of what can or cannot be concluded from your findings. In essence, they help temper and qualify the study implications you listed previously.

Study limitations can relate to research design, specific methodological or material choices, or unexpected issues that emerged while you conducted the research. Mention only those limitations directly relate to your research questions, and explain what impact these limitations had on how your study was conducted and the validity of any interpretations.

Possible types of study limitations:

• Insufficient sample size for statistical measurements
• Lack of previous research studies on the topic
• Methods/instruments/techniques used to collect the data
• Limited access to data
• Time constraints in properly preparing and executing the study

After discussing the study limitations, you can also stress that your results are still valid. Give some specific reasons why the limitations do not necessarily handicap your study or narrow its scope.

Phrase examples: Limitations sentence beginners

• “There may be some possible limitations in this study.”
• “The findings of this study have to be seen in light of some limitations.”
•  “The first limitation is the…The second limitation concerns the…”
•  “The empirical results reported herein should be considered in the light of some limitations.”
• “This research, however, is subject to several limitations.”
• “The primary limitation to the generalization of these results is…”
• “Nonetheless, these results must be interpreted with caution and a number of limitations should be borne in mind.”

Discussion Part 5: Giving Recommendations for Further Research

Based on your interpretation and discussion of the findings, your recommendations can include practical changes to the study or specific further research to be conducted to clarify the research questions. Recommendations are often listed in a separate Conclusion section , but often this is just the final paragraph of the Discussion section.

Suggestions for further research often stem directly from the limitations outlined. Rather than simply stating that “further research should be conducted,” provide concrete specifics for how future can help answer questions that your research could not.

Phrase examples: Recommendation sentence beginners

• Further research is needed to establish …
• There is abundant space for further progress in analyzing…
• A further study with more focus on X should be done to investigate…
• Further studies of X that account for these variables must be undertaken.

Consider Receiving Professional Language Editing

As you edit or draft your research manuscript, we hope that you implement these guidelines to produce a more effective Discussion section. And after completing your draft, don’t forget to submit your work to a professional proofreading and English editing service like Wordvice, including our manuscript editing service for  paper editing , cover letter editing , SOP editing , and personal statement proofreading services. Language editors not only proofread and correct errors in grammar, punctuation, mechanics, and formatting but also improve terms and revise phrases so they read more naturally. Wordvice is an industry leader in providing high-quality revision for all types of academic documents.

For additional information about how to write a strong research paper, make sure to check out our full  research writing series !

Wordvice Writing Resources

• How to Write a Research Paper Introduction 
• Which Verb Tenses to Use in a Research Paper
• How to Write an Abstract for a Research Paper
• How to Write a Research Paper Title
• Useful Phrases for Academic Writing
• Common Transition Terms in Academic Papers
• Active and Passive Voice in Research Papers
• 100+ Verbs That Will Make Your Research Writing Amazing
• Tips for Paraphrasing in Research Papers

Additional Academic Resources

•   Guide for Authors.  (Elsevier)
•  How to Write the Results Section of a Research Paper.  (Bates College)
•   Structure of a Research Paper.  (University of Minnesota Biomedical Library)
•   How to Choose a Target Journal  (Springer)
•   How to Write Figures and Tables  (UNC Writing Center)

Welcome to the new OASIS website! We have academic skills, library skills, math and statistics support, and writing resources all together in one new home.

• Walden University
• Faculty Portal

General Research Paper Guidelines: Discussion

Discussion section.

The overall purpose of a research paper’s discussion section is to evaluate and interpret results, while explaining both the implications and limitations of your findings. Per APA (2020) guidelines, this section requires you to “examine, interpret, and qualify the results and draw inferences and conclusions from them” (p. 89). Discussion sections also require you to detail any new insights, think through areas for future research, highlight the work that still needs to be done to further your topic, and provide a clear conclusion to your research paper. In a good discussion section, you should do the following:

• Clearly connect the discussion of your results to your introduction, including your central argument, thesis, or problem statement.
• Provide readers with a critical thinking through of your results, answering the “so what?” question about each of your findings. In other words, why is this finding important?
• Detail how your research findings might address critical gaps or problems in your field
• Compare your results to similar studies’ findings
• Provide the possibility of alternative interpretations, as your goal as a researcher is to “discover” and “examine” and not to “prove” or “disprove.” Instead of trying to fit your results into your hypothesis, critically engage with alternative interpretations to your results.

For more specific details on your Discussion section, be sure to review Sections 3.8 (pp. 89-90) and 3.16 (pp. 103-104) of your 7 th edition APA manual

*Box content adapted from:

University of Southern California (n.d.). Organizing your social sciences research paper: 8 the discussion . https://libguides.usc.edu/writingguide/discussion

Limitations

Limitations of generalizability or utility of findings, often over which the researcher has no control, should be detailed in your Discussion section. Including limitations for your reader allows you to demonstrate you have thought critically about your given topic, understood relevant literature addressing your topic, and chosen the methodology most appropriate for your research. It also allows you an opportunity to suggest avenues for future research on your topic. An effective limitations section will include the following:

• Detail (a) sources of potential bias, (b) possible imprecision of measures, (c) other limitations or weaknesses of the study, including any methodological or researcher limitations.
• Sample size: In quantitative research, if a sample size is too small, it is more difficult to generalize results.
• Lack of available/reliable data : In some cases, data might not be available or reliable, which will ultimately affect the overall scope of your research. Use this as an opportunity to explain areas for future study.
• Lack of prior research on your study topic: In some cases, you might find that there is very little or no similar research on your study topic, which hinders the credibility and scope of your own research. If this is the case, use this limitation as an opportunity to call for future research. However, make sure you have done a thorough search of the available literature before making this claim.
• Flaws in measurement of data: Hindsight is 20/20, and you might realize after you have completed your research that the data tool you used actually limited the scope or results of your study in some way. Again, acknowledge the weakness and use it as an opportunity to highlight areas for future study.
• Limits of self-reported data: In your research, you are assuming that any participants will be honest and forthcoming with responses or information they provide to you. Simply acknowledging this assumption as a possible limitation is important in your research.
• Access: Most research requires that you have access to people, documents, organizations, etc.. However, for various reasons, access is sometimes limited or denied altogether. If this is the case, you will want to acknowledge access as a limitation to your research.
• Time: Choosing a research focus that is narrow enough in scope to finish in a given time period is important. If such limitations of time prevent you from certain forms of research, access, or study designs, acknowledging this time restraint is important. Acknowledging such limitations is important, as they can point other researchers to areas that require future study.
• Potential Bias: All researchers have some biases, so when reading and revising your draft, pay special attention to the possibilities for bias in your own work. Such bias could be in the form you organized people, places, participants, or events. They might also exist in the method you selected or the interpretation of your results. Acknowledging such bias is an important part of the research process.
• Language Fluency: On occasion, researchers or research participants might have language fluency issues, which could potentially hinder results or how effectively you interpret results. If this is an issue in your research, make sure to acknowledge it in your limitations section.

University of Southern California (n.d.). Organizing your social sciences research paper: Limitations of the study . https://libguides.usc.edu/writingguide/limitations

In many research papers, the conclusion, like the limitations section, is folded into the larger discussion section. If you are unsure whether to include the conclusion as part of your discussion or as a separate section, be sure to defer to the assignment instructions or ask your instructor.

The conclusion is important, as it is specifically designed to highlight your research’s larger importance outside of the specific results of your study. Your conclusion section allows you to reiterate the main findings of your study, highlight their importance, and point out areas for future research. Based on the scope of your paper, your conclusion could be anywhere from one to three paragraphs long. An effective conclusion section should include the following:

• Describe the possibilities for continued research on your topic, including what might be improved, adapted, or added to ensure useful and informed future research.
• Provide a detailed account of the importance of your findings
• Reiterate why your problem is important, detail how your interpretation of results impacts the subfield of study, and what larger issues both within and outside of your field might be affected from such results

University of Southern California (n.d.). Organizing your social sciences research paper: 9. the conclusion . https://libguides.usc.edu/writingguide/conclusion

• Previous Page: Results
• Next Page: References
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Cost of Attendance
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

• Methodology
• Research Methodology
• Research Papers

How to Write an Effective Discussion in a Research Paper; a Guide to Writing the Discussion Section of a Research Article

• University Canada West

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Sergio Tobon
• David Chávez-Herting
• Panagiota Xanthopoulou

• Vasiliki Tsipouri

• J Assoc Phys India

• Mohammad Ali Salmani Nodoushan
• Nafiseh Khakbaz
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Med Res Methodol

A tutorial on methodological studies: the what, when, how and why

Lawrence mbuagbaw.

1 Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON Canada

2 Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario L8N 4A6 Canada

3 Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Daeria O. Lawson

Livia puljak.

4 Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000 Zagreb, Croatia

David B. Allison

5 Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN 47405 USA

Lehana Thabane

6 Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON Canada

7 Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON Canada

8 Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON Canada

Associated Data

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 – 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 – 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig.  1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 – 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

• Comparing two groups
• Determining a proportion, mean or another quantifier
• Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

• Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.
• Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].
• Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]
• Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 – 67 ].
• Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].
• Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].
• Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].
• Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n  = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n  = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n  = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

• What is the aim?

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

• 2. What is the design?

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

• 3. What is the sampling strategy?

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n  = 103) [ 30 ].

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

• 4. What is the unit of analysis?

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig.  2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Acknowledgements

Abbreviations.

Authors’ contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

This work did not receive any dedicated funding.

Availability of data and materials

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Communicating in STEM Disciplines
• Features of Academic STEM Writing
• STEM Writing Tips
• Academic Integrity in STEM
• Strategies for Writing
• Science Writing Videos – YouTube Channel
• Educator Resources
• Lesson Plans, Activities and Assignments
• Strategies for Teaching Writing
• Grading Techniques

IMRAD (Introduction, Methods, Results and Discussion)

Academic research papers in STEM disciplines typically follow a well-defined I-M-R-A-D structure: Introduction, Methods, Results And Discussion (Wu, 2011). Although not included in the IMRAD name, these papers often include a Conclusion.

Introduction

The Introduction typically provides everything your reader needs to know in order to understand the scope and purpose of your research. This section should provide:

• Context for your research (for example, the nature and scope of your topic)
• A summary of how relevant scholars have approached your research topic to date, and a description of how your research makes a contribution to the scholarly conversation
• An argument or hypothesis that relates to the scholarly conversation
• A brief explanation of your methodological approach and a justification for this approach (in other words, a brief discussion of how you gather your data and why this is an appropriate choice for your contribution)
• The main conclusions of your paper (or the “so what”)
• A roadmap, or a brief description of how the rest of your paper proceeds

The Methods section describes exactly what you did to gather the data that you use in your paper. This should expand on the brief methodology discussion in the introduction and provide readers with enough detail to, if necessary, reproduce your experiment, design, or method for obtaining data; it should also help readers to anticipate your results. The more specific, the better!  These details might include:

• An overview of the methodology at the beginning of the section
• A chronological description of what you did in the order you did it
• Descriptions of the materials used, the time taken, and the precise step-by-step process you followed
• An explanation of software used for statistical calculations (if necessary)
• Justifications for any choices or decisions made when designing your methods

Because the methods section describes what was done to gather data, there are two things to consider when writing. First, this section is usually written in the past tense (for example, we poured 250ml of distilled water into the 1000ml glass beaker). Second, this section should not be written as a set of instructions or commands but as descriptions of actions taken. This usually involves writing in the active voice (for example, we poured 250ml of distilled water into the 1000ml glass beaker), but some readers prefer the passive voice (for example, 250ml of distilled water was poured into the 1000ml beaker). It’s important to consider the audience when making this choice, so be sure to ask your instructor which they prefer.

The Results section outlines the data gathered through the methods described above and explains what the data show. This usually involves a combination of tables and/or figures and prose. In other words, the results section gives your reader context for interpreting the data. The results section usually includes:

• A presentation of the data obtained through the means described in the methods section in the form of tables and/or figures
• Statements that summarize or explain what the data show
• Highlights of the most important results

Tables should be as succinct as possible, including only vital information (often summarized) and figures should be easy to interpret and be visually engaging. When adding your written explanation to accompany these visual aids, try to refer your readers to these in such a way that they provide an additional descriptive element, rather than simply telling people to look at them. This can be especially helpful for readers who find it hard to see patterns in data.

The Discussion section explains why the results described in the previous section are meaningful in relation to previous scholarly work and the specific research question your paper explores. This section usually includes:

• Engagement with sources that are relevant to your work (you should compare and contrast your results to those of similar researchers)
• An explanation of the results that you found, and why these results are important and/or interesting

Some papers have separate Results and Discussion sections, while others combine them into one section, Results and Discussion. There are benefits to both. By presenting these as separate sections, you’re able to discuss all of your results before moving onto the implications. By presenting these as one section, you’re able to discuss specific results and move onto their significance before introducing another set of results.

The Conclusion section of a paper should include a brief summary of the main ideas or key takeaways of the paper and their implications for future research. This section usually includes:

• A brief overview of the main claims and/or key ideas put forth in the paper
• A brief discussion of potential limitations of the study (if relevant)
• Some suggestions for future research (these should be clearly related to the content of your paper)

Sample Research Article

Resource Download

Wu, Jianguo. “Improving the writing of research papers: IMRAD and beyond.” Landscape Ecology 26, no. 10 (November 2011): 1345–1349. http://dx.doi.org/10.1007/s10980-011-9674-3.

Further reading:

• Organization of a Research Paper: The IMRAD Format by P. K. Ramachandran Nair and Vimala D. Nair
• George Mason University Writing Centre’s guide on Writing a Scientific Research Report (IMRAD)
• University of Wisconsin Writing Centre’s guide on Formatting Science Reports

• Open access
• Published: 07 September 2020

A tutorial on methodological studies: the what, when, how and why

• Lawrence Mbuagbaw   ORCID: orcid.org/0000-0001-5855-5461 1 , 2 , 3 ,
• Daeria O. Lawson 1 ,
• Livia Puljak 4 ,
• David B. Allison 5 &
• Lehana Thabane 1 , 2 , 6 , 7 , 8  

BMC Medical Research Methodology volume  20 , Article number:  226 ( 2020 ) Cite this article

41k Accesses

60 Citations

60 Altmetric

Metrics details

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Peer Review reports

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 , 2 , 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 , 7 , 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig.  1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 , 13 , 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

Comparing two groups

Determining a proportion, mean or another quantifier

Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.

Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].

Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]

Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 , 66 , 67 ].

Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].

Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].

Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].

Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n  = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n  = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n  = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

What is the aim?

Methodological studies that investigate bias

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies that investigate quality (or completeness) of reporting

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Methodological studies that investigate the consistency of reporting

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

Methodological studies that investigate factors associated with reporting

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies that investigate methods

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Methodological studies that summarize other methodological studies

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Methodological studies that investigate nomenclature and terminology

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

Other types of methodological studies

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

What is the design?

Methodological studies that are descriptive

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Methodological studies that are analytical

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

What is the sampling strategy?

Methodological studies that include the target population

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n  = 103) [ 30 ].

Methodological studies that include a sample of the target population

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

What is the unit of analysis?

Methodological studies with a research report as the unit of analysis

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Methodological studies with a design, analysis or reporting item as the unit of analysis

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig.  2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Availability of data and materials

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Abbreviations

Consolidated Standards of Reporting Trials

Evidence, Participants, Intervention, Comparison, Outcome, Timeframe

Grading of Recommendations, Assessment, Development and Evaluations

Participants, Intervention, Comparison, Outcome, Timeframe

Preferred Reporting Items of Systematic reviews and Meta-Analyses

Studies Within a Review

Studies Within a Trial

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

PubMed   Google Scholar  

Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.

PubMed   PubMed Central   Google Scholar  

Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj. 2017;358:j4008.

Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Studies. 2020;6(1):13.

Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020.

Abbade LPF, Wang M, Sriganesh K, Jin Y, Mbuagbaw L, Thabane L. The framing of research questions using the PICOT format in randomized controlled trials of venous ulcer disease is suboptimal: a systematic survey. Wound Repair Regen. 2017;25(5):892–900.

Gohari F, Baradaran HR, Tabatabaee M, Anijidani S, Mohammadpour Touserkani F, Atlasi R, Razmgir M. Quality of reporting randomized controlled trials (RCTs) in diabetes in Iran; a systematic review. J Diabetes Metab Disord. 2015;15(1):36.

Wang M, Jin Y, Hu ZJ, Thabane A, Dennis B, Gajic-Veljanoski O, Paul J, Thabane L. The reporting quality of abstracts of stepped wedge randomized trials is suboptimal: a systematic survey of the literature. Contemp Clin Trials Commun. 2017;8:1–10.

Shanthanna H, Kaushal A, Mbuagbaw L, Couban R, Busse J, Thabane L: A cross-sectional study of the reporting quality of pilot or feasibility trials in high-impact anesthesia journals Can J Anaesthesia 2018, 65(11):1180–1195.

Kosa SD, Mbuagbaw L, Borg Debono V, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemporary Clinical Trials. 2018;65:144–50.

Zhang Y, Florez ID, Colunga Lozano LE, Aloweni FAB, Kennedy SA, Li A, Craigie S, Zhang S, Agarwal A, Lopes LC, et al. A systematic survey on reporting and methods for handling missing participant data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:57–66.

CAS   PubMed   Google Scholar  

Hernández AV, Boersma E, Murray GD, Habbema JD, Steyerberg EW. Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading? Am Heart J. 2006;151(2):257–64.

Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, Fruci V, Dennis B, Bawor M, Thabane L. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary-of-findings tables with a new format. J Clin Epidemiol. 2016;74:7–18.

The Northern Ireland Hub for Trials Methodology Research: SWAT/SWAR Information [ https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/ ]. Accessed 31 Aug 2020.

Chick S, Sánchez P, Ferrin D, Morrice D. How to conduct a successful simulation study. In: Proceedings of the 2003 winter simulation conference: 2003; 2003. p. 66–70.

Google Scholar  

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106(3):485–8.

Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med New York. 1996;63(3–4):216–24.

CAS   Google Scholar  

Areia M, Soares M, Dinis-Ribeiro M. Quality reporting of endoscopic diagnostic studies in gastrointestinal journals: where do we stand on the use of the STARD and CONSORT statements? Endoscopy. 2010;42(2):138–47.

Knol M, Groenwold R, Grobbee D. P-values in baseline tables of randomised controlled trials are inappropriate but still common in high impact journals. Eur J Prev Cardiol. 2012;19(2):231–2.

Chen M, Cui J, Zhang AL, Sze DM, Xue CC, May BH. Adherence to CONSORT items in randomized controlled trials of integrative medicine for colorectal Cancer published in Chinese journals. J Altern Complement Med. 2018;24(2):115–24.

Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

The Cochrane Methodology Register Issue 2 2009 [ https://cmr.cochrane.org/help.htm ]. Accessed 31 Aug 2020.

Mbuagbaw L, Kredo T, Welch V, Mursleen S, Ross S, Zani B, Motaze NV, Quinlan L. Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis. J Clin Epidemiol. 2016;74:66–72.

Barton S, Peckitt C, Sclafani F, Cunningham D, Chau I. The influence of industry sponsorship on the reporting of subgroup analyses within phase III randomised controlled trials in gastrointestinal oncology. Eur J Cancer. 2015;51(18):2732–9.

Setia MS. Methodology series module 5: sampling strategies. Indian J Dermatol. 2016;61(5):505–9.

Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;3(4):337–46.

Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, Neumann I, Li L, Lopes LC, Briel M, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, et al. Is this clinical trial fully registered?: a statement from the International Committee of Medical Journal Editors*. Ann Intern Med. 2005;143(2):146–8.

Ohtake PJ, Childs JD. Why publish study protocols? Phys Ther. 2014;94(9):1208–9.

Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.

Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002;359(9302):248–52.

Porta M (ed.): A dictionary of epidemiology, 5th edn. Oxford: Oxford University Press, Inc.; 2008.

El Dib R, Tikkinen KAO, Akl EA, Gomaa HA, Mustafa RA, Agarwal A, Carpenter CR, Zhang Y, Jorge EC, Almeida R, et al. Systematic survey of randomized trials evaluating the impact of alternative diagnostic strategies on patient-important outcomes. J Clin Epidemiol. 2017;84:61–9.

Helzer JE, Robins LN, Taibleson M, Woodruff RA Jr, Reich T, Wish ED. Reliability of psychiatric diagnosis. I. a methodological review. Arch Gen Psychiatry. 1977;34(2):129–33.

Chung ST, Chacko SK, Sunehag AL, Haymond MW. Measurements of gluconeogenesis and Glycogenolysis: a methodological review. Diabetes. 2015;64(12):3996–4010.

CAS   PubMed   PubMed Central   Google Scholar  

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002;21(11):1513–24.

Moen EL, Fricano-Kugler CJ, Luikart BW, O’Malley AJ. Analyzing clustered data: why and how to account for multiple observations nested within a study participant? PLoS One. 2016;11(1):e0146721.

Zyzanski SJ, Flocke SA, Dickinson LM. On the nature and analysis of clustered data. Ann Fam Med. 2004;2(3):199–200.

Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152.

Bui DDA, Del Fiol G, Hurdle JF, Jonnalagadda S. Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform. 2016;64:265–72.

Bui DD, Del Fiol G, Jonnalagadda S. PDF text classification to leverage information extraction from publication reports. J Biomed Inform. 2016;61:141–8.

Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19(1):32.

Speich B. Blinding in surgical randomized clinical trials in 2015. Ann Surg. 2017;266(1):21–2.

Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, et al. A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews. J Clin Epidemiol. 2017;84:37–46.

Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, Min J, Wang G, Fu J, Mao B. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: A survey of reports indexed in the Science Citation Index Expanded. Eur J Integrative Med. 2011;3(4):e309–16.

Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007;50(4):266–77.

Oltean H, Gagnier JJ. Use of clustering analysis in randomized controlled trials in orthopaedic surgery. BMC Med Res Methodol. 2015;15:17.

Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS One. 2014;9(5):e96407.

Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg. 2006;244(5):663–7.

de Vries TW, van Roon EN. Low quality of reporting adverse drug reactions in paediatric randomised controlled trials. Arch Dis Child. 2010;95(12):1023–6.

Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13.

Kaiser KA, Cofield SS, Fontaine KR, Glasser SP, Thabane L, Chu R, Ambrale S, Dwary AD, Kumar A, Nayyar G, et al. Is funding source related to study reporting quality in obesity or nutrition randomized control trials in top-tier medical journals? Int J Obes. 2012;36(7):977–81.

Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes. 2008;32(10):1531–6.

Khan NR, Saad H, Oravec CS, Rossi N, Nguyen V, Venable GT, Lillard JC, Patel P, Taylor DR, Vaughn BN, et al. A review of industry funding in randomized controlled trials published in the neurosurgical literature-the elephant in the room. Neurosurgery. 2018;83(5):890–7.

Hansen C, Lundh A, Rasmussen K, Hrobjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev. 2019;8:Mr000047.

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280–5.

Liu LQ, Morris PJ, Pengel LH. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.

Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, Vandvik PO, Malaga G, Johnston BC, Dahm P, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95.

Lee SY, Teoh PJ, Camm CF, Agha RA. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–72.

Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494–500.

Alvarez F, Meyer N, Gourraud PA, Paul C. CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals. Br J Dermatol. 2009;161(5):1159–65.

Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemporary Clin trials. 2014;38(2):245–50.

Thabane L, Chu R, Cuddy K, Douketis J. What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes. 2007;31(10):1554–9.

Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence Based Med. 2017;22(4):139.

METRIC - MEthodological sTudy ReportIng Checklist: guidelines for reporting methodological studies in health research [ http://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#METRIC ]. Accessed 31 Aug 2020.

Jager KJ, Zoccali C, MacLeod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int. 2008;73(3):256–60.

Parker SG, Halligan S, Erotocritou M, Wood CPJ, Boulton RW, Plumb AAO, Windsor ACJ, Mallett S. A systematic methodological review of non-randomised interventional studies of elective ventral hernia repair: clear definitions and a standardised minimum dataset are needed. Hernia. 2019.

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.

Schiller P, Burchardi N, Niestroj M, Kieser M. Quality of reporting of clinical non-inferiority and equivalence randomised trials--update and extension. Trials. 2012;13:214.

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Thabut G, Estellat C, Boutron I, Samama CM, Ravaud P. Methodological issues in trials assessing primary prophylaxis of venous thrombo-embolism. Eur Heart J. 2005;27(2):227–36.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64.

Ritchie A, Seubert L, Clifford R, Perry D, Bond C. Do randomised controlled trials relevant to pharmacy meet best practice standards for quality conduct and reporting? A systematic review. Int J Pharm Pract. 2019.

Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, Pericic TP, Pieper D, Puljak L. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2019.

Tan A, Porcher R, Crequit P, Ravaud P, Dechartres A. Differences in treatment effect size between overall survival and progression-free survival in immunotherapy trials: a Meta-epidemiologic study of trials with results posted at ClinicalTrials.gov. J Clin Oncol. 2017;35(15):1686–94.

Croitoru D, Huang Y, Kurdina A, Chan AW, Drucker AM. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182(6):1469–76.

Khan MS, Ochani RK, Shaikh A, Vaduganathan M, Khan SU, Fatima K, Yamani N, Mandrola J, Doukky R, Krasuski RA: Assessing the Quality of Reporting of Harms in Randomized Controlled Trials Published in High Impact Cardiovascular Journals. Eur Heart J Qual Care Clin Outcomes 2019.

Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME. From conference abstract to full paper: differences between data presented in conferences and journals. FASEB J. 2005;19(7):673–80.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, Scott P. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, et al. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol. 2017;17(1):181.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

Analytical study [ https://medical-dictionary.thefreedictionary.com/analytical+study ]. Accessed 31 Aug 2020.

Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6 e381.

Schalken N, Rietbergen C. The reporting quality of systematic reviews and Meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol. 2017;8:1395.

Ranker LR, Petersen JM, Fox MP. Awareness of and potential for dependent error in the observational epidemiologic literature: A review. Ann Epidemiol. 2019;36:15–9 e12.

Paquette M, Alotaibi AM, Nieuwlaat R, Santesso N, Mbuagbaw L. A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation. Syst Rev. 2019;8(1):241.

Download references

Acknowledgements

This work did not receive any dedicated funding.

Author information

Authors and affiliations.

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Lawrence Mbuagbaw, Daeria O. Lawson & Lehana Thabane

Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario, L8N 4A6, Canada

Lawrence Mbuagbaw & Lehana Thabane

Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Lawrence Mbuagbaw

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN, 47405, USA

David B. Allison

Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

Lehana Thabane

Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Lawrence Mbuagbaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mbuagbaw, L., Lawson, D.O., Puljak, L. et al. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 20 , 226 (2020). https://doi.org/10.1186/s12874-020-01107-7

Download citation

Received : 27 May 2020

Accepted : 27 August 2020

Published : 07 September 2020

DOI : https://doi.org/10.1186/s12874-020-01107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodological study
• Meta-epidemiology
• Research methods
• Research-on-research

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Organizing Academic Research Papers: 6. The Methodology

• Purpose of Guide
• Design Flaws to Avoid
• Glossary of Research Terms
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Executive Summary
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tertiary Sources
• What Is Scholarly vs. Popular?
• Qualitative Methods
• Quantitative Methods
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Annotated Bibliography
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• How to Manage Group Projects
• Multiple Book Review Essay
• Reviewing Collected Essays
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Research Proposal
• Acknowledgements

The methods section of a research paper provides the information by which a study’s validity is judged. The method section answers two main questions: 1) How was the data collected or generated? 2) How was it analyzed? The writing should be direct and precise and written in the past tense.

Importance of a Good Methodology Section

You must explain how you obtained and analyzed your results for the following reasons:

• Readers need to know how the data was obtained because the method you choose affects the results and, by extension, how you likely interpreted those results.
• Methodology is crucial for any branch of scholarship because an unreliable method produces unreliable results and it misappropriates interpretations of findings .
• In most cases, there are a variety of different methods you can choose to investigate a research problem. Your methodology section of your paper should make clear the reasons why you chose a particular method or procedure .
• The reader wants to know that the data was collected or generated in a way that is consistent with accepted practice in the field of study. For example, if you are using a questionnaire, readers need to know that it offered your respondents a reasonable range of answers to choose from.
• The research method must be appropriate to the objectives of the study . For example, be sure you have a large enough sample size to be able to generalize and make recommendations based upon the findings.
• The methodology should discuss the problems that were anticipated and the steps you took to prevent them from occurring . For any problems that did arise, you must describe the ways in which their impact was minimized or why these problems do not affect the findings in any way that impacts your interpretation of the data.
• Often in social science research, it is useful for other researchers to adapt or replicate your methodology. Therefore, it is important to always provide sufficient information to allow others to use or replicate the study . This information is particularly important when a new method had been developed or an innovative use of an existing method has been utilized.

Bem, Daryl J. Writing the Empirical Journal Article . Psychology Writing Center. University of Washington; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008.

Structure and Writing Style

I. Groups of Research Methods

There are two main groups of research methods in the social sciences:

• The empirical-analytical group approaches the study of social sciences in a similar manner that researchers study the natural sciences. This type of research focuses on objective knowledge, research questions that can be answered yes or no, and operational definitions of variables to be measured. The empirical-analytical group employs deductive reasoning that uses existing theory as a foundation for hypotheses that need to be tested. This approach is focused on explanation .
• The interpretative group is focused on understanding phenomenon in a comprehensive, holistic way . This research method allows you to recognize your connection to the subject under study. Because the interpretative group focuses more on subjective knowledge, it requires careful interpretation of variables.

II. Content

An effectively written methodology section should:

• Introduce the overall methodological approach for investigating your research problem . Is your study qualitative or quantitative or a combination of both (mixed method)? Are you going to take a special approach, such as action research, or a more neutral stance?
• Indicate how the approach fits the overall research design . Your methods should have a clear connection with your research problem. In other words, make sure that your methods will actually address the problem. One of the most common deficiencies found in research papers is that the proposed methodology is unsuited to achieving the stated objective of your paper.
• Describe the specific methods of data collection you are going to use , such as, surveys, interviews, questionnaires, observation, archival research. If you are analyzing existing data, such as a data set or archival documents, describe how it was originally created or gathered and by whom.
• Explain how you intend to analyze your results . Will you use statistical analysis? Will you use specific theoretical perspectives to help you analyze a text or explain observed behaviors?
• Provide background and rationale for methodologies that are unfamiliar for your readers . Very often in the social sciences, research problems and the methods for investigating them require more explanation/rationale than widely accepted rules governing the natural and physical sciences. Be clear and concise in your explanation.
• Provide a rationale for subject selection and sampling procedure . For instance, if you propose to conduct interviews, how do you intend to select the sample population? If you are analyzing texts, which texts have you chosen, and why? If you are using statistics, why is this set of statisics being used? If other data sources exist, explain why the data you chose is most appropriate.
• Address potential limitations . Are there any practical limitations that could affect your data collection? How will you attempt to control for potential confounding variables and errors? If your methodology may lead to problems you can anticipate, state this openly and show why pursuing this methodology outweighs the risk of these problems cropping up.

NOTE :  Once you have written all of the elements of the methods section, subsequent revisions should focus on how to present those elements as clearly and as logically as possibly. The description of how you prepared to study the research problem, how you gathered the data, and the protocol for analyzing the data should be organized chronologically. For clarity, when a large amount of detail must be presented, information should be presented in sub-sections according to topic.

III.  Problems to Avoid

Irrelevant Detail The methodology section of your paper should be thorough but to the point. Don’t provide any background information that doesn’t directly help the reader to understand why a particular method was chosen, how the data was gathered or obtained, and how it was analyzed. Unnecessary Explanation of Basic Procedures Remember that you are not writing a how-to guide about a particular method. You should make the assumption that readers possess a basic understanding of how to investigate the research problem on their own and, therefore, you do not have to go into great detail about specific methodological procedures. The focus should be on how you applied a method , not on the mechanics of doing a method. NOTE: An exception to this rule is if you select an unconventional approach to doing the method; if this is the case, be sure to explain why this approach was chosen and how it enhances the overall research process. Problem Blindness It is almost a given that you will encounter problems when collecting or generating your data. Do not ignore these problems or pretend they did not occur. Often, documenting how you overcame obstacles can form an interesting part of the methodology. It demonstrates to the reader that you can provide a cogent rationale for the decisions you made to minimize the impact of any problems that arose. Literature Review Just as the literature review section of your paper provides an overview of sources you have examined while researching a particular topic, the methodology section should cite any sources that informed your choice and application of a particular method [i.e., the choice of a survey should include any citations to the works you used to help construct the survey].

It’s More than Sources of Information! A description of a research study's method should not be confused with a description of the sources of information. Such a list of sources is useful in itself, especially if it is accompanied by an explanation about the selection and use of the sources. The description of the project's methodology complements a list of sources in that it sets forth the organization and interpretation of information emanating from those sources.

Azevedo, L.F. et al. How to Write a Scientific Paper: Writing the Methods Section. Revista Portuguesa de Pneumologia 17 (2011): 232-238; Butin, Dan W. The Education Dissertation A Guide for Practitioner Scholars . Thousand Oaks, CA: Corwin, 2010; Carter, Susan. Structuring Your Research Thesis . New York: Palgrave Macmillan, 2012; Lunenburg, Frederick C. Writing a Successful Thesis or Dissertation: Tips and Strategies for Students in the Social and Behavioral Sciences . Thousand Oaks, CA: Corwin Press, 2008. Methods Section . The Writer’s Handbook. Writing Center. University of Wisconsin, Madison; Writing the Experimental Report: Methods, Results, and Discussion . The Writing Lab and The OWL. Purdue University; Methods and Materials . The Structure, Format, Content, and Style of a Journal-Style Scientific Paper. Department of Biology. Bates College.

Writing Tip

Statistical Designs and Tests? Do Not Fear Them!

Don't avoid using a quantitative approach to analyzing your research problem just because you fear the idea of applying statistical designs and tests. A qualitative approach, such as conducting interviews or content analysis of archival texts, can yield exciting new insights about a research problem, but it should not be undertaken simply because you have a disdain for running a simple regression. A well designed quantitative research study can often be accomplished in very clear and direct ways, whereas, a similar study of a qualitative nature usually requires considerable time to analyze large volumes of data and a tremendous burden to create new paths for analysis where previously no path associated with your research problem had existed.

Another Writing Tip

Knowing the Relationship Between Theories and Methods

There can be multiple meaning associated with the term "theories" and the term "methods" in social sciences research. A helpful way to delineate between them is to understand "theories" as representing different ways of characterizing the social world when you research it and "methods" as representing different ways of generating and analyzing data about that social world. Framed in this way, all empirical social sciences research involves theories and methods, whether they are stated explicitly or not. However, while theories and methods are often related, it is important that, as a researcher, you deliberately separate them in order to avoid your theories playing a disproportionate role in shaping what outcomes your chosen methods produce.

Introspectively engage in an ongoing dialectic between theories and methods to help enable you to use the outcomes from your methods to interrogate and develop new theories, or ways of framing conceptually the research problem. This is how scholarship grows and branches out into new intellectual territory.

Reynolds, R. Larry. Ways of Knowing. Alternative Microeconomics. Part 1, Chapter 3. Boise State University; The Theory-Method Relationship . S-Cool Revision. United Kingdom.

• << Previous: What Is Scholarly vs. Popular?
• Next: Qualitative Methods >>
• Last Updated: Jul 18, 2023 11:58 AM
• URL: https://library.sacredheart.edu/c.php?g=29803
• QuickSearch
• Library Catalog
• Databases A-Z
• Publication Finder
• Course Reserves
• Citation Linker
• Digital Commons
• Our Website

Research Support

• Ask a Librarian
• Appointments
• Interlibrary Loan (ILL)
• Research Guides
• Databases by Subject
• Citation Help

Using the Library

• Reserve a Group Study Room
• Renew Books
• Honors Study Rooms
• Off-Campus Access
• Library Policies
• Library Technology

User Information

• Grad Students
• Online Students
• COVID-19 Updates
• Staff Directory
• News & Announcements
• Library Newsletter

My Accounts

• Interlibrary Loan
• Staff Site Login

FIND US ON  

What is Research Methodology? Definition, Types, and Examples

Research methodology 1,2 is a structured and scientific approach used to collect, analyze, and interpret quantitative or qualitative data to answer research questions or test hypotheses. A research methodology is like a plan for carrying out research and helps keep researchers on track by limiting the scope of the research. Several aspects must be considered before selecting an appropriate research methodology, such as research limitations and ethical concerns that may affect your research.

The research methodology section in a scientific paper describes the different methodological choices made, such as the data collection and analysis methods, and why these choices were selected. The reasons should explain why the methods chosen are the most appropriate to answer the research question. A good research methodology also helps ensure the reliability and validity of the research findings. There are three types of research methodology—quantitative, qualitative, and mixed-method, which can be chosen based on the research objectives.

What is research methodology ?

A research methodology describes the techniques and procedures used to identify and analyze information regarding a specific research topic. It is a process by which researchers design their study so that they can achieve their objectives using the selected research instruments. It includes all the important aspects of research, including research design, data collection methods, data analysis methods, and the overall framework within which the research is conducted. While these points can help you understand what is research methodology, you also need to know why it is important to pick the right methodology.

Having a good research methodology in place has the following advantages: 3

• Helps other researchers who may want to replicate your research; the explanations will be of benefit to them.
• You can easily answer any questions about your research if they arise at a later stage.
• A research methodology provides a framework and guidelines for researchers to clearly define research questions, hypotheses, and objectives.
• It helps researchers identify the most appropriate research design, sampling technique, and data collection and analysis methods.
• A sound research methodology helps researchers ensure that their findings are valid and reliable and free from biases and errors.
• It also helps ensure that ethical guidelines are followed while conducting research.
• A good research methodology helps researchers in planning their research efficiently, by ensuring optimum usage of their time and resources.

Writing the methods section of a research paper? Let Paperpal help you achieve perfection  

Types of research methodology.

There are three types of research methodology based on the type of research and the data required. 1

• Quantitative research methodology focuses on measuring and testing numerical data. This approach is good for reaching a large number of people in a short amount of time. This type of research helps in testing the causal relationships between variables, making predictions, and generalizing results to wider populations.
• Qualitative research methodology examines the opinions, behaviors, and experiences of people. It collects and analyzes words and textual data. This research methodology requires fewer participants but is still more time consuming because the time spent per participant is quite large. This method is used in exploratory research where the research problem being investigated is not clearly defined.
• Mixed-method research methodology uses the characteristics of both quantitative and qualitative research methodologies in the same study. This method allows researchers to validate their findings, verify if the results observed using both methods are complementary, and explain any unexpected results obtained from one method by using the other method.

What are the types of sampling designs in research methodology?

Sampling 4 is an important part of a research methodology and involves selecting a representative sample of the population to conduct the study, making statistical inferences about them, and estimating the characteristics of the whole population based on these inferences. There are two types of sampling designs in research methodology—probability and nonprobability.

• Probability sampling

In this type of sampling design, a sample is chosen from a larger population using some form of random selection, that is, every member of the population has an equal chance of being selected. The different types of probability sampling are:

• Systematic —sample members are chosen at regular intervals. It requires selecting a starting point for the sample and sample size determination that can be repeated at regular intervals. This type of sampling method has a predefined range; hence, it is the least time consuming.
• Stratified —researchers divide the population into smaller groups that don’t overlap but represent the entire population. While sampling, these groups can be organized, and then a sample can be drawn from each group separately.
• Cluster —the population is divided into clusters based on demographic parameters like age, sex, location, etc.
• Convenience —selects participants who are most easily accessible to researchers due to geographical proximity, availability at a particular time, etc.
• Purposive —participants are selected at the researcher’s discretion. Researchers consider the purpose of the study and the understanding of the target audience.
• Snowball —already selected participants use their social networks to refer the researcher to other potential participants.
• Quota —while designing the study, the researchers decide how many people with which characteristics to include as participants. The characteristics help in choosing people most likely to provide insights into the subject.

What are data collection methods?

During research, data are collected using various methods depending on the research methodology being followed and the research methods being undertaken. Both qualitative and quantitative research have different data collection methods, as listed below.

Qualitative research 5

• One-on-one interviews: Helps the interviewers understand a respondent’s subjective opinion and experience pertaining to a specific topic or event
• Document study/literature review/record keeping: Researchers’ review of already existing written materials such as archives, annual reports, research articles, guidelines, policy documents, etc.
• Focus groups: Constructive discussions that usually include a small sample of about 6-10 people and a moderator, to understand the participants’ opinion on a given topic.
• Qualitative observation : Researchers collect data using their five senses (sight, smell, touch, taste, and hearing).

Quantitative research 6

• Sampling: The most common type is probability sampling.
• Interviews: Commonly telephonic or done in-person.
• Observations: Structured observations are most commonly used in quantitative research. In this method, researchers make observations about specific behaviors of individuals in a structured setting.
• Document review: Reviewing existing research or documents to collect evidence for supporting the research.
• Surveys and questionnaires. Surveys can be administered both online and offline depending on the requirement and sample size.

Let Paperpal help you write the perfect research methods section. Start now!

What are data analysis methods.

The data collected using the various methods for qualitative and quantitative research need to be analyzed to generate meaningful conclusions. These data analysis methods 7 also differ between quantitative and qualitative research.

Quantitative research involves a deductive method for data analysis where hypotheses are developed at the beginning of the research and precise measurement is required. The methods include statistical analysis applications to analyze numerical data and are grouped into two categories—descriptive and inferential.

Descriptive analysis is used to describe the basic features of different types of data to present it in a way that ensures the patterns become meaningful. The different types of descriptive analysis methods are:

• Measures of frequency (count, percent, frequency)
• Measures of central tendency (mean, median, mode)
• Measures of dispersion or variation (range, variance, standard deviation)
• Measure of position (percentile ranks, quartile ranks)

Inferential analysis is used to make predictions about a larger population based on the analysis of the data collected from a smaller population. This analysis is used to study the relationships between different variables. Some commonly used inferential data analysis methods are:

• Correlation: To understand the relationship between two or more variables.
• Cross-tabulation: Analyze the relationship between multiple variables.
• Regression analysis: Study the impact of independent variables on the dependent variable.
• Frequency tables: To understand the frequency of data.
• Analysis of variance: To test the degree to which two or more variables differ in an experiment.

Qualitative research involves an inductive method for data analysis where hypotheses are developed after data collection. The methods include:

• Content analysis: For analyzing documented information from text and images by determining the presence of certain words or concepts in texts.
• Narrative analysis: For analyzing content obtained from sources such as interviews, field observations, and surveys. The stories and opinions shared by people are used to answer research questions.
• Discourse analysis: For analyzing interactions with people considering the social context, that is, the lifestyle and environment, under which the interaction occurs.
• Grounded theory: Involves hypothesis creation by data collection and analysis to explain why a phenomenon occurred.
• Thematic analysis: To identify important themes or patterns in data and use these to address an issue.

How to choose a research methodology?

Here are some important factors to consider when choosing a research methodology: 8

• Research objectives, aims, and questions —these would help structure the research design.
• Review existing literature to identify any gaps in knowledge.
• Check the statistical requirements —if data-driven or statistical results are needed then quantitative research is the best. If the research questions can be answered based on people’s opinions and perceptions, then qualitative research is most suitable.
• Sample size —sample size can often determine the feasibility of a research methodology. For a large sample, less effort- and time-intensive methods are appropriate.
• Constraints —constraints of time, geography, and resources can help define the appropriate methodology.

Got writer’s block? Kickstart your research paper writing with Paperpal now!

How to write a research methodology .

A research methodology should include the following components: 3,9

• Research design —should be selected based on the research question and the data required. Common research designs include experimental, quasi-experimental, correlational, descriptive, and exploratory.
• Research method —this can be quantitative, qualitative, or mixed-method.
• Reason for selecting a specific methodology —explain why this methodology is the most suitable to answer your research problem.
• Research instruments —explain the research instruments you plan to use, mainly referring to the data collection methods such as interviews, surveys, etc. Here as well, a reason should be mentioned for selecting the particular instrument.
• Sampling —this involves selecting a representative subset of the population being studied.
• Data collection —involves gathering data using several data collection methods, such as surveys, interviews, etc.
• Data analysis —describe the data analysis methods you will use once you’ve collected the data.
• Research limitations —mention any limitations you foresee while conducting your research.
• Validity and reliability —validity helps identify the accuracy and truthfulness of the findings; reliability refers to the consistency and stability of the results over time and across different conditions.
• Ethical considerations —research should be conducted ethically. The considerations include obtaining consent from participants, maintaining confidentiality, and addressing conflicts of interest.

Streamline Your Research Paper Writing Process with Paperpal  

The methods section is a critical part of the research papers, allowing researchers to use this to understand your findings and replicate your work when pursuing their own research. However, it is usually also the most difficult section to write. This is where Paperpal can help you overcome the writer’s block and create the first draft in minutes with Paperpal Copilot, its secure generative AI feature suite.  

With Paperpal you can get research advice, write and refine your work, rephrase and verify the writing, and ensure submission readiness, all in one place. Here’s how you can use Paperpal to develop the first draft of your methods section.  

• Generate an outline: Input some details about your research to instantly generate an outline for your methods section 
• Develop the section: Use the outline and suggested sentence templates to expand your ideas and develop the first draft.  
• P araph ras e and trim : Get clear, concise academic text with paraphrasing that conveys your work effectively and word reduction to fix redundancies. 
• Choose the right words: Enhance text by choosing contextual synonyms based on how the words have been used in previously published work.  
• Check and verify text : Make sure the generated text showcases your methods correctly, has all the right citations, and is original and authentic. .   

You can repeat this process to develop each section of your research manuscript, including the title, abstract and keywords. Ready to write your research papers faster, better, and without the stress? Sign up for Paperpal and start writing today!

Frequently Asked Questions

Q1. What are the key components of research methodology?

A1. A good research methodology has the following key components:

• Research design
• Data collection procedures
• Data analysis methods
• Ethical considerations

Q2. Why is ethical consideration important in research methodology?

A2. Ethical consideration is important in research methodology to ensure the readers of the reliability and validity of the study. Researchers must clearly mention the ethical norms and standards followed during the conduct of the research and also mention if the research has been cleared by any institutional board. The following 10 points are the important principles related to ethical considerations: 10

• Participants should not be subjected to harm.
• Respect for the dignity of participants should be prioritized.
• Full consent should be obtained from participants before the study.
• Participants’ privacy should be ensured.
• Confidentiality of the research data should be ensured.
• Anonymity of individuals and organizations participating in the research should be maintained.
• The aims and objectives of the research should not be exaggerated.
• Affiliations, sources of funding, and any possible conflicts of interest should be declared.
• Communication in relation to the research should be honest and transparent.
• Misleading information and biased representation of primary data findings should be avoided.

Q3. What is the difference between methodology and method?

A3. Research methodology is different from a research method, although both terms are often confused. Research methods are the tools used to gather data, while the research methodology provides a framework for how research is planned, conducted, and analyzed. The latter guides researchers in making decisions about the most appropriate methods for their research. Research methods refer to the specific techniques, procedures, and tools used by researchers to collect, analyze, and interpret data, for instance surveys, questionnaires, interviews, etc.

Research methodology is, thus, an integral part of a research study. It helps ensure that you stay on track to meet your research objectives and answer your research questions using the most appropriate data collection and analysis tools based on your research design.

Accelerate your research paper writing with Paperpal. Try for free now!  

• Research methodologies. Pfeiffer Library website. Accessed August 15, 2023. https://library.tiffin.edu/researchmethodologies/whatareresearchmethodologies
• Types of research methodology. Eduvoice website. Accessed August 16, 2023. https://eduvoice.in/types-research-methodology/
• The basics of research methodology: A key to quality research. Voxco. Accessed August 16, 2023. https://www.voxco.com/blog/what-is-research-methodology/
• Sampling methods: Types with examples. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/types-of-sampling-for-social-research/
• What is qualitative research? Methods, types, approaches, examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-qualitative-research-methods-types-examples/
• What is quantitative research? Definition, methods, types, and examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-quantitative-research-types-and-examples/
• Data analysis in research: Types & methods. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/data-analysis-in-research/#Data_analysis_in_qualitative_research
• Factors to consider while choosing the right research methodology. PhD Monster website. Accessed August 17, 2023. https://www.phdmonster.com/factors-to-consider-while-choosing-the-right-research-methodology/
• What is research methodology? Research and writing guides. Accessed August 14, 2023. https://paperpile.com/g/what-is-research-methodology/
• Ethical considerations. Business research methodology website. Accessed August 17, 2023. https://research-methodology.net/research-methodology/ethical-considerations/

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Dangling Modifiers and How to Avoid Them in Your Writing 
• Research Outlines: How to Write An Introduction Section in Minutes with Paperpal Copilot
• How to Paraphrase Research Papers Effectively
• What is a Literature Review? How to Write It (with Examples)

Language and Grammar Rules for Academic Writing

Climatic vs. climactic: difference and examples, you may also like, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide), maintaining academic integrity with paperpal’s generative ai writing..., research funding basics: what should a grant proposal..., how to write an abstract in research papers....

• Privacy Policy

Home » Research Methodology – Types, Examples and writing Guide

Research Methodology – Types, Examples and writing Guide

Table of Contents

Research Methodology

Definition:

Research Methodology refers to the systematic and scientific approach used to conduct research, investigate problems, and gather data and information for a specific purpose. It involves the techniques and procedures used to identify, collect , analyze , and interpret data to answer research questions or solve research problems . Moreover, They are philosophical and theoretical frameworks that guide the research process.

Structure of Research Methodology

Research methodology formats can vary depending on the specific requirements of the research project, but the following is a basic example of a structure for a research methodology section:

I. Introduction

• Provide an overview of the research problem and the need for a research methodology section
• Outline the main research questions and objectives

II. Research Design

• Explain the research design chosen and why it is appropriate for the research question(s) and objectives
• Discuss any alternative research designs considered and why they were not chosen
• Describe the research setting and participants (if applicable)

III. Data Collection Methods

• Describe the methods used to collect data (e.g., surveys, interviews, observations)
• Explain how the data collection methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or instruments used for data collection

IV. Data Analysis Methods

• Describe the methods used to analyze the data (e.g., statistical analysis, content analysis )
• Explain how the data analysis methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or software used for data analysis

V. Ethical Considerations

• Discuss any ethical issues that may arise from the research and how they were addressed
• Explain how informed consent was obtained (if applicable)
• Detail any measures taken to ensure confidentiality and anonymity

VI. Limitations

• Identify any potential limitations of the research methodology and how they may impact the results and conclusions

VII. Conclusion

• Summarize the key aspects of the research methodology section
• Explain how the research methodology addresses the research question(s) and objectives

Research Methodology Types

Types of Research Methodology are as follows:

Quantitative Research Methodology

This is a research methodology that involves the collection and analysis of numerical data using statistical methods. This type of research is often used to study cause-and-effect relationships and to make predictions.

Qualitative Research Methodology

This is a research methodology that involves the collection and analysis of non-numerical data such as words, images, and observations. This type of research is often used to explore complex phenomena, to gain an in-depth understanding of a particular topic, and to generate hypotheses.

Mixed-Methods Research Methodology

This is a research methodology that combines elements of both quantitative and qualitative research. This approach can be particularly useful for studies that aim to explore complex phenomena and to provide a more comprehensive understanding of a particular topic.

Case Study Research Methodology

This is a research methodology that involves in-depth examination of a single case or a small number of cases. Case studies are often used in psychology, sociology, and anthropology to gain a detailed understanding of a particular individual or group.

Action Research Methodology

This is a research methodology that involves a collaborative process between researchers and practitioners to identify and solve real-world problems. Action research is often used in education, healthcare, and social work.

Experimental Research Methodology

This is a research methodology that involves the manipulation of one or more independent variables to observe their effects on a dependent variable. Experimental research is often used to study cause-and-effect relationships and to make predictions.

Survey Research Methodology

This is a research methodology that involves the collection of data from a sample of individuals using questionnaires or interviews. Survey research is often used to study attitudes, opinions, and behaviors.

Grounded Theory Research Methodology

This is a research methodology that involves the development of theories based on the data collected during the research process. Grounded theory is often used in sociology and anthropology to generate theories about social phenomena.

Research Methodology Example

An Example of Research Methodology could be the following:

Research Methodology for Investigating the Effectiveness of Cognitive Behavioral Therapy in Reducing Symptoms of Depression in Adults

Introduction:

The aim of this research is to investigate the effectiveness of cognitive-behavioral therapy (CBT) in reducing symptoms of depression in adults. To achieve this objective, a randomized controlled trial (RCT) will be conducted using a mixed-methods approach.

Research Design:

The study will follow a pre-test and post-test design with two groups: an experimental group receiving CBT and a control group receiving no intervention. The study will also include a qualitative component, in which semi-structured interviews will be conducted with a subset of participants to explore their experiences of receiving CBT.

Participants:

Participants will be recruited from community mental health clinics in the local area. The sample will consist of 100 adults aged 18-65 years old who meet the diagnostic criteria for major depressive disorder. Participants will be randomly assigned to either the experimental group or the control group.

Intervention :

The experimental group will receive 12 weekly sessions of CBT, each lasting 60 minutes. The intervention will be delivered by licensed mental health professionals who have been trained in CBT. The control group will receive no intervention during the study period.

Data Collection:

Quantitative data will be collected through the use of standardized measures such as the Beck Depression Inventory-II (BDI-II) and the Generalized Anxiety Disorder-7 (GAD-7). Data will be collected at baseline, immediately after the intervention, and at a 3-month follow-up. Qualitative data will be collected through semi-structured interviews with a subset of participants from the experimental group. The interviews will be conducted at the end of the intervention period, and will explore participants’ experiences of receiving CBT.

Data Analysis:

Quantitative data will be analyzed using descriptive statistics, t-tests, and mixed-model analyses of variance (ANOVA) to assess the effectiveness of the intervention. Qualitative data will be analyzed using thematic analysis to identify common themes and patterns in participants’ experiences of receiving CBT.

Ethical Considerations:

This study will comply with ethical guidelines for research involving human subjects. Participants will provide informed consent before participating in the study, and their privacy and confidentiality will be protected throughout the study. Any adverse events or reactions will be reported and managed appropriately.

Data Management:

All data collected will be kept confidential and stored securely using password-protected databases. Identifying information will be removed from qualitative data transcripts to ensure participants’ anonymity.

Limitations:

One potential limitation of this study is that it only focuses on one type of psychotherapy, CBT, and may not generalize to other types of therapy or interventions. Another limitation is that the study will only include participants from community mental health clinics, which may not be representative of the general population.

Conclusion:

This research aims to investigate the effectiveness of CBT in reducing symptoms of depression in adults. By using a randomized controlled trial and a mixed-methods approach, the study will provide valuable insights into the mechanisms underlying the relationship between CBT and depression. The results of this study will have important implications for the development of effective treatments for depression in clinical settings.

How to Write Research Methodology

Writing a research methodology involves explaining the methods and techniques you used to conduct research, collect data, and analyze results. It’s an essential section of any research paper or thesis, as it helps readers understand the validity and reliability of your findings. Here are the steps to write a research methodology:

• Start by explaining your research question: Begin the methodology section by restating your research question and explaining why it’s important. This helps readers understand the purpose of your research and the rationale behind your methods.
• Describe your research design: Explain the overall approach you used to conduct research. This could be a qualitative or quantitative research design, experimental or non-experimental, case study or survey, etc. Discuss the advantages and limitations of the chosen design.
• Discuss your sample: Describe the participants or subjects you included in your study. Include details such as their demographics, sampling method, sample size, and any exclusion criteria used.
• Describe your data collection methods : Explain how you collected data from your participants. This could include surveys, interviews, observations, questionnaires, or experiments. Include details on how you obtained informed consent, how you administered the tools, and how you minimized the risk of bias.
• Explain your data analysis techniques: Describe the methods you used to analyze the data you collected. This could include statistical analysis, content analysis, thematic analysis, or discourse analysis. Explain how you dealt with missing data, outliers, and any other issues that arose during the analysis.
• Discuss the validity and reliability of your research : Explain how you ensured the validity and reliability of your study. This could include measures such as triangulation, member checking, peer review, or inter-coder reliability.
• Acknowledge any limitations of your research: Discuss any limitations of your study, including any potential threats to validity or generalizability. This helps readers understand the scope of your findings and how they might apply to other contexts.
• Provide a summary: End the methodology section by summarizing the methods and techniques you used to conduct your research. This provides a clear overview of your research methodology and helps readers understand the process you followed to arrive at your findings.

When to Write Research Methodology

Research methodology is typically written after the research proposal has been approved and before the actual research is conducted. It should be written prior to data collection and analysis, as it provides a clear roadmap for the research project.

The research methodology is an important section of any research paper or thesis, as it describes the methods and procedures that will be used to conduct the research. It should include details about the research design, data collection methods, data analysis techniques, and any ethical considerations.

The methodology should be written in a clear and concise manner, and it should be based on established research practices and standards. It is important to provide enough detail so that the reader can understand how the research was conducted and evaluate the validity of the results.

Applications of Research Methodology

Here are some of the applications of research methodology:

• To identify the research problem: Research methodology is used to identify the research problem, which is the first step in conducting any research.
• To design the research: Research methodology helps in designing the research by selecting the appropriate research method, research design, and sampling technique.
• To collect data: Research methodology provides a systematic approach to collect data from primary and secondary sources.
• To analyze data: Research methodology helps in analyzing the collected data using various statistical and non-statistical techniques.
• To test hypotheses: Research methodology provides a framework for testing hypotheses and drawing conclusions based on the analysis of data.
• To generalize findings: Research methodology helps in generalizing the findings of the research to the target population.
• To develop theories : Research methodology is used to develop new theories and modify existing theories based on the findings of the research.
• To evaluate programs and policies : Research methodology is used to evaluate the effectiveness of programs and policies by collecting data and analyzing it.
• To improve decision-making: Research methodology helps in making informed decisions by providing reliable and valid data.

Purpose of Research Methodology

Research methodology serves several important purposes, including:

• To guide the research process: Research methodology provides a systematic framework for conducting research. It helps researchers to plan their research, define their research questions, and select appropriate methods and techniques for collecting and analyzing data.
• To ensure research quality: Research methodology helps researchers to ensure that their research is rigorous, reliable, and valid. It provides guidelines for minimizing bias and error in data collection and analysis, and for ensuring that research findings are accurate and trustworthy.
• To replicate research: Research methodology provides a clear and detailed account of the research process, making it possible for other researchers to replicate the study and verify its findings.
• To advance knowledge: Research methodology enables researchers to generate new knowledge and to contribute to the body of knowledge in their field. It provides a means for testing hypotheses, exploring new ideas, and discovering new insights.
• To inform decision-making: Research methodology provides evidence-based information that can inform policy and decision-making in a variety of fields, including medicine, public health, education, and business.

Advantages of Research Methodology

Research methodology has several advantages that make it a valuable tool for conducting research in various fields. Here are some of the key advantages of research methodology:

• Systematic and structured approach : Research methodology provides a systematic and structured approach to conducting research, which ensures that the research is conducted in a rigorous and comprehensive manner.
• Objectivity : Research methodology aims to ensure objectivity in the research process, which means that the research findings are based on evidence and not influenced by personal bias or subjective opinions.
• Replicability : Research methodology ensures that research can be replicated by other researchers, which is essential for validating research findings and ensuring their accuracy.
• Reliability : Research methodology aims to ensure that the research findings are reliable, which means that they are consistent and can be depended upon.
• Validity : Research methodology ensures that the research findings are valid, which means that they accurately reflect the research question or hypothesis being tested.
• Efficiency : Research methodology provides a structured and efficient way of conducting research, which helps to save time and resources.
• Flexibility : Research methodology allows researchers to choose the most appropriate research methods and techniques based on the research question, data availability, and other relevant factors.
• Scope for innovation: Research methodology provides scope for innovation and creativity in designing research studies and developing new research techniques.

Research Methodology Vs Research Methods

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Research Paper Title Page – Example and Making...

Research Contribution – Thesis Guide

Data Collection – Methods Types and Examples

Research Paper Title – Writing Guide and Example

Research Objectives – Types, Examples and...

Structure of a Research Paper

Structure of a Research Paper: IMRaD Format

I. The Title Page

• Title: Tells the reader what to expect in the paper.
• Author(s): Most papers are written by one or two primary authors. The remaining authors have reviewed the work and/or aided in study design or data analysis (International Committee of Medical Editors, 1997). Check the Instructions to Authors for the target journal for specifics about authorship.
• Keywords [according to the journal]
• Corresponding Author: Full name and affiliation for the primary contact author for persons who have questions about the research.
• Financial & Equipment Support [if needed]: Specific information about organizations, agencies, or companies that supported the research.
• Conflicts of Interest [if needed]: List and explain any conflicts of interest.

II. Abstract: “Structured abstract” has become the standard for research papers (introduction, objective, methods, results and conclusions), while reviews, case reports and other articles have non-structured abstracts. The abstract should be a summary/synopsis of the paper.

III. Introduction: The “why did you do the study”; setting the scene or laying the foundation or background for the paper.

IV. Methods: The “how did you do the study.” Describe the --

• Context and setting of the study
• Specify the study design
• Population (patients, etc. if applicable)
• Sampling strategy
• Intervention (if applicable)
• Identify the main study variables
• Data collection instruments and procedures
• Outline analysis methods

V. Results: The “what did you find” --

• Report on data collection and/or recruitment
• Participants (demographic, clinical condition, etc.)
• Present key findings with respect to the central research question
• Secondary findings (secondary outcomes, subgroup analyses, etc.)

VI. Discussion: Place for interpreting the results

• Main findings of the study
• Discuss the main results with reference to previous research
• Policy and practice implications of the results
• Strengths and limitations of the study

VII. Conclusions: [occasionally optional or not required]. Do not reiterate the data or discussion. Can state hunches, inferences or speculations. Offer perspectives for future work.

VIII. Acknowledgements: Names people who contributed to the work, but did not contribute sufficiently to earn authorship. You must have permission from any individuals mentioned in the acknowledgements sections. 

IX. References:  Complete citations for any articles or other materials referenced in the text of the article.

• IMRD Cheatsheet (Carnegie Mellon) pdf.
• Adewasi, D. (2021 June 14).  What Is IMRaD? IMRaD Format in Simple Terms! . Scientific-editing.info. 
• Nair, P.K.R., Nair, V.D. (2014). Organization of a Research Paper: The IMRAD Format. In: Scientific Writing and Communication in Agriculture and Natural Resources. Springer, Cham. https://doi.org/10.1007/978-3-319-03101-9_2
• Sollaci, L. B., & Pereira, M. G. (2004). The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey.   Journal of the Medical Library Association : JMLA ,  92 (3), 364–367.
• Cuschieri, S., Grech, V., & Savona-Ventura, C. (2019). WASP (Write a Scientific Paper): Structuring a scientific paper.   Early human development ,  128 , 114–117. https://doi.org/10.1016/j.earlhumdev.2018.09.011

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved September 9, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

• Skip to Guides Search
• Skip to breadcrumb
• Skip to main content
• Skip to footer
• Skip to chat link
• Report accessibility issues and get help
• Go to Penn Libraries Home
• Go to Franklin catalog

Critical Writing Program: Climate Science and Action: Earth in Crisis - Fall 2024: Researching the White Paper

• Getting started
• News and Opinion Sites
• Academic Sources
• Grey Literature
• Substantive News Sources
• What to Do When You Are Stuck
• Understanding a citation
• Examples of Quotation
• Examples of Paraphrase
• Chicago Manual of Style: Citing Images
• Researching the Op-Ed
• Researching Prospective Employers
• Resume Resources
• Cover Letter Resources

Research the White Paper

Researching the white paper:.

The process of researching and composing a white paper shares some similarities with the kind of research and writing one does for a high school or college research paper. What’s important for writers of white papers to grasp, however, is how much this genre differs from a research paper.  First, the author of a white paper already recognizes that there is a problem to be solved, a decision to be made, and the job of the author is to provide readers with substantive information to help them make some kind of decision--which may include a decision to do more research because major gaps remain. 

Thus, a white paper author would not “brainstorm” a topic. Instead, the white paper author would get busy figuring out how the problem is defined by those who are experiencing it as a problem. Typically that research begins in popular culture--social media, surveys, interviews, newspapers. Once the author has a handle on how the problem is being defined and experienced, its history and its impact, what people in the trenches believe might be the best or worst ways of addressing it, the author then will turn to academic scholarship as well as “grey” literature (more about that later).  Unlike a school research paper, the author does not set out to argue for or against a particular position, and then devote the majority of effort to finding sources to support the selected position.  Instead, the author sets out in good faith to do as much fact-finding as possible, and thus research is likely to present multiple, conflicting, and overlapping perspectives. When people research out of a genuine desire to understand and solve a problem, they listen to every source that may offer helpful information. They will thus have to do much more analysis, synthesis, and sorting of that information, which will often not fall neatly into a “pro” or “con” camp:  Solution A may, for example, solve one part of the problem but exacerbate another part of the problem. Solution C may sound like what everyone wants, but what if it’s built on a set of data that have been criticized by another reliable source?  And so it goes. 

For example, if you are trying to write a white paper on the opioid crisis, you may focus on the value of  providing free, sterilized needles--which do indeed reduce disease, and also provide an opportunity for the health care provider distributing them to offer addiction treatment to the user. However, the free needles are sometimes discarded on the ground, posing a danger to others; or they may be shared; or they may encourage more drug usage. All of those things can be true at once; a reader will want to know about all of these considerations in order to make an informed decision. That is the challenging job of the white paper author.     
 The research you do for your white paper will require that you identify a specific problem, seek popular culture sources to help define the problem, its history, its significance and impact for people affected by it.  You will then delve into academic and grey literature to learn about the way scholars and others with professional expertise answer these same questions. In this way, you will create creating a layered, complex portrait that provides readers with a substantive exploration useful for deliberating and decision-making. You will also likely need to find or create images, including tables, figures, illustrations or photographs, and you will document all of your sources. 

Computer Science & Engineering Librarian

Connect to a Librarian Live Chat or "Ask a Question"

• Librarians staff live chat from 9-5 Monday through Friday . You can also text to chat: 215-543-7674
• You can submit a question 24 hours a day and we aim to respond within 24 hours 
• You can click the "Schedule Appointment" button above in librarian's profile box (to the left), to schedule a consultation with her in person or by video conference.  
• You can also make an appointment with a  Librarian by subject specialization . 
• Connect by email with a subject librarian

Find more easy contacts at our Quick Start Guide

• Next: Getting started >>
• Last Updated: Sep 10, 2024 9:20 AM
• URL: https://guides.library.upenn.edu/climatescience

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Advertisement

The Impact of Nomophobia: Exploring the Interplay Between Loneliness, Smartphone Usage, Self-control, Emotion Regulation, and Spiritual Meaningfulness in an Indonesian Context

• Published: 11 September 2024

Cite this article

• Triantoro Safaria   ORCID: orcid.org/0000-0003-3551-1460 1 ,
• Nofrans Eka Saputra 2 &
• Diana Putri Arini 3  

Nomophobia is characterized as an irrational fear or anxiety that arises when one is unable to use, contact, communicate, or access mobile phones. Previous research on nomophobia has been conducted mainly through an exploratory approach. Few studies have tested the theoretical model of nomophobia through a confirmatory analysis approach. Thus, this research contributes to filling the existing gap by testing a theoretical model of nomophobia. This cross-sectional study was conducted in Yogyakarta, Palembang, and Jambi, Indonesia. We used purposive sampling to recruit 689 students from various levels in those three cities to participate in this study. Specifically, the participants consisted of junior high school students ( n  = 245, 35.5%), high school students ( n  = 235, 34.2%), and college students ( n  = 209, 30.3%). Among them, 380 (55.2%) were women, and 309 (44.8%) were men. We used questionnaires to measure nomophobia, emotion regulation, self-control, spiritual meaningfulness, loneliness, and smartphone use. Data were analyzed using the structural equation model (SEM) analysis. Our findings revealed that emotional regulation, spiritual meaningfulness, and self-control had significant indirect effects on nomophobia. Furthermore, the intensity of smartphone use is a significant mediator that increases nomophobia in this model. Furthermore, the intensity of smartphone use is a significant mediator in this fit model. Future research should explore interventions that enhance emotional regulation, spiritual meaningfulness, and self-control to reduce nomophobia. Additionally, examining the specific mechanisms through which smartphone use mediates this relationship could provide deeper insights. Implementing educational programs on mindful smartphone usage and developing strategies to balance digital engagement may also prove beneficial.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Data Availability

Data are publicly available from Zenodo at https://zenodo.org/records/11443599 .

Adawi, M., Zerbetto, R., Re, T. S., Bisharat, B., Mahamid, M., Amital, H., Del Puente, G., & Bragazzi, N. L. (2019). Psychometric properties of the brief symptom inventory in homophobic subjects: Insights from primary confirmatory factor, exploratory factor, and clustering analyses in a sample of healthy Italian volunteers. Psychology Research and Behavior Management, 12 , 145–154. https://doi.org/10.2147/PRBM.S173282

Article   PubMed   PubMed Central   Google Scholar  

Adler, N. E. (2015). Disadvantage, self-control, and health. Proceedings of the National Academy of Sciences of the United States of America, 112 (33), 10078–10079. https://doi.org/10.1073/pnas.1512781112

Agbaria, Q., Mahamid, F., Ziya Berte, D., 2017. Social support, self-control, religiousness and engagement in high risk behaviors among adolescents. International Journal of Indian Psychology 4 (4), 13–33. https://doi.org/10.25215/0404.142

Aguilera-Manrique, G., Márquez-Hernández, V. V., Alcaraz-Córdoba, T., Granados-Gámez, G., Gutiérrez-Puertas, V., & Gutiérrez-Puertas, L. (2018). The relationship between nomophobia and distraction associated with smartphone use among nursing students in their clinical practicum. PLoS ONE, 13 (8), 1–14. https://doi.org/10.1371/journal.pone.0202953

Article   Google Scholar  

Ai, A. L., Hall, D., Pargament, K., et al. (2013). Posttraumatic growth in patients who survived cardiac surgery: The predictive and mediating roles of faith-based factors. Journal of Behavioral Medicine, 36 , 186–198. https://doi.org/10.1007/s10865-012-9412-6

Article   PubMed   Google Scholar  

Akn, A., Arslan, S., Arslan, N., Uysal, R., Sahran, M., 2015. Self-control/management and internet addiction. International Online Journal of Educational Sciences 7 (3), 95–100. https://doi.org/10.15345/iojes.2015.03.016

Aldao, A. (2013). The future of emotion regulation research: Capturing context. Perspectives on Psychological Science, 8 , 155–172. https://doi.org/10.1177/1745691612459518

Aldao, A., Nolen-Hoeksema, S., & Schweizer, S. (2010). Emotion-regulation strategies across psychopathology: A meta-analytic review. Clinical Psychology Review, 30 (2), 217–237.

Ali, A., Muda, M., Ridzuan, A. R., Nuji, M. N. N., Izzamuddin, M. H. M., & Latiff, D. I. A. (2017). The relationship between phone usage factors and nomophobia. Advanced Science Letters, 23 , 7610–7613.

Anderson, J. C., & Gerbing, D. W. (1988). Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin, 103 (3), 411–423. https://doi.org/10.1037/0033-2909.103.3.411

Argumosa-Villar, L.; Boada-Grau, J.; Vigil-Colet, A. (2017). Exploratory investigation of theoretical predictors of nomophobia using the Mobile Phone Involvement Questionnaire (PIQ). Journal of Adolescence, 56 , 127–135. [CrossRef]

Arpaci, I., Baloglu, M., Özteke, H. I., & Kesici, S. (2017). Individual differences in the relationship between attachment and nomophobia among college students: The mediating role of mindfulness. Journal of Medical Internet Research, 2017 (19), e404.

Arpaci, I.; Baloglu, M.; Kesici, ¸S. (2019). A multigroup analysis of the effects of individual differences in mindfulness on nomophobia. Inf. Dev Information Development., 35 , 333–341. [CrossRef]

Arpaci, I. (2022). Gender differences in the relationship between problematic Internet use and nomophobia. Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues, 41 (9), 6558–6567. https://doi.org/10.1007/s12144-020-01160-x

Avci, E. (2022). The difference between gender in terms of nomophobia in Turkey: A meta-analysis. The European Research Journal , 8(1), 74–83. https://doi.org/10.18621/eurj.865153 .

Baumeister, R. F., Vohs, K. D., Tice, D. M., Baumeister, R. F., Vohs, K. D., & Tice, D. M. (2007). The strength model of self control. Current Directions in Psychological Science, 16 (6), 351–355. https://doi.org/10.1111/j.1467-8721.2007.00534.x

Bekaroglu, E., & Yilmaz, T. (2020). Nomophobia: Differential diagnosis and treatment. Current Approaches in Psychiatry, 12 (1), 131–142. https://doi.org/10.18863/pgy.528897

Bian, M., & Leung, L. (2015). Linking loneliness, shyness, smartphone addiction symptoms, and patterns of smartphone use to social capital. Social Science Computer Review, 33 (1), 61–79. https://doi.org/10.1177/0894439314528779

Billieux, J., Van Der Linden, M., & Rochat, L. (2008). The role of impulsivity in actual and problematic use of the mobile phone. Applied Cognitive Psychology, 22 (9), 1195–1210. https://doi.org/10.1002/acp.1429

Billieux, J., Maurage, P., Lopez-Fernandez, O., Kuss, D. J., & Griffiths, M. D. (2015). Can disordered mobile phone use be considered a behavioral addiction? An update on current evidence and a comprehensive model for future research. Current Addiction Reports, 2 (2), 154–162.

Bonanno, G. A., & Burton, C. L. (2013). Regulatory flexibility: An individual differences perspective on coping and emotion regulation. Perspectives on Psychological Science, 8 , 591–612. https://doi.org/10.1177/174569161350411

Bragazzi, N. L., & Del Puente, G. (2014). A proposal for including nomophobia in the new DSM-V. Psychology Research and Behavior Management, 7 , 155–160.

Buctot, D. B., Kim, N., & Kim, S. H. (2020). The role of nomophobia and smartphone addiction in the lifestyle profiles of junior and senior high school students in the Philippines. Social Sciences & Humanities Open, 2 (1), 100035. https://doi.org/10.1016/j.ssaho.2020.100035

Byrne, B. M. (2001a). Structural equation modeling: Perspectives on the present and the future. International Journal of Testing, 1 (3–4), 327–334. https://doi.org/10.1080/15305058.2001.9669479

Byrne, B. M. (2001b). Structural equation modeling with AMOS, EQS, and LISREL: Comparative approaches to testing for the factorial validity of a measuring instrument. International Journal of Testing, 1 (1), 55–86. https://doi.org/10.1207/S15327574IJT0101_4

Cameron, L. D., & Overall, N. C. (2017). Suppression and expression as distinct emotion-regulation processes in daily interactions: Longitudinal and meta-analyses. Emotion, 18 , 465–480. https://doi.org/10.1037/emo00003

Carver, C. S., & Schreier, M. (2000). “On the structure of behavioral self-regulation,” in Handbook of Self-Regulation, eds M. Boekaerts, P. Pintrich, and M. Zeidner (San Diego, CA: Academic Press), 42–85.

Casey, D. M., Williams, R. J., Mossière, A. M., Schopflocher, D. P., el Guebaly, N., Hodgins, D. C., Smith, G. J., & Wood, R. T. (2011). The role of family, religiosity, and behavior in adolescent gambling. Journal of Adolescence, 34 , 841–851. https://doi.org/10.1016/j.adolescence.2011.02.002

Chiu, S. I. (2014). The relationship between life stress and smartphone addiction on Taiwanese university student: A mediation model of learning self-efficacy and social self-efficacy. Computers in Human Behavior, 34 , 49–57.

Clark, E. M., Williams, B. R., Huang, J., Roth, D. L., & Holt, C. L. (2018). A longitudinal study of religiosity, spiritual health locus of control, and health behaviors in a national sample of African Americans. Journal of Religion and Health, 57 (6), 2258. https://doi.org/10.1007/s10943-017-0548-0

Cosmides, L., and Tooby, J. (2000). “Evolutionary psychology and the emotions” in Handbook of emotions, eds M. Lewis and J. M. Haviland-Jones (New York, NY: Guilford Press), 91–115.

Dai, C., Tai, Z., & Ni, S. (2021). Smartphone use and psychological well-being among college students in China: A qualitative assessment. Frontiers in Psychology, 12 (September), 1–13. https://doi.org/10.3389/fpsyg.2021.708970

Damasio, A. (1994). L’erreur de Descartes . Odile Jacob.

Google Scholar  

Darvishi, M., Noori, M., Nazer, M. R., Sheikholeslami, S., & Karimi, E. (2019). Investigating different dimensions of nomophobia among medical students: A cross-sectional study. Open Access Macedonian Journal of Medical Sciences, 7 (4), 573–578. https://doi.org/10.3889/OAMJMS.2019.138

de Ridder, D. T. D., Lensvelt-Mulders, G., Finkenauer, C., Stok, F. M., & Baumeister, R. F. (2012). Taking stock of self-control: A meta-analysis of how trait self-control relates to a wide range of behaviors. Personality and Social Psychology Review, 16 (1), 76–99. https://doi.org/10.1177/1088868311418749

Debnam, K. J., Bradshaw, C. P., Milam, A. J., Mullen, M. M., & Lacey, K. (2017). The moderating role of spirituality in the association between stress and substance use among adolescents: Differences by gender. Journal of Youth Adolescence . https://doi.org/10.1007/s10964-017-0687-3

Dixit, S., Shukla, H., Bhagwat, A., Bindal, A., Goyal, A., Zaidi, A. K., & Shrivastava, A. (2010). A study to evaluate mobile phone dependence among students of a medical college and associated hospital of central India. Indian Journal of Community Medicine, 35 , 339–341.

Dongre, A. S., Inamdar, I. F., & Gattani, P. L. (2017). Nomophobia: A study to evaluate mobile phone dependence and impact of cell phone on health. National Journal of Community Medicine, 8 (11), 688–693.

Dragoi, V., & Staddon, J. E. R. (1999). The dynamics of operant conditioning. Psychological Review, 106 (1), 20–61. https://doi.org/10.1037/0033-295X.106.1.20

Durak, H. Y. (2018). Investigation of nomophobia and smartphone addiction predictors among adolescents in Turkey: Demographic variables and academic performance Investigation of nomophobia and smartphone addiction variables and academic performance. The Social Science Journal October. https://doi.org/10.1016/j.soscij.2018.09.003

Ekşi, H., & Kardaş, S. (2017). Spiritual well-being: Scale development and validation. Spiritual Psychology and Counseling, 2(1), 73–88 https://dergipark.org.tr/en/pub/spiritualpc/issue/29971/323508 .

Elhai, J. D., Dvorak, R. D., Levine, J. C., & Hall, B. J. (2017). Problematic smartphone use: A conceptual overview and systematic review of relations with anxiety and depression psychopathology. Journal of Affective Disorders, 207 , 251–259.

Elhai, J. D., Tiamiyu, M. F., Weeks, J. W., Levine, J. C., Picard, K. J., & Hall, B. J. (2018a). Depression and emotion regulation predict objective smartphone use measured over one week. Personality and Individual Differences, 133 , 21–28.

Elhai, J. D., Levine, J. C., Hall, B. J., & Dvorak, R. D. (2018b). Types of smartphone usage and relations with problematic smartphone behaviors: The role of content consumption vs. social smartphone use. Cyberpsychology Behavior and Social Networking, 21 (11), 725–732.

A Enez Darcin S Kose CO Noyan S Nurmedov O Yılmaz N Dilbaz 2016 Smartphone addiction and its relationship with social anxiety and loneliness Behaviour and Information Technology 35 520 525

Fabes, R. A., Eisenberg, N., Jones, S., Smith, M., Guthrie, I., Poulin, R., Shepard, S., & Friedman, J. (1999). Regulation, emotionality, and preschoolers’ socially competent peer interactions. Child Development, 70 (2), 432–442. https://doi.org/10.1111/1467-8624.00031

Fishbach, A., and Ferguson, M. J. (2007). “The goal construct in social psychology,” in Social psychology: Handbook of basic principles, 2nd Edn, eds A. W. Kruglanski and E. T. Higgins (New York, NY: The Guilford Press), 490–515

Frijda, N. H. (1987). Emotion, cognitive structure, and action tendency. Cogn. Emot. 1, 115–143. oi: https://doi.org/10.1080/02699938708408043

Garland, E. L., Hanley, A. W., Riquino, M. R., Reese, S. E., Baker, A. K., Salas, K., Yack, B. P., Bedford, C. E., Bryan, M. A., Atchley, R., Nakamura, Y., Froeliger, B., & Howard, M. O. (2019). Mindfulness-oriented recovery enhancement reduces opioid misuse risk via analgesic and positive psychological mechanisms: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 87 (10), 927–940. https://doi.org/10.1037/ccp0000390

Gatzounis, R., Schrooten, M. G., Crombez, G., & Vlaeyen, J. W. (2012). Operant learning theory in pain and chronic pain rehabilitation. Current Pain and Headache Reports, 16 (2), 117–126. https://doi.org/10.1007/s11916-012-0247-1

Geng, Y., Gu, J., Wang, J., & Zhang, R. (2021). Smartphone addiction and depression, anxiety: The role of bedtime procrastination and self-control. Journal of Affective Disorders, 293 (July), 415–421. https://doi.org/10.1016/j.jad.2021.06.062

Gratz, K. L., & Roemer, L. (2004). Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the difficulties in emotion regulation scale. Journal of Psychopathology and Behavioral Assessment, 26 , 41–54. https://doi.org/10.1023/B:JOBA.0000007455.08539.94

Gross, J. J. (2015). Emotion regulation: Current status and future prospects. Psychological Inquiry, 26 (1), 1–26.

Gross, J. J., & Thompson, R. A. (2007). Emotion regulation: Conceptual foundations. in Handbook of Emotion Regulation, ed J. J. Gross (New York, NY: Guilford Press), 3–24

Gross, J. J., Richards, J. M., & John, O. P. (2006). Emotion regulation in everyday life. In D. K. Snyder, J. Simpson, & J. N. Hughes (Eds.), Emotion regulation in couples and families: Pathways to dysfunction and health (pp. 13–35). American Psychological Association. https://doi.org/10.1037/11468-001

Grubbs, J. B., Exline, J. J., Pargament, K. I., Volk, F., & Lindberg, M. J. (2017). Internet pornography use, perceived addiction, and religious/spiritual struggles. Archives of Sexual Behavior, 46(6), 17331745. Available from https://doi.org/10.1007/s10508-016-0772-9

Gruber, J., Mauss, I. B., & Tamir, M. (2011). A dark side of happiness? How, when, and why happiness is not always good. Perspectives on Psychological Science, 6 , 222–233. https://doi.org/10.1177/1745691611406927

Güner, T. A., & Demir, İ. (2021). Relationship between smartphone addiction and nomophobia, anxiety, self-control in high school students. The Turkish Journal on Addictions , 1–7. https://doi.org/10.5152/ADDICTA.2021.21089

Gutiérrez-Puertas, L., Márquez-Hernández, V. V., & Aguilera-Manrique, G. (2016). Adaptation and validation of the Spanish version of the nomophobia questionnaire in nursing studies. CIN - Computers Informatics Nursing, 34 (10), 470–475. https://doi.org/10.1097/CIN.0000000000000268

Gutiérrez-Puertas, L., Márquez-Hernández, V. V., São-Romão-Preto, L., Granados-Gámez, G., Gutiérrez-Puertas, V., & Aguilera-Manrique, G. (2019). Comparative study of nomophobia among Spanish and Portuguese nursing students. Nurse Education in Practice, 34 , 79–84. https://doi.org/10.1016/j.nepr.2018.11.010

Haines, S. J., Gleeson, J., Kuppens, P., Hollenstein, T., Ciarrochi, J., Labuschagne, I., et al. (2016). The difference between strategy and situation fit in emotion regulation in daily life is associated with well-being. Psychological Science, 27 , 1651–1659. https://doi.org/10.1177/0956797616669086

Hair, J. F., Black, B., Babin, B., Anderson, R. E., & Tatham, R. L. (2005). Multivariate data analysis (6th ed.). Prentice Hall.

Hayes, A. F. (2009). Beyond Baron and Kenny: Statistical mediation analysis in the new millennium. Communication Monographs, 76 (4), 408–420. https://doi.org/10.1080/03637750903310360

Heffner, K. L., Waring, M. E., Roberts, M. B., Eaton, C. B., & Gramling, R. (2011). Social isolation, C-reactive protein, and coronary heart disease mortality among community-dwelling adults. Social Science & Medicine, 72 , 1482–1488. https://doi.org/10.1016/j.socscimed.2011.03.016

Holmbeck, G. N. (2002). Post hoc probing of significant moderating and mediating effects in studies of pediatric populations. Journal of Pediatric Psychology Special Issue on Methodology and Design, 27 (1), 87–96.

Horton, G. E., Luna, N., & Malloy, T. (2016). Associations between spirituality, meaning in life, and depressive disorders among asample of individuals in treatment for substance use. Journal of Spirituality in Mental Health, 18 (4), 283–299.

Hwang, T.-J., Rabheru, K., Peisah, C., Reichman, W., & Ikeda, M. (2020). Loneliness and social isolation during the COVID-19 pandemic. International Psychogeriatrics, 32 (10), 1217–1220.

International Telecommunications Union (2014). Measuring the Information Society Report 2014. International Telecommunication Union Place des Nations CH-1211 Geneva Switzerland.

International Test Commission. (2017b). The ITC guidelines for translating and adapting tests (Second edition) . ITC Guidelines.

ITC Guidelines for Translating and adapting tests (Second Edition). (2017a). International Journal of Testing , 18 (2), 101–134. https://doi.org/10.1080/15305058.2017.1398166

Jahagirdar, V., Rama, K., Soppari, P., & Kumar, M. V. (2021). Mobile phones: Vital addiction or lethal addiction? Mobile phone usage patterns and assessment of mobile addiction among undergraduate medical students in Telangana, India. Journal of Addiction, 2021 , 1–6. https://doi.org/10.1155/2021/8750650

Jahrami, H., Trabelsi, K., Boukhris, O., Hussain, J. H., Alenezi, A. F., Humood, A., Saif, Z., Pandi-Perumal, S. R., & Seeman, M. V. (2023). The prevalence of mild, moderate, and severe nomophobia symptoms: A systematic review, meta-analysis, and meta-regression. Behavioral Sciences, 13 (1), 35. https://doi.org/10.3390/bs13010035

Jeste, D. V., Lee, E. E., & Cacioppo, S. (2020). Battling the modern behavioral epidemic of loneliness: Suggestions for research and interventions. JAMA Psychiatry . https://doi.org/10.1001/jamapsychiatry.2020.0027

M Kara K Baytemir F Inceman-kara M Kara 2019 Duration of daily smartphone usage as an antecedent of nomophobia: Exploring multiple mediation of loneliness and anxiety multiple mediation of loneliness and anxiety Behavior & Information Technology 1 14 https://doi.org/10.1080/0144929X.2019.1673485

Kardefelt-Winther, D. (2014). A conceptual and methodological critique of internet addiction research: Toward a model of compensatory internet use. Computers in Human Behavior, 31 , 351–354. https://doi.org/10.1016/j.chb.2013.10.059

Katz, E., Blumler, J. G., & Gurevitch, M. (1973). Uses and gratifications research. The Public Opinion Quarterly, 37 (4), 509–523.

Kaur, A., Sharma, P., & Manu. (2015). A descriptive study to assess the risk of developing nomophobia among students of selected nursing colleges Ludhiana, Punjab. International Journal of Psychiatric Nursing , 1(2),1. https://doi.org/10.5958/2395-180X.2015.00051.1

Kaviani, F., Robards, B., Young, K. L., & Koppel, S. (2020). Nomophobia: Is the fear of being without a smartphone associated with problematic use? International Journal of Environmental Research and Public Health, 17 (17), 1–19. https://doi.org/10.3390/ijerph17176024

Keltner, D., & Kring, A. M. (1998). Emotion, social function, and psychopathology. Review of General Psychology, 2 , 320–342. https://doi.org/10.1037/1089-2680.2.3.320

Kim, J., LaRose, R., & Peng, W. (2009). Loneliness as the cause and the effect of problematic internet use: The relationship between internet use and psychological well-being. Cyberpsychology & Behavior : THe Impact of the Internet, Multimedia and Virtual Reality on Behavior and Society, 12 (4), 451–455. https://doi.org/10.1089/cpb.2008.0327

Kim, H. J., & Lee, J. M. (2018). Determinants of mobile digital information usage among senior consumers: Focusing on secondary digital divide. Family and Environment Research, 56 (4), 489–506.

Kinanti, D. R., & Hartati, N. (2017). The effect of self esteem, social support, and religious orientation toward cyberbullying intention on adolescent at senior high school. Tazkiyah: Journal of Psychology 22(2), 191–202. https://doi.org/10.15408/tazkiya.v22i2.8400

King, A. L. S., Valença, A. M., & Nardi, A. E. (2010). Nomophobia: The mobile phone in panic disorder with agoraphobia: Reducing phobias or worsening of dependence? Cognitive and Behavioral Neurology, 23 (1), 52–54. https://doi.org/10.1097/WNN.0b013e3181b7eabc

King, A. L. S., Valença, A. M., Silva, A. C. O., Baczynski, T., Carvalho, M. R., & Nardi, A. E. (2013). Nomophobia: Dependency on virtual environments or social phobia? Computers in Human Behavior, 2013 (29), 140–144.

King, A. L. S., Martins, A., Valença, A. M., Cardoso, A., Sancassiani, F., Machado, S., & Egidio, A. (2014). Nomophobia: Impact of cell phone use interfering with symptoms and emotions of individuals with panic disorder compared with a control group. Clinical Practice & Epidemiology in Mental Health, 2014 (10), 28–35.

Kirsch, I., Lynn, S. J., Vigorito, M., & Miller, R. R. (2004). The role of cognition in classical and operant conditioning. Journal of Clinical Psychology, 60 (4), 369–392. https://doi.org/10.1002/jclp.10251

Kline, R. B. (1998). Principles and practice of structural equation modeling . Guilford Press.

Kobayashi, L. C., & Steptoe, A. (2018). Social isolation, loneliness, and health behaviors at older age: Longitudinal cohort study. Annals of Behavioral Medicine, 52 , 582–593. https://doi.org/10.1093/ABM/kax033

Koenig, H. G. (2014). Religion, spirituality, and health: A review and update. Advances in Mind-Body Medicine, 29 (3), 19–26.

Koob, G. F., & Volkow, N. D. (2010). Neurocircuitry of addiction. Neuropsychopharmacology, 35 (1), 217–238.

Kuiper, J. S., et al. (2015). Social relationships and risk of dementia: A systematic review and meta-analysis of longitudinal cohort studies. Aging Research Reviews, 22 , 39–57. https://doi.org/10.1016/j.arr.2015.04.006

Kuscu, T. D., Gumustas, F., & Arman, A. R. (2020). The relationship between nomophobia and psychiatric symptoms in adolescents. International Journal of Psychiatry in Clinical Practice , 0 (0), 1–6. https://doi.org/10.1080/13651501.2020.1819334 .

Kwon, M., Kim, D. J., Cho, H., & Yang, S. (2013). The smartphone addiction scale: Development and validation of a short version for adolescents. PLoS ONE, 8 (12), e83558.

Landor, A., Simons, R. L., Gibbons, F. X., Simons, L. G., & Brody, G. H. (2011). The role of religiosity in the relationship between parents, peers, and adolescent risky sexual behavior. Journal of Youth Adolescence, 40 , 296–309. https://doi.org/10.1007/s10964-010-9598

Lee, S. Y. (2014). Examining the factors that influence early adopters’ smartphone adoption: The case of college students. Telematics and Informatics, 31 (2), 308–318.

Lee, S., Kim, M., Mendoza, J. S., & McDonough, I. M. (2018). Addicted to cellphones: Exploring the psychometric properties between the nomophobia questionnaire and obsessiveness in college students. Heliyon, 4 (11), e00895. https://doi.org/10.1016/j.heliyon.2018.e00895

Lee, E. E., Depp, C., Palmer, B. W., Glorioso, D., Daly, R., Liu, J., Tu, X. M., Kim, H. C., Tarr, P., Yamada, Y., & Jeste, D. V. (2019). High prevalence and adverse health effects of loneliness in community-dwelling adults across the lifespan: Role of wisdom as a protective factor. International Psychogeriatrics, 31 (10), 1447–1462. https://doi.org/10.1017/S10416102

Leigh-Hunt, N., et al. (2017). This is an overview of systematic reviews on the public health consequences of social isolation and loneliness. Public Health, 152 , 157–171. https://doi.org/10.1016/j.puhe.2017.07.035

León-Mejía, A. C., Gutiérrez-Ortega, M., Serrano-Pintado, I., & González-Cabrera, J. (2021). A systematic review on nomophobia prevalence: Surfacing results and standard guidelines for future research. PLoS ONE, 16(5 May), 1–22. https://doi.org/10.1371/journal.pone.0250509

Lin, Y. (2011). A study on the negative effects of smartphones on social life and wellbeing. International Journal of Business and Social Science, 2 (18), 19–23.

Lin, Y. H., & Lu, H. P. (2011). Why people use social networking sites: An empirical study integrating network externalities and motivation theory. Computers in Human Behavior, 27 (3), 1152–1161.

Liu, W., Chen, J.-S., Gan, W. Y., Poon, W. C., Tung, S. E. H., Lee, L. J., Xu, P., Chen, I.-H., Griffiths, M. D., & Lin, C.-Y. (2022). Associations of problematic internet use, weight-related self-stigma, and nomophobia with physical activity: Findings from Mainland China, Taiwan, and Malaysia.  International Journal of Environmental Research and Public Health, 19 (19), 12135. https://doi.org/10.3390/ijerph191912135

Luminet, O., and Curci, A. (2009). Flashbulb memories: New issues and new perspectives. Hove; New York: Psychology Press

Mauss, I. B., Evers, C., Wilhelm, F. H., & Gross, J. J. (2006). How to bite your tongue without blowing your top: Implicit evaluation of emotion regulation predicts affective responding to anger provocation. PSPB, 32 (5), 1–14. https://doi.org/10.1177/0146167205283841

Mendoza, J. S., Pody, B. C., Lee, S., Kim, M., & McDonough, I. M. (2018). The effect of cellphones on attention and learning: The influences of time, distraction, and nomophobia. Computers in Human Behavior, 86 , 52–60. https://doi.org/10.1016/j.chb.2018.04.027

Mertkan, D.; Burcin, N.; Sezen-Gultekin, G.; Gemikonakli, O. (2018). Relationship between nomophobia and fear of missing out among Turkish university students. Cypriot J. Educ. Sci. 2018, 13, 549–561. [CrossRef].

Metcalfe, J., Mischel, W., 1999. A hot/cool-system analysis of delay of gratification: Dynamics of willpower. Psychol. Rev. 106 (1), 3–19. https://doi.org/10.1037/0033-295X.106.1.3

Meyers, L. S., Gamst, G., & Guarino, A. J. (2006). Applied multivariate research: Design and interpretation . Sage Publications Inc.

Millgram, Y., Sheppes, G., Kalokerinos, E. K., Kuppens, P., & Tamir, M. (2019). Do the ends dictate the means in emotion regulation? Journal of Experimental Psychology: General, 148 , 80–96. https://doi.org/10.1037/xge0000477

Mir, R., & Akhtar, M. (2020). Effect of nomophobia on the anxiety levels of undergraduate students. Journal of the Pakistan Medical Association, 70 (9), 1492–1497. https://doi.org/10.5455/JPMA.31286

Moffitt, T. E., Arseneault, L., Belsky, D., Dickson, N., Hancox, R. J., & Harrington, H. (2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the National Academic of Sciences of the United States of America, 108 (7), 2693–2698. https://doi.org/10.1073/pnas.1010076108

Moreno-Guerrero, A. J., Belmonte, J., Romero-Rodríguez, J. M., & Rodríguez-García, A. M. (2020). Nomophobia: Impact of cell phone use and time to rest among teacher students. Heliyon, 6 (5), e04084. https://doi.org/10.1016/j.heliyon.2020.e04084

Notara, V., Vagka, E., Gnardellis, C., & Lagiou, A. (2021). The emerging phenomenon of nomophobia in young adults: A systematic review study. Addiction and Health, 13 (2), 120–136. https://doi.org/10.22122/ahj.v13i2.309

Ohman, A., Flykt, A., & Esteves, F. (2001). Emotion drives attention: Detecting the snake in the grass. Journal of Experimental Psychology, 130 , 466–478. https://doi.org/10.1037/0096-3445.130.3.466

Olivencia-Carrión, M. A., Ferri-García, R., del Rueda, M., Jiménez-Torres, M. G., & López-Torrecillas, F. (2018). Temperament and characteristics related to nomophobia. Psychiatry Research, 266 (May), 5–10. https://doi.org/10.1016/j.psychres.2018.04.056

Olson, M. M., Trevino, D. B., Geske, J. A., & Vanderpool, H. (2012). Religious coping and mental health outcomes: An exploratory study of socioeconomically disadvantaged patients. Explore (New York, N.Y.) , 8 (3), 172–176. https://doi.org/10.1016/j.explore.2012.02.005 .

Online, M. (2008). Nomophobia is the fear of being out of mobile phone contact – And it’s the plague of our 24/7 age. Retrieved, 15 (05), 2022.

Opatz, J. P. (1986). Stevens point: A longstanding program for students at a Midwestern university. American Journal of Health Promotion, 1 (1), 60–67.

Oulasvirta, A., Rattenbury, T., Ma, L., & Raita, E. (2012). Habits make smartphone use more pervasive. Personal and Ubiquitous Computing, 16 (1), 105–114.

Ownsworth, T. L., McFarland, K., & Young, R. M. (2002). The investigation of factors underlying deficits in selfawareness and self-regulation. Brain Injury, 16 (4), 291–309. https://doi.org/10.1080/02699050110103986

Ozdemir, B., Cakir, O., & Hussain, I. (2018). Prevalence of nomophobia among university students: A comparative study of Pakistani and Turkish undergraduate students. Eurasia Journal of Mathematics, Science and Technology Education , 14 (4), 1519–1532. https://doi.org/10.29333/ejmste/84839

Pargament, K. I. (2007). The psychology of religion and coping: Theory, research, practice . Guilford Press.

Parrott, W. G. (1993). Beyond hedonism: Motives for inhibiting good moods and for maintaining bad moods. In D. M. Wegner & J. W. Pennebaker (Eds.), Handbook of mental control (pp. 278–305). Prentice-Hall Inc.

Pessoa, L., Japee, S., & Ungerleider, L. G. (2005). Visual awareness and the detection of fearful faces. Emotion, 5 , 243–247. https://doi.org/10.1037/1528-3542.5.2.243

Phelps, E. A. (2006). Emotion and cognition: Insights from studies of the human amygdala. Annual Review of Psychology, 57 , 27–53. https://doi.org/10.1146/annurev.psych.56.091103.070234

Phillips, R. E., & Stein, C. H. (2007). God’s will, God’s punishment, or God’s limitations? Religious coping strategies reported by young adults living with serious mental illness. Journal of Clinical Psychology, 63 (6), 529–540.

Pieper, J. Z. T. (2004). Religious coping in highly religious psychiatric inpatients. Mental Health, Religion & Culture, 7 (4), 349–363. https://doi.org/10.1080/13674670410001719805

Pivetta, E., Harkin, L., Billieux, J., Kanjo, E., & Kuss, D. J. (2019). Problematic smartphone use: An empirically validated model. Computers in Human Behavior, 100 (May), 105–117. https://doi.org/10.1016/j.chb.2019.06.013

Przybylski, A. K., & Weinstein, N. (2017). A large-scale test of the goldilocks hypothesis: Quantifying the relations between digital-screen use and the mental well-being of adolescents. Psychological Science, 28 (2), 204–215.

Qiufeng, G., Fu, N., Xiang, Y., Jia, G., & Wu, S. (2021). Self-esteem and addictive smartphone use: The mediator role of anxiety and the moderator role of self-control. Children and Youth Services Review, 124 (5), 105990. https://doi.org/10.1016/j.childyouth.2021.105990

Rahmati, S. (2017). The relationship between psychological capital and spirituality with internet addiction among students at Jondishapuor University of Medical Sciences. Medical Ethics Journal, 10 (38), 7–17.

Ramirez, S. P., Macêdo, D. S., Sales, P. M., Figueiredo, S. M., Daher, E. F., Araújo, S. M., Pargament, K. I., Hyphantis, T. N., & Carvalho, A. F. (2012). The relationship between religious coping, psychological distress and quality of life in hemodialysis patients. Journal of Psychosomatic Research, 72 (2), 129–135. https://doi.org/10.1016/j.jpsychores.2011.11.012

Ramos-Soler, I., López-Sánchez, C., & Quiles-Soler, M. C. (2017). Adaptación y validación de la escala de nomofobia de Yildirim y Correia en estudiantes españoles de la Educación Secundaria Obligatoria. Health and Addictions/Salud y Drogas , 17 (2), 201–213. https://doi.org/10.21134/haaj.v17i2.332

Rodrıguez-Garcia, A. M., Moreno-Guerrero, A. J., & Lopez, B. J. (2020). Nomophobia: An individual’s growing fear of being without a smartphone-A systematic literature review. International Journal of Environmental Research and Public Health, 17 (2), 580. https://doi.org/10.3390/ijerph17020580

Roseman, I. J., Wiest, C., & Swartz, T. S. (1994). Phenomenology, behaviors, and goals differentiate discrete emotions. Journal of Personality and Social Psychology, 67 , 206–221. https://doi.org/10.1037/0022-3514.67.2.206

Rosen, L. D., Carrier, L. M., & Cheever, N. A. (2013a). Facebook and texting made me do it: Media-induced task-switching while studying. Computers in Human Behavior, 29 (3), 948–958.

Rosen, L. D., Whaling, K., Carrier, L. M., Cheever, N. A., & Rokkum, J. (2013b). The media and technology usage and attitudes scale: An empirical investigation. Computers in Human Behavior, 29 (6), 2501–2511.

Rosmarin, D. H., Bigda-Peyton, J. S., Öngur, D., Pargament, K. I., & Björgvinsson, T. (2013). Religious coping among psychotic patients: Relevance to suicidality and treatment outcomes. Psychiatry Research, 210 (1), 182–187. https://doi.org/10.1016/j.psychres.2013.03.023

Rudolph, K. D., Troop-Gordon, W., & Llewellyn, N. (2013). Interactive contributions of self-regulation deficits and social motivation to psychopathology: Unraveling divergent pathways to aggressive behavior and depressive symptoms. Development and Psychopathology, 25 (2), 407–418. https://doi.org/10.1017/S0954579412001149

Ruggiero, T. E. (2000). Uses and gratifications theory in the 21st century. Mass Communication & Society, 3 (1), 3–37.

Russell, D., Peplau, L. A., & Cutrona, C. E. (1980). The revised UCLA Loneliness Scale: Concurrent and discriminant validity evidence. Journal of Personality and Social Psychology, 39 (3), 472–480. https://doi.org/10.1037/0022-3514.39.3.472

Salehan, M., & Negahban, A. (2013). Social networking on smartphones: When mobile phones become addictive. Computers in Human Behavior, 29 (6), 2632–2639.

Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, Cognition and Personality, 9 (3), 185–211. https://doi.org/10.2190/DUGG-P24E-52WK-6CDG

SecurEnvoy, 2012 Sixty-six percent of the population suffers from nomophobia due to the fear of being without their phone Retrieved 12 05 14

Servidio, R. (2021). Self-control and problematic smartphone use among Italian University students: The mediating role of the fear of missing out and of smartphone use patterns. Current Psychology, 40 , 4101–4111.

Sheldrake, P. (2013). The new SCM dictionary of Christian Spirituality . SCM Press.

Sheppes, G., Scheibe, S., Suri, G., & Gross, J. J. (2011). Emotion-regulation choice. Psychological Science, 22 (11), 1391–1396. https://doi.org/10.1177/0956797611418350

Shim, J. Y. (2019). Christian spirituality and smartphone addiction in adolescents: A comparison of high-risk, potential-risk, and normal control groups. Journal of Religion and Health, 58 (4), 1272–1285. https://doi.org/10.1007/s10943-018-00751-0

Shmueli, D., & Prochaska, J. J. (2012). A test of positive affect induction for countering self-control depletion in cigarette smokers. Psychology of Addictive Behaviors, 26 (1), 157.

Skinner, B. F. (1984). The evolution of behavior. Journal of the Experimental Analysis of Behavior, 41 , 217–221. https://doi.org/10.1901/jeab.1984.41-217

Skinner, B. F. (1985). Cognitive science and behaviorism. British Journal of Psychology, 76 , 291–301. https://doi.org/10.1111/j.2044-8295.1985.tb01953.x

Skues, J., Williams, B., Oldmeadow, J., & Wise, L. (2016). The effects of boredom, loneliness, and distress tolerance on problem internet use among university students. International Journal of Mental Health and Addiction, 14 , 167–180.

Society for the Study of Addiction. (2017). Briefing paper on cognitive–behavioral therapy (CBT) for substance use disorders. Retrieved from https://www.addiction-ssa.org/resources/briefing-paper-on-cognitive-behavioural-therapy-cbt-for-substance-use-disorders/

Stinson, J. D., Becker, J. V., & Sales, B. D. (2008). Self-regulation and the etiology of sexual deviance: evaluating causal theory. Violence and Victims, 23 (1), 35–51. https://doi.org/10.1891/0886-6708.23.1.35

Susskind, J. M., Lee, D. H., Cusi, A., Feiman, R., Grabski, W., & Anderson, A. K. (2008). Expressing fear enhances sensory acquisition. Nature Neuroscience, 11 , 843–850. https://doi.org/10.1038/nn.2138

Tamir, M., Schwartz, S. H., Cieciuch, J., Riediger, M., Torres, C., Scollon, C., Dzokoto, V., Zhou, X., & Vishkin, A. (2016). Desired emotions across cultures: A value-based account. Journal of Personality and Social Psychology, 111 (1), 67–82. https://doi.org/10.1037/pspp0000072

Tams, S., Legoux, R., & Léger, P.M. (2018). Smartphone withdrawal creates stress: A moderated mediation model of nomophobia, social threat, and phone withdrawal context. Comput. Hum. Behav , 81, 1–9. [CrossRef]

Tangney, J. P., Baumeister, R. F., & Boone, A. L. (2004). High self-control predicts good adjustment, less pathology, better grades, and interpersonal success. Journal of Personality, 72 (2), 271–334.

Thoresen, C. E. (1998). Spirituality, health, and science: The coming revival? In S. Roth-Roemer, S. R. Kurpius, & C. Carmin (Eds.), The emerging role of counseling psychology in health care (pp. 409–431). W W Norton & Co.

Trevino, K. M., Archambault, E., Schuster, J., Richardson, P., & Moye, J. (2012). Religious coping and psychological distress in military veteran cancer survivors. Journal of Religion and Health, 51 (1), 87–98. https://doi.org/10.1007/s10943-011-9526-0

Vermeulen, N., Godefroid, J., & Mermillod, M. (2009). Emotional modulation of attention: Fear increases but disgust reduces the attentional blink. PLoS ONE, 4 , e7924. https://doi.org/10.1371/journal.pone.0007924

Vohs, K. D., & Faber, R. J. (2007). Spent resources: Self-regulatory resource availability affects impulse buying. Journal of Consumer Research, 33 (4), 537–547. https://doi.org/10.1086/510228

Wang, P., Chen, Y., & Chen, Y. (2021). Attachment anxiety and nomophobia among college students in China: The mediating effect of social support. Current Psychology, 40 (5), 2365–2373.

Wagner, D. D., Altman, M., Boswell, R. G., Kelley, W. M., & Heatherton, T. F. (2013). Self-regulatory depletion enhances neural responses to rewards and impairs top-down control. Psychological Science, 24 (11), 2262–2271. https://doi.org/10.1177/0956797613492985

Webb, T. L., Lindquist, K. A., Jones, K., Avishai, A., & Sheeran, P. (2018). Situation selection is a particularly effective emotion regulation strategy for people who need help regulating their emotions. Cognition and Emotion, 32 , 231–248. https://doi.org/10.1080/02699931.2017.1295922

Williams, L. M., Liddell, B. J., Rathjen, J., Brown, K. J., Gray, J., Phillips, M., et al. (2004). Mapping the time course of nonconscious and conscious perception of fear: An integration of central and peripheral measures. Human Brain Mapping, 21 , 64–74. https://doi.org/10.1002/hbm.10154

Wilmer, H. H., Sherman, L. E., & Chein, J. M. (2017). Smartphones and cognition: A review of research exploring the links between mobile technology habits and cognitive functioning. Frontiers in Psychology, 8 , 605.

Xia, N., & Li, H. (2018). Loneliness, social isolation, and cardiovascular health. Antioxidants & Redox Signaling, 28 , 837–851. https://doi.org/10.1089/ars.2017.7312

Yanguas, J., Pinazo-Henandis, S., & Tarazona-Santabalbina, F. J. (2018). The complexity of loneliness. Acta Bio Med. Atenei Parm, 89 , 302.

Yildirim, C., & Correia, A. (2015). Exploring the dimensions of nomophobia: Development and validation of a self-report questionnaire. Computers in Human Behavior, 49 , 130–137. https://doi.org/10.1016/j.chb.2015.02.059

Yildirim, C., Sumuer, E., Adnan, M., & Yildirim, S. (2016). A growing fear: Prevalence of nomophobia among Turkish college students. Information Development, 32 (5), 1322–1331. https://doi.org/10.1177/0266666915599025

Yılmaz, T., & Bekaroğlu, E. (2021). Does interpersonal sensitivity and paranoid ideation predict nomophobia: An analysis with a young adult sample. Current Psychology , 2015 . https://doi.org/10.1007/s12144-021-01501-4 .

Yu, B., Steptoe, A., Chen, L.-J., Chen, Y.-H., Lin, C.-H., & Ku, P.-W. (2020). Social isolation, loneliness, and all-cause mortality in patients with cardiovascular disease: A 10-year follow-up study. Psychosomatic Medicine, 82 , 208–214. https://doi.org/10.1097/psy.0000000000000777

Yun, I., Kim, S. G., & Kwon, S. (2016). Low self-control among South Korean adolescents: A test of Gottfredson and Hirschi’s generality hypothesis. International Journal of Offender Therapy and Comparative Criminology, 60 (10), 1185–1208. https://doi.org/10.1177/0306624x15574683

Zeidi, I. M., Divsalar, S., Morshedi, H., & Alizadeh, H. (2020). The effectiveness of group cognitive–behavioral therapy on general self-efficacy, self-control, and internet addiction prevalence among medical university students. Social Health and Behavior, 3 (3), 93–102. https://doi.org/10.4103/SHB.SHB2020

Download references

Acknowledgements

We thank all the adolescents who participated in this study.

This research was funded by the Ministry of Education, Culture, Research Technology and Higher Education with contract number 157/E.5/PG.02.00.PT/2022.

Author information

Authors and affiliations.

Ahmad Dahlan University, Yogyakarta, Indonesia

Triantoro Safaria

Jambi University, Jambi, Indonesia

Nofrans Eka Saputra

Charitas Musi Catholic University, Palembang, Indonesia

Diana Putri Arini

You can also search for this author in PubMed   Google Scholar

Contributions

All the authors contributed equally to the study’s conceptualization, interpretation of the data, and review, and editing of the manuscript. T.S. performed the statistical analyses; wrote the methods, results, and conclusions; and finalized the manuscript. N.E.S. wrote the introduction, while D.P.A. wrote the discussion. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Triantoro Safaria .

Ethics declarations

Institutional review board statement.

The study followed the guidelines of the Declaration of Helsinki and was approved by The Research Ethics Committee of Universitas Ahmad Dahlan Indonesia, which approved this study’s ethical approval on 8 July 2022 (Ethics Number: 012206071 KEP UAD). The researchers also obtained a data collection permit from the school and university administration.

Informed Consent

All the informants agreed to participate in this study and signed a voluntary informed consent statement. The anonymity and confidentiality of all participants were guaranteed.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Table  6 and Figs. 3 , 4 , 5

Path Diagrams

The result of path analysis among college students

The result of path analysis among senior high school students

The result of path analysis among junior high school students

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Safaria, T., Saputra, N.E. & Arini, D.P. The Impact of Nomophobia: Exploring the Interplay Between Loneliness, Smartphone Usage, Self-control, Emotion Regulation, and Spiritual Meaningfulness in an Indonesian Context. J. technol. behav. sci. (2024). https://doi.org/10.1007/s41347-024-00438-2

Download citation

Received : 03 June 2024

Revised : 11 August 2024

Accepted : 25 August 2024

Published : 11 September 2024

DOI : https://doi.org/10.1007/s41347-024-00438-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Emotion regulation
• Self-control
• Spiritual meaningfulness
• Smartphone use intensity
• Find a journal
• Publish with us
• Track your research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 12 September 2024

Five latent factors underlie response to immunotherapy

• Joseph Usset 1 , 2 , 3 ,
• Axel Rosendahl Huber 1   na1   nAff10 ,
• Maria A. Andrianova 1   na1 ,
• Eduard Batlle   ORCID: orcid.org/0000-0003-2422-0326 1 , 4 , 5 ,
• Joan Carles 2 , 6 ,
• Edwin Cuppen   ORCID: orcid.org/0000-0002-0400-9542 3 , 7 ,
• Elena Elez 2 , 6 ,
• Enriqueta Felip 2 , 6 ,
• Marina Gómez-Rey 2 ,
• Deborah Lo Giacco 2 ,
• Francisco Martinez-Jimenez   ORCID: orcid.org/0000-0001-7804-1134 2 , 3 ,
• Eva Muñoz-Couselo   ORCID: orcid.org/0000-0001-7556-7608 2 , 6 ,
• Lillian L. Siu   ORCID: orcid.org/0000-0002-3500-0540 8 ,
• Josep Tabernero   ORCID: orcid.org/0000-0002-2495-8139 2 , 5 , 6 ,
• Ana Vivancos 2 ,
• Ferran Muiños   ORCID: orcid.org/0000-0002-4651-4168 1   nAff10 ,
• Abel Gonzalez-Perez   ORCID: orcid.org/0000-0002-8582-4660 1 , 5 , 9 &
• Nuria Lopez-Bigas   ORCID: orcid.org/0000-0003-4925-8988 1 , 4 , 5 , 9  

Nature Genetics ( 2024 ) Cite this article

Metrics details

• Cancer therapy

Only a subset of patients treated with immune checkpoint inhibitors (CPIs) respond to the treatment, and distinguishing responders from non-responders is a major challenge. Many proposed biomarkers of CPI response and survival probably represent alternative measurements of the same aspects of the tumor, its microenvironment or the host. Thus, we currently ignore how many truly independent biomarkers there are. With an unbiased analysis of genomics, transcriptomics and clinical data of a cohort of patients with metastatic tumors ( n  = 479), we discovered five orthogonal latent factors: tumor mutation burden, T cell effective infiltration, transforming growth factor-beta activity in the microenvironment, prior treatment and tumor proliferative potential. Their association with CPI response and survival was observed across all tumor types and validated across six independent cohorts ( n  = 1,491). These five latent factors constitute a frame of reference to organize current and future knowledge on biomarkers of CPI response and survival.

The development of CPIs has had a tremendous impact on cancer therapy 1 . However, the response of patients with cancer to these agents varies considerably 1 , 2 , 3 , 4 , 5 , 6 , and important immune-related adverse events may appear as a result of treatment 7 . Consequently, intense research has been dedicated in recent years to identifying features that influence the response to CPIs 2 , 4 , 8 , 9 , 10 , 11 , 12 , 13 , leading to the identification of potential biomarkers.

These studies have made it increasingly clear that the response to CPIs is mediated by several characteristics of the tumor, its microenvironment and the host 12 , which we may regard as latent factors defining CPI response and survival across patients. However, it is likely that different biomarkers identified across a multitude of studies—often focused on one or a small group of features—represent different versions of the same underlying latent factor. For example, the expression of a number of genes and gene sets previously identified as biomarkers may represent the degree of infiltration of cytotoxic cells in the tumor 14 , 15 , 16 , 17 . Furthermore, given that separate research groups independently test different sets of potential biomarkers, there is no effective control of the potential false positives associated with multiple testing. As a result of these problems, it is not clear at present how many such independent latent factors of CPI response and survival there are, what aspects of the tumor, its microenvironment or the host they represent and whether they are relevant across different tumor types.

To answer these questions, we exploited a richly profiled and annotated cohort of patients with metastatic tumors (fresh–frozen biopsied) treated with CPIs (part of the cohort profiled by the Hartwig Medical Foundation (HMF) 18 , 19 ; n  = 479). Specifically, we aimed to identify features of the tumors, their microenvironment or the host that appeared to be significantly associated with CPI response and survival, both across the pan-cancer HMF-CPI cohort and all represented cancer types. To this end, we used an exhaustive—not biased by prior knowledge—analysis of thousands of molecular and clinical features to detect their association with CPI response or survival. We discovered that all significantly associated features collapse into one of five independent latent factors that are relevant across all tumor types represented in this cohort. They are the tumor mutation burden (TMB), effective T cell infiltration, whether the patients received any prior treatment, the activity of transforming growth factor-beta (TGF-β) in the tumor microenvironment and the proliferative potential of the tumor. We verified that at the current level of statistical power, there are no other latent factors of CPI response and survival common to all cancer types analyzed. We validated the association of these five latent factors with CPI response and survival in six independent cohorts ( n  = 1,491 patients) spanning six major cancer types; to our knowledge, the largest such validation effort.

Extracting features from a metastatic cancer cohort

Within the HMF 18 , 19 cohort ( n  = 5,288), 479 patients with metastatic cancer who were part of the Center for Personalized Cancer Treatment study ( https://www.cpct.nl/cpct-02 ) received anti-PD1/PDL1 or a combination of anti-PD1/PDL1 and anti-CTLA4 therapy. We refer to these patients as the HMF-CPI cohort. These include patients who had suffered from primary tumors of the skin (melanomas, n  = 191), lung ( n  = 110), bladder ( n  = 88) and other cancer types (other; n  = 90; Fig. 1a,b and Supplementary Table 1 ). Whole-genome somatic alterations of the metastatic tumors before CPI treatment were identified across all tumors in the HMF-CPI cohort and, for 396 of them, the whole transcriptome of the tumor was also sequenced. Rich clinical data, including treatments received before the diagnosis of their metastatic tumors, response to the CPI therapies, following Response Evaluation Criteria 20 ( n  = 467) and survival information ( n  = 479), were also available (Supplementary Table 1 ).

To carry out a systematic de novo discovery of biomarkers of CPI response, we computed 27,923 features (Fig. 1c,d and Supplementary Note 1 ). These included the mutational (single nucleotide variants + indels) status of 15,829 genes, the copy number status of 2,415 genomic regions, 64 aggregated somatic mutation features (for example, TMB, frameshift indel burden, activity of mutational signatures) and the occurrence of known driver structural variants as well as features summarizing the genomic instability (for example, total number of chromosomal fragments, ploidy, whole-genome doubling, and so forth). We also used the expression level of 8,817 genes and clinical characteristics of the patients such as sex, type of treatment received for the primary tumor and age at the time of diagnosis of the metastasis as features. Finally, human leukocyte antigen (HLA) features that can affect the immune response to the tumor were also included, such as their HLA haplotype and the number of somatically lost HLA alleles.

a – c , For 479 patients with metastatic cancer in the HMF-CPI database of different cancer types, we obtained 18 clinical features, 19 germline HLA allotype features, 18,382 somatic features (based on single base substitutions, indels, copy number variants and other structural variants affecting specific genomic elements or summaries thereof) and 8,817 transcriptomic features, corresponding to all expressed genes. d , Numeric feature values were rescaled and re-normalized ( Methods ), yielding a large table describing the cohort. LOH, loss of heterozygosity, RECIST, Response Evaluation Criteria, CNV, copy number variant; SV, structural variant; WGD, whole-genome doubling; CR, complete response; PR, partial response; SD, stable disease; PD, progressive disease; OS, overall survival; PFS, progression-free survival.

Five latent factors of CPI response and survival

To identify which of the more than 27,000 features computed per patient were significantly associated with CPI response, we performed univariate regressions (adjusted for the site of origin of the tumor, age of the patients, site of biopsy of the metastasis and tumor purity). After controlling for multiple testing 21 , we identified several hundred features that appeared to be significantly associated with CPI response (Fig. 2a , Extended Data Fig. 1a–c and Supplementary Note 1 ).

a , Logistic regression analysis (represented as a volcano plot) identified features significantly associated with CPI response ( Methods and Supplementary Note 1 ). Dots with larger sizes represent significant features, and they are colored following the type of feature. P  values shown in the plots were computed by logistic regressions. These are, by definition, two-sided. b , All significant features were selected and clustered based on their pairwise correlations. The colors denoting the clusters are inherited from the type of feature included in each of them according to the color legend in a . c , Mean expression values of cluster R3 (x-axis), and the 'T-cell effector' gene set (y-axis), across patients (dots). The Pearson's correlation coefficient is indicated. d , To discern the nature of cluster R3, the correlation of its mean to 255 gene sets collected from the literature was computed across patients (as illustrated in panel ( c )). Dots represent gene sets. e , Relationship between the significance of the association with the response (y axis) and the correlation (x axis) to the mean of the cluster of the features in each cluster. P  values shown in the plots were computed by logistic regressions. These are, by definition, two-sided. Dots in these three panels appear in darker color if they represent features significantly associated with CPI response and with a correlation coefficient above 0.5 with the mean of their respective cluster. In ( a ), ( d ) and ( e ), the horizontal dashed lines represent the significance threshold according to the Benjamini–Yekutieli correction.

Then, we asked how these significant features relate to each other and which underlying latent factors of CPI response they represent. To answer these questions, we clustered all significant features based on their pairwise correlations (Fig. 2b ). Virtually all (Supplementary Note 1 ) could be unambiguously assigned to one of three clusters (R1, R2 or R3, encompassing somatic, clinical or transcriptomics features). This implies that only three latent factors associated with CPI response were detected from the more than 27,000 features analyzed.

To understand the nature of cluster R1, we first computed the mean of its integrating features. The single feature in the cluster with the highest correlation to the mean was the overall TMB, with other aggregated mutational features (for example, clonal TMB) also showing a high correlation. Specifically, the increase of TMB is associated with a higher probability of response and also increased survival (Extended Data Fig. 2a and Supplementary Fig. 1 ). Thus, we named this latent factor TMB, and although it could be measured using any of the features in the cluster, we selected the TMB to represent it. Importantly, the mutation rate of virtually all genes (some of which have been previously associated with CPI response 3 , 4 , 10 , 22 , 23 , 24 ) also appear to be highly correlated with the TMB as part of this cluster of features, and indeed, some heavily mutated genes exhibit lower P  values in the regression analysis than TMB (Supplementary Note 1 ). This implies that identifying the mutations of individual genes as biomarkers of CPI response independently of the TMB is a very challenging task.

Cluster R2 was integrated by two highly correlated clinical features: prior exposure to systemic therapy 11 and prior exposure to any therapy. These two features appear to be significantly negatively associated with CPI response and survival (Fig. 2b , Extended Data Fig. 2b , Supplementary Fig. 1 and Supplementary Note 1 ), perhaps owing to increased tumor aggressiveness or deteriorated patient condition. Thus, we named cluster R2 ‘prior treatment’, and for the following analyses, we represented this cluster using exposure to any prior treatment.

Cluster R3 grouped the expression of 48 genes. We reasoned that the expression of gene sets representing biological functions in the tumor or its microenvironment could aid in the interpretation of this cluster. Thus, we computed the mean expression of 255 gene sets (225 representing all Kyoto Encyclopedia of Genes and Genomes (KEGG) pathways and cancer hallmarks obtained from the Molecular Signatures Database (MSigDB) 25 , 26 and 30 collected from the literature 12 , 27 , 28 ; Supplementary Table 2 , Fig. 2c,d and Methods ). The mean expression of 13 gene sets was significantly associated with the response to CPI (Fig. 2d , Extended Data Figs. 2c and 3a and Supplementary Fig. 1 ), and all of them showed a high correlation (Pearson’s coefficient of >0.8) with the mean expression of the genes in the cluster. They all represent some aspect of immune infiltration in the tumor; most specifically, T lymphocyte infiltration. Therefore, we named this third latent factor ‘effective T cell infiltration’ and represented it through the mean expression of all genes in the cluster. The increase in effective T cell infiltration appears significantly associated with a higher probability of CPI response and longer survival.

There is a clear positive relationship between the correlation of every feature to the representative of its corresponding latent factor (that is, TMB, prior treatment and the mean expression of genes in cluster R3) and the significance of its association with CPI response. The higher the correlation of a feature with the mean of its corresponding cluster, the more significant its association with CPI response (Fig. 2e and Extended Data Fig. 2a–c ). These three latent factors are also significantly associated with overall survival and progression-free survival upon CPI treatment (Extended Data Fig. 2a–c ).

We then asked whether any other latent factors of the tumor, its microenvironment or the host specifically influence the survival of patients, independently of the previous three latent factors associated with response (given that response is, by itself, a major determinant of survival). To answer this question, we focused on features that appeared to be significantly associated with overall survival, after controlling (in addition to the aforementioned covariates) for the three latent factors previously associated with response (Fig. 3a ). Again, to discern how many latent factors were represented by these features, we clustered them based on their pairwise correlations (Fig. 3b ).

a , Features significantly associated with survival residuals, that is, after correction for the three latent factors associated with response. Larger dots represent significant features. Features with high correlation (Pearson coefficient of >0.5) with any of the three previously identified latent factors are removed. P  values shown in the plots were computed by Cox regressions. These are, by definition, two-sided. b , Clusters of features based on their pairwise correlations. c , d , Cluster S1 and cluster S2.1 are highly correlated with gene sets representing the tumor proliferative potential and the activity of TGF-β in the tumor microenvironment, respectively. Dots represent the mean expression of two gene sets (y axis) and the mean expression of the genes in clusters S1 and S2.1 (x axis) across patients. Pearson's correlation coefficients are indicated. e , Features significantly associated with overall survival. Larger dots represent significant features. f , Significance of the association with the response ( y axis) and the correlation ( x axis) to the mean of the cluster of the features in each cluster. P  values shown in the plots were computed by Cox regressions. These are, by definition, two-sided. g , Depiction of the five latent factors associated with CPI response and survival. Upwards arrows, positive association with response and/or survival; downwards arrows, negative associations. In ( a ), ( e ) and ( f ), the horizontal dashed lines represent the significance threshold according to the Benjamini–Yekutieli correction.

One of the three clusters was clearly orthogonal to the other two, which exhibited a certain degree of inter-correlation. Thus, we named them clusters S1, S2.1 and S2.2, as they only represent two mutually orthogonal latent factors (Fig. 3b and Supplementary Note 1 ). To interpret them, we analyzed the correlation of their mean expression with that of 255 gene sets (Supplementary Table 2 ), as explained above. The mean of cluster S1 showed the highest correlation with a gene set named ‘Proliferation potential’ (Fig. 3c ) and a high correlation with other gene sets representing cell cycle and overall cell proliferation (Extended Data Figs. 2d and 3b,c and Supplementary Fig. 1 ). We thus named it ‘tumor proliferative potential’ and represented it through the mean expression of all genes in the cluster.

The mean expression of the genes in cluster S2.1 showed the highest correlation with a gene set representing TGF-β in fibroblasts (Fig. 3d ) and a high correlation with other gene sets related to this biological process (Extended Data Figs. 2e and 3b,d–f ). As in other cases, we represented this latent factor through the mean expression of the genes in the cluster. Low values of this latent factor (TGF-β activity in the microenvironment) are associated with longer survival of patients upon CPI treatment even without correcting for the effect of the three response-associated latent factors (Fig. 3e ).

We next asked whether the latent factors are specifically associated with CPI treatment or whether they represent general elements that influence response and survival upon any type of therapy. To answer these questions, we analyzed the data for 2,497 patients in the HMF cohort who received non-CPI therapies and found that the effects of TMB, effective T cell infiltration and TGF-β activity in the microenvironment are unique to CPI therapies, whereas prior treatment appears to affect the response to both CPI and non-CPI therapies, and the effect of tumor proliferative potential appears even larger across non-CPI than CPI-treated patients (Extended Data Fig. 4 ). Virtually all features that appear significantly associated with CPI response and/or survival are grouped in one of the five latent factors (Extended Data Fig. 5 ), indicating that no latent factor remains to be discovered in the HMF-CPI cohort.

In summary, five mutually orthogonal latent factors underlying CPI response and survival across the HMF-CPI cohort (Fig. 3g ) emerged from this unbiased analysis. Supplementary Dataset 1 lists the results of the unbiased analysis of features in their association with CPI response and survival. Each of them can be represented through a number of features that are clustered by virtue of their pairwise correlations.

Validation of the five latent factors

Next, we asked whether the five latent factors, identified across HMF-CPI, were of comparable importance in the four groups of tumors with different tissues of origin represented in the cohort. To answer this question, we conducted, for each latent factor, multivariate regressions (adjusted for age, tumor purity, biopsy location and the remaining four latent factors) of their effect on CPI response and survival (Fig. 4 ). We found that the direction of the association of each factor (with response or survival) was maintained for all tumor types as in the pan-cancer analysis, with small differences in the effect size and the significance of their associations. One exception is the association of effective T cell infiltration with response in patients with lung tumors, which was not significant (although the significance in the association with survival is maintained). The other is prior treatment, which does not exhibit a significant association with response across bladder tumors. Very similar results were obtained in cancer type-wise univariate regressions (Extended Data Fig. 6a ). In summary, we find that the latent factors, with few exceptions, appear to underlie CPI response and survival across all tumor types represented in the HMF-CPI cohort (Supplementary Note 1 ).

Forest plots illustrating the association of latent factors across groups of tumors with different origins in the HMF-CPI cohort (left) and across six independent cohorts (right) with CPI response and overall survival. The value of each latent factor was computed as the mean of the cluster of features obtained in the HMF-CPI cohort across each validation cohort, except in the VHIO cohort, where the transcriptomics latent factors were estimated from alternative sets of genes ( Methods ). In the forest plots, the dots represent the strength (coefficients estimated through multivariate logistic or Cox regression) of the association between the latent factor and response or survival across cohorts. The horizontal bars denote the 95% confidence intervals. Gray dots represent latent factors whose estimates are within one standard error at either side of 0, dots with a light color (green or red) represent non-significant associations with coefficient estimates above (or below) one standard error of 0 and dark-colored dots represent significant associations. Green dots represent positive associations with improved outcomes (higher response odds or lower hazard ratio), while red dots represent negative associations (lower response or higher hazard ratio). Mixed denotes cohorts integrated by patients with multiple tumor types.

We next asked whether the five latent factors are validated in independent cohorts of the same and other tumor types representing the wide diversity of approaches of sample processing and tumor profiling used in the clinic. To this end, we collected data from the literature for five independent cohorts or metacohorts (INSPIRE 29 , Lyon 30 , MARIATHASAN 27 , PARKER ICI 31 , RAVI 32 ) and obtained the data from another cohort of patients treated at the Vall d’Hebron Institute of Oncology (VHIO). These validation cohorts comprised 1,491 patients with primary or metastatic tumors of different organs (Supplementary Table 1 and Supplementary Note 1 ). For 339 of these patients, we obtained sufficient information to compute the five latent factors, while for the remaining 1,152, we could compute only four or three latent factors. Using the available clinical information, we were able to evaluate the association of the latent factors with CPI response for 1,294 patients across all cohorts, while the association with overall survival could be computed for 1,165 patients across five cohorts. Unlike in the case of the HMF-CPI cohort, most of these studies (except INSPIRE) started from formalin-fixed, paraffin-embedded samples. The approaches used to identify somatic mutations range from whole-exome tumor-normal paired sequencing to tumor-only sequencing of a panel of 432 genes. The expression of genes was measured by whole-transcriptome, targeted RNA sequencing (RNA-seq) or a panel of 170 genes using the nCounter (NanoString) platform.

In a multivariate analysis, pooling all external cohorts, the associations were consistent between each of the five latent factors and CPI response or survival, with all except tumor proliferative potential reaching significance (Fig. 4 ). In some individual cohorts, the association of a particular latent factor with CPI response or survival could not be verified, such as the TMB in the VHIO cohort. In this case, owing to the lack of a control sample to reliably call somatic mutations, the calculation of TMB is probably not reliable (Supplementary Note 1 ). Nevertheless, despite the differences in cohorts, profiling and sample collections, the associations observed in the HMF-CPI cohort for the five latent factors were, overall, reproduced across the validation cohorts. T cell effective infiltration was positively associated with CPI response across five validation cohorts (three significantly), TGF-β activity in the microenvironment was negatively associated with survival in the five cohorts in which it could be evaluated (four significantly) and tumor proliferative potential was negatively associated with survival in four out of five cohorts (two significantly; Fig. 4 ). The association with prior treatment was validated in all (two significantly) but one cohort (Fig. 4 and Supplementary Note 1 ). Genes closer to the mean of clusters R1 (T cell effective infiltration), S2.1 (TGF-β activity in the microenvironment) and S1 (tumor proliferative potential) in HMF-CPI also tend to correlate better with one another across the four validation cohorts with transcriptomics data ( Methods ; Extended Data Fig. 6b ).

In summary, despite the wide differences in tumor sample processing and profiling, many of the associations between the five latent factors and CPI response or survival previously discovered in the HMF-CPI are also observed across six independent cohorts.

Multivariate models to predict CPI response and survival

We next asked how the effects of the five latent factors combine (through accumulation or interaction) to influence CPI response and survival. To that end, we trained multivariate machine-learning (tree-based gradient-boosting) models 33 to predict the response, overall survival or progression-free survival of patients in the HMF-CPI cohort. To exploit the higher statistical power provided by the full cohort and the specificity inherent in the response across cancer types, we first constructed pan-cancer models and then used them as the base to obtain hybrid models; that is, subjecting the pan-cancer models to added rounds of training on the data corresponding to each tumor type (Fig. 5a and Supplementary Note 1 ). The hybrid models trained on the five latent factors outperformed models trained solely on tumor type-specific data (Supplementary Fig. 2a ) as well as equivalent models trained solely on values of TMB and PDL1 expression (Supplementary Fig. 2b ) within the HMF-CPI cohort. Models trained on different representations of the five latent factors showed comparable performance, supporting the idea that the features of each cluster constitute alternative representations of the latent factors (Supplementary Fig. 3a,b ). The variability in the influence of the five latent factors across different tumor types in the HMF-CPI cohort observed in the multivariate regression analysis described above is verified through a survey of their relative importance on the prediction cast by the multivariate machine-learning models of CPI response and overall survival ( Methods ; Extended Data Fig. 7a,b ).

a , The values of the representative biomarkers of the five latent factors across patients in the HMF-CPI cohort were used to train hybrid (pan-cancer-informed tumor type-specific) gradient-boosting models to predict CPI response and survival. The performance of the models was assessed through cross-validation ( Methods and Supplementary Note 1 ). b , Stratifying patients based on model predictions. We separated the patients in the HMF-CPI cohort into three groups based on their predicted probability of response (histograms) and the three-segment bar below. We then calculated the fraction of responders within each group (bar plots below each histogram). c , Differences in overall survival between the three groups of patients are represented by Kaplan–Meier curves. The P  value for each cohort (annotated in the plot) was calculated with a one-sided log-rank test. The line colors correspond to the three groups of patients defined in a . d , The TMB for each patient in the HMF-CPI cohort (with complete data for all five latent factors) was computed with a measure commonly used in the clinic: the number of mutations per genomic megabase. Tumors were classified as low-TMB or high-TMB based on a simple cutoff (10 mutations per megabase). The bars are colored according to the fraction of patients with high or low TMB in each of them. Interestingly, a number of patients with high-TMB tumors are predicted to have a low probability of response, whereas some patients with low-TMB tumors appear in the high probability of response group. The bottom bar plots present the percentage of patients in the low-TMB and high-TMB groups that showed clinical response to CPIs. OS, overall survival; BOR, best overall response according to RECIST; MB, megabase.

We then stratified 396 patients in the HMF-CPI cohort with all data types (jointly and separately by tumor type) into three groups of low (below 0.1), medium (between 0.1 and 0.5) and high (greater than 0.5) predicted probability of response to CPI. Only 2 (3%) of the 67 patients in the low-probability group actually responded to CPI treatment, compared with 61 out of 97 (63%) patients in the high probability of response group (Fig. 5b ). This stratification also significantly separated patients in the HMF-CPI cohort based on their survival (Fig. 5c ). Stratifying the patients based on a threshold of TMB used in the clinical practice (ten mutations per Mbp 34 , 35 ) to separate high-TMB and low-TMB tumors is less optimal, with 17% of responders among patients with low-TMB tumors and 42% among those with high-TMB tumors (Fig. 5d ). Interestingly, patients in the group with low probability of response exhibit a range of predicted hazards according to the overall survival pan-cancer model (Extended Data Fig. 8a–g and Supplementary Note 1 ). Across the VHIO and INSPIRE cohorts, the stratification based on the predicted probability of response produced a perfect identification of patients with a low probability of response, while results were less accurate across the RAVI cohort (Extended Data Fig. 9a and Supplementary Fig. 4 ).

When applied to patients in the HMF cohort who did not receive CPIs, the multivariate models of response identified an important fraction of the patients with skin (35%), bladder (42%) and lung (16%) tumors with a high likelihood of response to the treatment (Extended Data Fig. 10 ). Interestingly, patients suffering from other metastatic malignancies (some not usually considered as candidates for CPI) were also identified as potentially good responders. For example, 18 (4%) patients with breast cancer, 10 (3%) patients with colorectal cancer, 10 (19%) patients with kidney tumors and 5 (15%) patients with liver cancer exhibited high probability of response to CPI.

In summary, we illustrate that multivariate models combining the five latent factors produce a more accurate stratification of patients according to their predicted probability of response than the TMB alone.

In this work, we followed a completely unbiased approach to discover genomics, transcriptomics and clinical features associated with CPI response and survival across patients with cancer. We aimed to answer how many and which aspects of the tumor, its microenvironment and the host influence the response to CPIs across patients (that is, latent factors), in an effort to provide a framework of reference to existing copious reports of biomarkers. First, through univariate logistic and Cox regressions, we identified a few hundred features that are significantly associated with CPI response and/or survival. Five latent factors emerge when these significant features are clustered based on their pairwise correlation. These represent mutually independent aspects of the tumor, its microenvironment and the host that influence the response of a patient to CPI and their hazard after the treatment.

Although the fact that some genomics and transcriptomics features may represent the same aspect of tumors, their microenvironment or the host had been reported before, here we show that an array of different, highly intercorrelated features (for example, expression of genes related to T cell function) actually represent different measurements of the same latent factor. This is particularly striking in the case of TMB: the mutation rate of hundreds of genes (including cancer driver genes) appears to be highly correlated with TMB, suggesting that the association of mutations in a given gene with CPI response rather than an independent biomarker is just an alternative proxy measurement of TMB. Although the mutation status of some genes may still be bona fide biomarkers of CPI response, independently of TMB, any analysis to identify them should account for the confounding factor of their correlation with TMB. We also demonstrate that the associations of the latent factors with CPI response and survival are observed across different tumor types, and we validated them across six independent cohorts of patients. Most of the associations discovered in the HMF-CPI cohort were corroborated across these independent cohorts, despite differences in sample collection and processing procedures and profiling methods between these cohorts and the HMF-CPI. This indicates that these latent factors are mostly universally associated with CPI response and survival. They may thus be potentially used in the future within the clinical practice, despite the heterogeneity of sample processing and tumor profiling approaches used. The variability observed across tumors of different origins in more than one cohort (for example, the smaller association of effective T cell infiltration with the response of lung tumors) may point to findings that could be pursued further. To our knowledge, this constitutes the most extensive exploration, to date, of the biomarkers of CPI response and survival across cohorts with tumors from different organs.

Importantly, no features other than these five latent factors were significantly associated with CPI response or survival. That is, virtually all significant features cluster within one of them. However, some relevant features may still lay below the statistical power of the HMF-CPI cohort or appear significantly associated with CPI response or survival in only one tumor type. This is particularly important for features that may be relevant for a fraction of patients, such as mechanisms of immune escape (for example, B2M deletion, an event that we observe as significant across melanomas but not other tumors in the HMF-CPI cohort; see Supplementary Note 1 ) 8 , 22 , 24 , 36 . Other examples of such features relevant for specific groups of patients may include common polymorphisms that affect the immune response 37 and the heterozygosity at HLA loci 38 . Particularities of the tumor types not represented in the HMF-CPI cohort are, of course, also absent from our current catalog of proxy biomarkers. These will be discovered when larger CPI cohorts are analyzed; it is not inconceivable that even more latent factors will become apparent then. Our discovery of latent factors is also limited by the profiling technologies used, which rely on deconvolution of the immune infiltrate based on bulk RNA-seq data. More detailed studies of this infiltrate—based on fine mapping of immune populations and their interactions with other cells in the microenvironment—will contribute in the future to refine the landscape of biomarkers of CPI response and survival.

A test of the application of multivariate models combining the five latent factors produced a stratification of the patients in the HMF-CPI cohort based on their predicted response probabilities that discriminates better between responders and non-responders than the threshold of TMB frequently used in the clinic. This could be regarded as a proof-of-principle for application of the five latent factors to clinical practice. Being able to identify patients with a very low probability of response would be relevant to spare them the potential side effects of the therapy 7 . Additionally, it may aid in reducing the financial burden on healthcare providers 39 . There is also the possibility—illustrated through the analyses described above—to use such multivariate models to identify patients with tumors that are not usually considered suitable candidates for CPI who have a high probability to respond. In the future, when these types of models can be used in support of clinical decision-making, this could potentially contribute to expanding the therapeutic options for patients suffering from these malignancies.

In summary, we envision that the results of this work can provide a frame of reference to the research of biomarkers of CPI response and survival, resulting in the classification of all identified significant features falling into one of these five latent factors, or a completely independent one.

Discovery cohort

Whole-genome somatic mutations, copy number and other structural variants across metastatic tumors from 4,484 patients in the HMF cohort were obtained from the HMF database 18 , 19 (version DR-263_update1). Of these, 479 subsequently received CPI therapy (HMF-CPI cohort), for which all somatic variation information was available. Whole transcriptome from RNA-seq was available from the same source for a subset of 396 patients in the HMF-CPI cohort. Several features computed by the HMF pipeline from this data for each tumor (for example, number of neoepitopes, activity of mutational signatures) as well as the patients’ germline features (such as HLA allotypes) were obtained as part of the dataset. It also included all relevant clinical information regarding exposure to treatment for their primary malignancy, the subsequent treatment regimen for the metastasis and longitudinal measurements of the outcome (Supplementary Note 1 and Supplementary Table 1 ). The ethical approval to use this data in research has been obtained by the HMF.

Validation cohorts

Whole-exome somatic mutations, the whole-transcriptome RNA-seq gene expression of tumors and all clinical data pertaining to prior treatment as well as outcome upon treatment with pembrolizumab within the INSPIRE basket trial ( NCT02644369 ) 29 were obtained for 64 patients from https://github.com/pughlab/inspire-genomics .

Targeted RNA-seq (2,559 genes) and clinical data from 315 patients treated at several hospitals in Lyon and Paris 30 were obtained from https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE159067 , https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE161537 , https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE162519 and https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE162520 .

MARIATHASAN

Whole-exome somatic mutations, the whole-transcriptome RNA-seq gene expression of tumors and all clinical data in a previously published study 27 of 348 patients were obtained from http://research-pub.gene.com/IMvigor210CoreBiologies .

Whole-exome somatic mutations, the whole-transcriptome RNA-seq gene expression of tumors and all clinical data of several cohorts of tumors (including those within clinical trials CheckMate 038 and CheckMate 067 and two cohorts published within other studies) compiled in a previous publication 31 totaling 315 patients were obtained from https://github.com/ParkerICI/MORRISON-1-public .

Whole-exome somatic mutations, the whole-transcriptome RNA-seq gene expression of tumors and all clinical data of the SU2C-MARK cohort from a previous publication 32 comprising 352 patients were obtained from https://zenodo.org/records/7625517 .

The estimated TMB and the expression (via NanoString) of 170 genes across the tumors of 74 patients with cancer profiled and treated at the VHIO Hospital in Barcelona were obtained directly from the Cancer Genomics Group. Clinical data of these patients were provided by attending oncologists at VHIO.

Details of all cohorts appear in Supplementary Table 1 and Supplementary Note 1 .

Ethical approval to use the data of the first five validation cohorts in research was obtained by the original institutions, who obtained written informed consent from patients and made the data available through scientific publications. The Vall d’Hebron University Hospital Ethics Committee of Clinical Research approved the study according to local guidelines and regulations, and written consent was obtained from all the patients included in this study.

Feature extraction for systematic analysis

Somatic features (18,382) representing single nucleotide variants, indels, copy number variants and other structural variants were extracted from files directly downloaded from the HMF database. These included the list of variants as well as summary statistics, such as TMB, burden of structural variants, predicted neoantigen burden, and so on. Although some features were obtained directly from the files, others were derived. The level of expression (transcripts per million) of 8,817 genes (measured through whole-transcriptome RNA-seq) was also obtained from files downloaded from the HMF database after pre-processing (see below). Other RNA-seq features were derived from these values, mainly through the summarization of the expression of genesets in separate features 12 , 27 , 40 , 41 , or by Cibersort 17 derivation of immune cell populations from gene expression data (Supplementary Table 2 ). The HLA allotypes of HMF-CPI patients were directly obtained from files downloaded from the HMF database, while somatic HLA loss of heterozygosity in the tumors was estimated using the LILAC tool 22 . Clinical information regarding courses of treatment before the biopsy of the metastasis and the subsequent outcome of CPI treatment was also obtained from files downloaded from the HMF database. Again, part of that information was directly converted into features, while other features, such as the time elapsed between the end of the prior treatment and the biopsy of the metastasis, were derived from these data. Some of these clinical features were converted into outcomes of the analysis (best overall response to CPI, overall survival and progression-free survival upon CPI), while others were maintained as potentially predictive features. A detailed description of the strategy followed for the extraction of features in the HMF-CPI cohorts appears in Supplementary Note 1 .

Pre-processing

All outcomes and features were computed across all samples in the HMF database. Finally, the data was joined based on the sample identifiers to produce a data frame ready for statistical analyses. Before systematic analyses, several pre-processing steps were performed. First, to reduce multiple testing, we applied filters to remove features with little chance of providing meaningful associations. For somatic mutations by gene, only genes with at least one mutation per 20 samples were kept for the analyses. For RNA expression, only coding genes with a mean and standard deviation of adjusted transcript per million values greater than 0.5 were considered. For the driver features, only driver genes mutated in at least one in 30 samples were included. Similarly, for mutational signatures, only signatures with exposure greater than 0.02 for at least one in 20 samples were included. Second, all features were standardized to have a mean of zero and a standard deviation of one across the CPI samples. This standardization allowed for fair comparisons of estimated effect sizes. Outside of the primary tissue location, all features in the analyses were numeric or ordinal.

Systematic analyses

Each feature was tested individually for the strength of association to best overall response, progression-free survival and overall survival. Generalized linear models and their native maximum-likelihood-based tools were used for all estimation, standard error calculation and hypothesis testing.

The best overall response was modeled with logistic regression, in which we assumed that the probability of response followed a Bernoulli distribution with mean p . For each feature X , we accounted for primary tissue, biopsy location, tumor purity and age as model covariates. Formally, let I j represent the covariate indicator functions for primary tissue (skin, lung, bladder, other tissue), let I k represent the indicator function for biopsy location (lung, liver, lymph node, primary, skin, other tissue), let X age represent patient age and X purity represent the tumor purity. The full and reduced models were fit as follows.

Full model:

Reduced model:

The models were fitted with the base R glm function.

Progression-free and overall survival outcomes were modeled for each feature with Cox proportional hazards models. The hazard rates, denoted h ( t ), were modeled as follows.

Survival models were fitted using the coxph function from the survival package in R.

For all analyses, P  values were computed based on the likelihood ratio tests comparing the full and reduced models.

For the main analyses of best overall response, progression-free survival and overall survival, the covariates included were the indicators for primary tissue, the age of patients, the site of biopsy of the metastasis and the tumor purity. For the overall survival residuals analysis, the covariates additionally included the representative biomarkers of the three latent factors explaining response: TMB, T cell effective infiltration and pretreatment. For all model–feature–covariate combinations, the P  values were calculated from the likelihood ratio test comparing the full model to the reduced model (with the feature of interest removed). Effect sizes (log odds ratio for the logistic regression and hazard ratios for the Cox regression) and standard errors were estimated with maximum likelihood from the full models. All effect sizes, standard errors and corresponding P  values were stored for further analysis. Given the large dependency in tests, we used the Benjamini–Yekutieli multiple testing threshold to control the false discovery rate 21 . Several exhaustive analyses were run, with different sets of covariates each producing similar conclusions. Full documentation of all exhaustive analyses can be found in Supplementary Note 1 .

Identification of latent factors

Latent factors were defined as the independent biological mechanisms underlying the features most predictive of CPI response and survival. To label latent factors, we first focused on features passing the Benjamini–Yekutieli multiple test significance threshold. From these significant features, we computed their pairwise Pearson correlations and identified clusters using hierarchical clustering (hclust() in R with the Ward.D2 algorithm). The optimal number of clusters was defined using the R package ‘factoextra’ function fviz_nbclust, using silhouette’, ‘wss’ and ‘gap_stat’ options.

To label transcriptomics clusters, we computed the expression of 255 gene sets reported in the literature. The gene sets (Supplementary Table 2 ) were collected by downloading the Hallmark and KEGG annotated gene sets from MSigDB 25 , 26 (version 2023.1.Hs). These genesets were further complemented by others, obtained from previous publications 27 , including a paper describing the CPI-1000 analyses 12 . Genesets with a Pearson correlation of >0.8 with the mean of a specific cluster and passing the multiple test P  value threshold of association with CPI response or survival were considered cluster-specific and thus used to discern the nature of the cluster.

Stability of transcriptomics latent factors

For each gene in each transcriptomics cluster, we calculated the silhouette score 42 using the silhouette() function from the package ‘cluster’ in R. This score reflects how close the particular data point is to the cluster of assignment and how far it is from other clusters. The matrix of distances between genes used to calculate silhouette scores was obtained from the correlation matrix

where cor is the correlation matrix of gene expression levels.

We then calculated silhouette scores for each gene in every other cohort with available expression data (INSPIRE, MARIATHASAN, PARKER ICI, RAVI) using gene expression levels from the corresponding dataset but keeping the initial clustering obtained in the HMF cohort. Aggregation of silhouette scores across datasets was performed using the aggregateRanks (method = ‘stuart’) function from the ‘RobustRankAggreg’ R package 43 .

Multivariate machine-learning models

Multivariate models were fitted using the Extreme Gradient Boosting (XGBoost) package in R 44 . The training in all cases sought to find the tree function T (sum of trees) that minimizes the expected loss between the observed and predicted response values; that is:

where X and Y are the feature and response data, respectively, and \(L\) is the loss function of choice. In our setting, the loss function was chosen to be a negative likelihood compatible with the typical distributional assumptions for each type of data. Specifically, the best overall response was modeled using the logistic regression likelihood, while progression-free survival and overall survival were modeled using the Cox proportional hazards likelihood.

We trained three pan-cancer models (one per outcome) incorporating all available training data (479 patients). We also trained 12 hybrid models based on the three pan-cancer models followed by further cycles of training on patients of each tumor type, but maintaining exactly the same loss function and all hyperparameters. Patients suffering from malignancies other than skin melanomas and lung or bladder tumors were pooled within a group labeled as ‘other’ tumor types. This model fit procedure found a compromise between low variance but high bias from the pan-cancer models and low bias but high variance in pure tumor type-specific models. Finally, we also trained pure tumor type-specific models, starting from the patients in each of the four groups separately (details in Supplementary Note 1 ).

The XGBoost models require many tuning parameters (learning rate, depth, sub-sampling, minimum tree leaf size) that guide the internal model fitting. Initially, our model building used grid searches to select optimal internal tuning parameters. However, in our cross-validation study, we found that simple additive models (depth, 1; fast learning rate, 0.05; minimum leaf size, 5; sub-sampling, 0.75) had the best performance.

For the best overall response, the model casts the prediction outputs as log odds ratio scores that can then be recast into probability scores (continuous values between 0 and 1). For progression-free survival and overall survival, the models cast the prediction outputs as log hazard ratios that can then be recast into hazard ratios (continuous and positive).

Separate models were trained solely on TMB and PDL1 expression (the continuous value reported in the HMF-CPI cohort by whole-transcriptome RNA-seq). These models were used to represent the predictive power of clinically approved biomarkers across analyses of the performance of multivariate models.

Calculation of Shapley values

Given that the final tree-based models were additive, the calculation and extraction of Shapley values was straightforward. For each feature, for a given additive model and individual sample, there was 1-to-1 mapping from the feature values and the Shapley values. This relationship between feature and Shapley values is visualized by the marginal dependence plots in Extended Data Figure 7a,b . In R, using the predict function applied to the XGBoost output, we set the argument contribution = TRUE to extract the Shapley values. The extracted Shapley values measure additive feature contribution to the log odds ratio for response models and the log hazard ratio for the Cox survival models.

Proxy biomarkers in the VHIO cohort

In the VHIO cohort, the TMB was estimated from the mutations detected using a 432-gene hybrid capture-based panel 45 . The expression of 170 genes was measured using the nCounter (NanoString) platform 46 . Normalized NanoString counts were log transformed and standardized, and proxy biomarkers were selected based on their correlation with the representative biomarkers of the five latent factors in the HMF-CPI cohort. For the T-cell effective infiltration gene set, CXCL9 , CXCL10 , CXCL11 , GZMA , GZMB and IFNG were selected. Overall, this gene set was strongly correlated with the original T-cell effective infiltration gene set (ρ = 0.97) and showed high statistical significance in the exhaustive analysis ( P  = 7.0 × 10 −8 ). To select a set of genes to represent the latent factors of TGF-β activity in the tumor microenvironment and tumor proliferative potential, we selected genes with a correlation of >0.5 to the respective gene set. This process yielded BRCA1 , BRCA2 and TUBB for the tumor proliferative potential gene set. Although none of these genes were included in the representative biomarker obtained from the HMF-CPI cohort, they all showed a strong correlation to this gene set. The proxy gene set also showed a statistically significant association with overall survival residuals. The aforementioned process, in the case of the VHIO TGF-β gene set, yielded DLL4 , HEYL , NOTCH3 , NOTCH4 , SERPINE1 , TGFB1 and TGFB3 . This gene set also showed a very strong correlation to the representative TGF-β activity in the tumor microenvironment biomarker (Supplementary Note 1 ).

Statistics and reproducibility

The systematic analysis to identify features associated with CPI response and survival was carried out through logistic and Cox regressions, and the results were filtered for multiple testing as described in the Methods and Supplementary Information . These features were grouped into latent factors based on their pairwise correlations. Standard statistical approaches, such as univariate and multivariate regressions or Kaplan–Meier analysis, were used downstream for the analysis of the latent factors across validation cohorts. No statistical method was used to determine sample size for the analysis. All available samples from the discovery and validation cohorts were used; none were excluded from the analysis. Given that the study consisted entirely of the analysis of existing data, it was not randomized and the investigators were not blinded, as no allocation of samples in groups was carried out. All data used in this study are publicly available (see below) and the code used to reproduce the analysis described in the paper has been deposited in public repositories (see below).

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

Access to the HMF-CPI data can be obtained through a request to the HMF ( https://www.hartwigmedicalfoundation.nl/en/data/data-acces-request ) 18 , 19 . The validation datasets can be obtained as follows: INSPIRE 29 , INSPIRE github repository ( https://github.com/pughlab/inspire-genomics ); Lyon 30 , GEO GSE159067 , GEO 161537 , GEO GSE162519 and GEO 162520 ; MARIATHASAN 27 , public repository ( http://research-pub.gene.com/IMvigor210CoreBiologies ); PARKER ICI 31 , PARKER ICI github repository ( https://github.com/ParkerICI/MORRISON-1-public ); RAVI 32 , zenodo repository: 7625517 ; VHIO, this study github repository ( https://github.com/bbglab/immunebiomarkers ).

Code availability

All code necessary to carry out the extraction of the features from the HMF-CPI provided files (version DR-263_update1) and to generate the data frame needed for analysis is freely available in a public repository ( https://github.com/bbglab/hartwig_biomarkers ). The code to reproduce all analyses is also publicly available ( https://github.com/bbglab/immunebiomarkers ).

Korman, A. J., Garrett-Thomson, S. C. & Lonberg, N. The foundations of immune checkpoint blockade and the ipilimumab approval decennial. Nat. Rev. Drug Discov. 21 , 509–528 (2022).

Article   CAS   PubMed   Google Scholar  

Kalbasi, A. & Ribas, A. Tumour-intrinsic resistance to immune checkpoint blockade. Nat. Rev. Immunol. 20 , 25–39 (2020).

Havel, J. J., Chowell, D. & Chan, T. A. The evolving landscape of biomarkers for checkpoint inhibitor immunotherapy. Nat. Rev. Cancer 19 , 133–150 (2019).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Keenan, T. E., Burke, K. P. & Van Allen, E. M. Genomic correlates of response to immune checkpoint blockade. Nat. Med. 25 , 389–402 (2019).

Sanmamed, M. F., Berraondo, P., Rodriguez-Ruiz, M. E. & Melero, I. Charting roadmaps towards novel and safe synergistic immunotherapy combinations. Nat. Cancer 3 , 665–680 (2022).

Article   PubMed   Google Scholar  

Kubli, S. P., Berger, T., Araujo, D. V., Siu, L. L. & Mak, T. W. Beyond immune checkpoint blockade: emerging immunological strategies. Nat. Rev. Drug Discov. 20 , 899–919 (2021).

Martins, F. et al. Adverse effects of immune-checkpoint inhibitors: epidemiology, management and surveillance. Nat. Rev. Clin. Oncol. 16 , 563–580 (2019).

Bruni, D., Angell, H. K. & Galon, J. The immune contexture and Immunoscore in cancer prognosis and therapeutic efficacy. Nat. Rev. Cancer 20 , 662–680 (2020).

Chan, T. A. et al. Development of tumor mutation burden as an immunotherapy biomarker: utility for the oncology clinic. Ann. Oncol. 30 , 44–56 (2019).

Saigi, M., Alburquerque-Bejar, J. J. & Sanchez-Cespedes, M. Determinants of immunological evasion and immunocheckpoint inhibition response in non-small cell lung cancer: the genetic front. Oncogene 38 , 5921–5932 (2019).

Chowell, D. et al. Improved prediction of immune checkpoint blockade efficacy across multiple cancer types. Nat. Biotechnol. 40 , 499–506 (2022).

Litchfield, K. et al. Meta-analysis of tumor- and T cell-intrinsic mechanisms of sensitization to checkpoint inhibition. Cell 184 , 596–614.e14 (2021).

Chen, X. et al. CD8 + T effector and immune checkpoint signatures predict prognosis and responsiveness to immunotherapy in bladder cancer. Oncogene 40 , 6223–6234 (2021).

Charoentong, P. et al. Pan-cancer immunogenomic analyses reveal genotype–immunophenotype relationships and predictors of response to checkpoint blockade. Cell Rep. 18 , 248–262 (2017).

Bindea, G. et al. Spatiotemporal dynamics of intratumoral immune cells reveal the immune landscape in human cancer. Immunity 39 , 782–795 (2013).

Angelova, M. et al. Characterization of the immunophenotypes and antigenomes of colorectal cancers reveals distinct tumor escape mechanisms and novel targets for immunotherapy. Genome Biol. 16 , 64 (2015).

Article   PubMed   PubMed Central   Google Scholar  

Newman, A. M. et al. Robust enumeration of cell subsets from tissue expression profiles. Nat. Methods 12 , 453–457 (2015).

Priestley, P. et al. Pan-cancer whole-genome analyses of metastatic solid tumours. Nature 575 , 210–216 (2019).

Martínez-Jiménez, F. et al. Pan-cancer whole genome comparison of primary and metastatic solid tumors. Nature 618 , 333–341 (2023).

Schwartz, L. H. et al. RECIST 1.1—update and clarification: from the RECIST committee. Eur. J. Cancer 62 , 132–137 (2016).

Benjamini, Y. & Yekutieli, D. The control of the false discovery rate in multiple testing under dependency. Ann. Stat. 29 , 1165–1188 (2001).

Article   Google Scholar  

Martínez-Jiménez, F. et al. Genetic immune escape landscape in primary and metastatic cancer. Nat. Genet. 55 , 820–831 (2023).

Ma, X. et al. Functional landscapes of POLE and POLD1 mutations in checkpoint blockade-dependent antitumor immunity. Nat. Genet. 54 , 996–1012 (2022).

Tamborero, D. et al. A pan-cancer landscape of interactions between solid tumors and infiltrating immune cell populations. Clin. Cancer Res. 24 , 3717–3728 (2018).

Subramanian, A. et al. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. Proc. Natl Acad. Sci. 102 , 15545–15550 (2005).

Liberzon, A. A description of the molecular signatures database (MSigDB) web site. Methods Mol. Biol. 1150 , 153–160 (2014).

Mariathasan, S. et al. TGFβ attenuates tumour response to PD-L1 blockade by contributing to exclusion of T cells. Nature 554 , 544–548 (2018).

Tauriello, D. V. F. et al. TGFβ drives immune evasion in genetically reconstituted colon cancer metastasis. Nature 554 , 538–543 (2018).

Cindy Yang, S. Y. et al. Pan-cancer analysis of longitudinal metastatic tumors reveals genomic alterations and immune landscape dynamics associated with pembrolizumab sensitivity. Nat. Commun. 12 , 5137 (2021).

Foy, J.-P. et al. Datasets for gene expression profiles of head and neck squamous cell carcinoma and lung cancer treated or not by PD1/PD-L1 inhibitors. Data Brief. 44 , 108556 (2022).

Campbell, K. M. et al. Prior anti-CTLA-4 therapy impacts molecular characteristics associated with anti-PD-1 response in advanced melanoma. Cancer Cell 41 , 791–806.e4 (2023).

Ravi, A. et al. Genomic and transcriptomic analysis of checkpoint blockade response in advanced non-small cell lung cancer. Nat. Genet. 55 , 807–819 (2023).

Friedman, J. H. Greedy function approximation: a gradient boosting machine. Ann. Stat. 29 , 1189–1232 (2001).

Addeo, A., Friedlaender, A., Banna, G. L. & Weiss, G. J. TMB or not TMB as a biomarker: that is the question. Crit. Rev. Oncol. Hematol. 163 , 103374 (2021).

Addala, V. et al. Computational immunogenomic approaches to predict response to cancer immunotherapies. Nat. Rev. Clin. Oncol. 21 , 28–46 (2024).

McGranahan, N. et al. Allele-specific HLA loss and immune escape in lung cancer evolution. Cell 171 , 1259–1271.e11 (2017).

Sears, T. et al. Integrated germline and somatic features reveal divergent immune pathways driving ICB response. Preprint at https://doi.org/10.1101/2024.01.12.575430 (2024).

Krishna, C. et al. An immunogenetic basis for lung cancer risk. Science 383 , eadi3808 (2024).

Verma, V. et al. A systematic review of the cost and cost-effectiveness studies of immune checkpoint inhibitors. J. Immunother. Cancer 6 , 128 (2018).

Pabla, S. et al. Proliferative potential and resistance to immune checkpoint blockade in lung cancer patients. J. Immunother. Cancer 7 , 27 (2019).

McDermott, D. F. et al. Clinical activity and molecular correlates of response to atezolizumab alone or in combination with bevacizumab versus sunitinib in renal cell carcinoma. Nat. Med. 24 , 749–757 (2018).

Rousseeuw, P. J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20 , 53–65 (1987).

Kolde, R., Laur, S., Adler, P. & Vilo, J. Robust rank aggregation for gene list integration and meta-analysis. Bioinformatics 28 , 573–580 (2012).

Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. In Proc. 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (Association for Computing Machinery, 2016). https://doi.org/10.1145/2939672.2939785

Saura, C. et al. Early-stage breast cancer detection in breast milk. Cancer Discov. 13 , 2180–2191 (2023).

Hernando-Calvo, A. et al. A pan-cancer clinical platform to predict immunotherapy outcomes and prioritize immuno-oncology combinations in early-phase trials. Med 4 , 710–727.e5 (2023).

Download references

Acknowledgements

We wish to acknowledge the contribution of patients, families and biomedical researchers who shared, processed and sequenced the data used in the study. This publication and the underlying study have been made possible partly based on the data that HMF has made available. We also acknowledge the use of data from the INSPIRE clinical trial (NCT02644369) and data obtained from patients presented at the VHIO. N.L.-B. acknowledges funding from the European Research Council (ERC; consolidator grant 682398). This project has received funding from the European Union’s Horizon program HORIZON-HLTH-2021-CARE-05-02 for the project CGI-Clinics under grant agreement no. 101057509. E.B. receives support from Generalitat de Catalunya (2021 SGR 001278) and ERC Advanced Grant 884623. The Institute for Research in Biomedicine Barcelona is a recipient of a Severo Ochoa Centre of Excellence Award from the Spanish Ministry of Economy and Competitiveness (MINECO; Government of Spain) and an Excellence Institutional grant by the Asociacion Española contra el Cancer and is supported by CERCA (Generalitat de Catalunya).

Author information

Axel Rosendahl Huber & Ferran Muiños

Present address: Centro de Investigación Biomédica en Red en Cáncer (CIBERONC), Instituto de Salud Carlos III, Madrid, Spain

These authors contributed equally: Axel Rosendahl Huber, Maria A. Andrianova.

Authors and Affiliations

Institute for Research in Biomedicine (IRB Barcelona), The Barcelona Institute of Science and Technology, Barcelona, Spain

Joseph Usset, Axel Rosendahl Huber, Maria A. Andrianova, Eduard Batlle, Ferran Muiños, Abel Gonzalez-Perez & Nuria Lopez-Bigas

Vall d’Hebron Institute of Oncology (VHIO), Vall d’Hebron Barcelona Hospital Campus, Barcelona, Spain

Joseph Usset, Joan Carles, Elena Elez, Enriqueta Felip, Marina Gómez-Rey, Deborah Lo Giacco, Francisco Martinez-Jimenez, Eva Muñoz-Couselo, Josep Tabernero & Ana Vivancos

Hartwig Medical Foundation, Amsterdam, Netherlands

Joseph Usset, Edwin Cuppen & Francisco Martinez-Jimenez

Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain

Eduard Batlle & Nuria Lopez-Bigas

Centro de Investigación Biomédica en Red en Cáncer (CIBERONC), Instituto de Salud Carlos III, Madrid, Spain

Eduard Batlle, Josep Tabernero, Abel Gonzalez-Perez & Nuria Lopez-Bigas

Medical Oncology Department, Vall d’Hebron University Hospital, Vall d’Hebron Barcelona Hospital Campus, Barcelona, Spain

Joan Carles, Elena Elez, Enriqueta Felip, Eva Muñoz-Couselo & Josep Tabernero

Center for Molecular Medicine, University Medical Center Utrecht, Utrecht, Netherlands

Edwin Cuppen

Division of Medical Oncology & Haematology, Princess Margaret Cancer Centre, University of Health Network, Department of Medicine, University of Toronto, Toronto, Ontario, Canada

Lillian L. Siu

Department of Medicine and Life Sciences, Universitat Pompeu Fabra, Barcelona, Spain

Abel Gonzalez-Perez & Nuria Lopez-Bigas

You can also search for this author in PubMed   Google Scholar

Contributions

A.G.-P. and N.L.-B. conceptualized the project. J.U. designed and carried out the exhaustive analysis, training, testing and validating multivariate methods, and performed most other analyses presented in the manuscript and supplement and participated in the discussion of results. A.R.H. and M.A.A. participated in the analysis and interpretation of latent factors. F.M. participated in the conceptualization of the analyses, provided support to perform them and participated in the discussion of results. A.G.-P. and N.L.-B. supervised the project, participated in the discussion of results and wrote the first draft of the manuscript. D.L.G., M.G. and A.V. provided the molecular data of patients in the VHIO cohort. E.F., E.E., J.C., E.M.-C. and J.T. provided the clinical data of patients in the VHIO cohort. A.V., E.F., J.T., F.M.-J., E.B., E.C. and L.L.S. contributed ideas to the design of analyses and participated in the discussion of results. All authors have read and approved the manuscript.

Corresponding authors

Correspondence to Abel Gonzalez-Perez or Nuria Lopez-Bigas .

Ethics declarations

Competing interests.

E.M.-C. reports a consultant or advisory role for Bristol Myers Squibb, Merck Sharp & Dohme, Novartis, Pierre Fabre, Roche and Sanofi; research funding from MSD, Sanofi and BMS; speaking engagements for Amgen, Bristol Myers Squibb, Merck Sharp & Dohme, Novartis and Pierre Fabre; clinical trial participation (as principal investigator) for Amgen, Bristol Myers Squibb, GlaxoSmithKline, Merck Sharp & Dohme, Novartis, Pierre Fabre, Roche and Sanofi. L.L.S. has a consultant/advisory role for Pfizer, AstraZeneca, Roche, GlaxoSmithKline, Voronoi, Arvinas, Navire, Relay, Marengo, Daiichi Sankyo, Amgen, Medicenna, LTZ Therapeutics, Tubulis, Nerviano, Pangea, Incyte and Gilead; received grant/research support (Institution—for clinical trials) from Merck, Novartis, Bristol Myers Squibb, Pfizer/SeaGen, Boerhinger-Ingelheim, GlaxoSmithKline, Roche, Genentech, AstraZeneca, Bayer, Abbvie, Amgen, Symphogen, EMD Serono, 23Me, Daiichi Sankyo, Gilead, Marengo, Incyte, LegoChem, Loxo/Eli Lilly, Medicenna and Takara; reports a leadership position (spouse) at Treadwell Therapeutics (founder) and stock ownership (spouse) in Agios. E.B. is the author of a patent related to TGF-β inhibitors, a patent describing bispecific antibodies to target cancer stem cells; E.B.'s lab has received research funding from MERUS, INCYTE and Revolution Medicines; and received honoraria for consulting from Genentech. J.T. reports personal financial interest in the form of scientific consultancy role for Alentis Therapeutics, AstraZeneca, Aveo Oncology, Boehringer Ingelheim, Cardiff Oncology, CARSgen Therapeutics, Chugai, Daiichi Sankyo, F. Hoffmann–La Roche, Genentech, hC Bioscience, Ikena Oncology, Immodulon Therapeutics, Inspirna, Lilly, Menarini, Merck Serono, Merus, MSD, Mirati, Neophore, Novartis, Ona Therapeutics, Ono Pharma USA, Orion Biotechnology, Peptomyc, Pfizer, Pierre Fabre, Samsung Bioepis, Sanofi, Scandion Oncology, Scorpion Therapeutics, Seattle Genetics, Servier, Sotio Biotech, Taiho, Takeda Oncology and Tolremo Therapeutics; stocks in Oniria Therapeutics, Alentis Therapeutics, Pangaea Oncology and 1TRIALSP; and an educational collaboration with Medscape Education, PeerView Institute for Medical Education and Physicians Education Resource (PER). E.E. has received personal honoraria from Amgen, Bayer, BMS, Boehringer Ingelheim, Cure Teq AG, Hoffman–La Roche, Janssen, Lilly, Medscape, Merck Serono, MSD, Novartis, Organon, Pfizer, Pierre Fabre, Repare Therapeutics, RIN Institute, Sanofi, Seagen International, Servier and Takeda. E.F. reports a consulting or advisory role with Abbvie, Amgen, AstraZeneca, Bayer, Boehringer Ingelheim, BMS, Daiichi Sankyo, F. Hoffmann–La Roche, Genmab, Gilead, GSK, Janssen, Johnson & Johnson, Merck Serono, MSD, Novartis, Peptomyc, Pfizer, Regeneron, Sanofi, Takeda; and speakers’ bureau for Amgen, AstraZeneca, BMS, Daiichi Sankyo, Eli Lilly, F. Hoffmann–La Roche, Janssen, Medical Trends, Medscape, Merck Serono, MSD, Peervoice, Pfizer Regeneron, Seagen, Touch Oncology; board of directors role with Grifols; principal investigator in trials (institutional financial support for clinical trials) sponsored by AstraZeneca, Abbvie, Amgen, Bayer, Beigene, Boehringer Ingelheim, BMS, Daiichi Sankyo, Exelixis, F. Hoffmann-La Roche, Genentech, GSK, Janssen, MSD, Merck KGAA, Mirati, Novartis, Nuvalent, Pfizer and Takeda. J.C. reports a role with the advisory board of Astellas Pharma, AstraZeneca, Bayer, Bristol Myers Squibb, Exelixis, Ipsen, Johnson & Johnson, MSD Oncology, Novartis (AAA), Pfizer and Sanofi; institutional funding received from Janssen-Cilag International NV, Lilly, S.A, Medimmune, Novartis Farmacéutica, S.A. and Sanofi-Aventis, S.A.; other role as Member of the Comission Catalan Program of Ambulatory Medication Comission (CAHMDA). The remaining authors declare no competing interests.

Peer review

Peer review information.

Nature Genetics thanks Kevin Litchfield for their contribution to the peer review of this work. Peer reviewer reports are available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended data fig. 1 identification of latent factors associated with cpi response and survival across the hmf-cpi cohort..

The figure provides a broad comparison of the landscape of features identified as significantly associated with CPI response (BOR), Progression Free Survival (PFS) and Overall Survival (OS) through the systematic use of univariate regression models corrected with different sets of covariables (see main manuscript and Supplementary Note 1 ). The three panels illustrate the results of the systematic analysis using no covariables ( a ), only the tissue as covariable ( b ), or the tissue, age, biopsy site and tumor purity as covariables ( c ) for the regressions. All analyses described in the main manuscript were carried out taking into account all covariables described in c. Features of different nature are colored following the same legend as in the main Figures. All p-values shown in the plots were computed via logistic (response) or Cox (survival) regressions, as in Figs. 2 and 3 of the main manuscript. These are, by definition, two-sided, denoted by positive or negative odds ratios (logistic regressions) or the reverse of hazard estimates (Cox regressions). OS: overall survival; PFS: progression-free survival; BOR: best overall response according to RECIST.

Extended Data Fig. 2 The five latent factors are integrated by highly correlated and significant features, and are mutually orthogonal.

All graphs present the relationship between the significance of the association between individual features with CPI response or survival and their correlation to the mean of the clusters of features representing each latent factor. a ) TMB cluster. Features integrating this latent factor are significantly associated with CPI response and survival. b ) Pretreatment cluster. Only very few features, all capturing different treatments, appear correlated with the mean of this cluster. Their association is also apparent with CPI response and survival. c ) Effective T-cell infiltration cluster. Features integrating this latent factor are significantly associated with CPI response and survival. d ) TGF-β activity in the microenvironment cluster. Features included in this cluster are highly correlated with the mean of the cluster, while some features included in the effective T-cell infiltration cluster show a moderate correlation (~0.5). These features are only significantly associated with CPI survival (including survival residuals), but not with response. e ) Proliferative potential cluster. These features are only significantly associated with CPI survival residuals. Features of different nature are colored following the same legend as in the main Figures. All p-values shown in the plots were computed via logistic (response) or Cox (survival) regressions, as in Figs. 2 and 3 of the main manuscript. These are, by definition, two-sided. OS: overall survival; PFS: progression-free survival; BOR: best overall response according to RECIST.

Extended Data Fig. 3 Interpretation of significant expression features using genesets.

a ) Heatmap representing the pairwise correlation between genesets highlighted in Fig. 2 of the main paper. b ) Significance of the association of 255 genesets with CPI survival residuals and their correlation with the mean of cluster S1 (left) and S2.1 (right). Significant genesets and correlation above 0 are highlighted. c ) Heatmap representing the pairwise correlations between genesets that appear significantly associated with CPI survival residuals and correlated with cluster S2.1. d ) All significant features from the volcano plot represented in Fig. 3e which do not belong to any of the response clusters previously identified (TMB, T-cell effective infiltration, prior treatment) were selected and clustered based on their pairwise correlations. One large cluster (along a few unclustered features) is apparent, called cluster Survival. e ) We computed the correlation of the mean value of the Survival cluster with 255 genesets. It was highly correlated with genesets representing the activity of TGF-β in the tumor microenvironment (purple dots). Other significant genesets (uncorrelated with cluster Survival) represent T-cell effective infiltration (red dots). f ) Pairwise correlations between all genesets that appear significantly associated with CPI overall survival not corrected by TMB, T-cell effective infiltration and prior treatment. Two clusters are apparent. One of them represents T-cell effective infiltration. The other represents TGF-β activity in the microenvironment. P-values shown in the plot were computed via logistic (response) or Cox (survival) regressions, as in Figs. 2 and 3 of the main manuscript. These are, by definition, two-sided. OS: overall survival; PFS: progression-free survival; BOR: best overall response according to RECIST.

Extended Data Fig. 4 Association of the five latent factors with anti-cancer systemic therapies other than CPI.

Association of the five latent factors with the response to treatment ( a ) and overall survival ( b ) of patients in the HMF cohort who received CPI (left) or other therapies (right). All patients with an annotation of having received a treatment (other than CPI) for the metastatic tumor and for which an annotation of the organ of origin of the primary tumor was available were included in this group (N = 2,497). In each of the graphs the horizontal dotted line represents the threshold of statistical significance, while the vertical dotted line separates the positive (increased response or survival) and negative (decreased response or survival) effects. The association of each of the latent factors with CPI response or survival has been assessed using a univariate regression (on the values of the representative of the latent factor computed across tumors). Hence, a circle in the top right quadrant denotes a latent factor significantly associated with a positive outcome (increased response or survival); a circle in the top left quadrant represents a latent factor associated with a negative outcome (decreased response or survival). A circle in either of the two bottom quadrants represents a latent factor not significantly associated with the outcome measured. P-values shown in the plots were computed via logistic (response) or Cox (survival) regressions, using as independent variable, in each case, the estimator of each latent factor. These are, by definition, two-sided, denoted by positive or negative odds ratios (logistic regressions) or the reverse of hazard estimates (Cox regressions).

Extended Data Fig. 5 The five latent factors capture all the signal of features associated with CPI response and survival.

a ) Features of different types significantly associated with CPI response or survival. The three first graphs correspond to Extended Data Figure 1C . The fourth graph presents the regression of survival residuals (that is, controlling for the features identified as associated with response) on all features. b ) Volcano plots resulting from the regression analyses presented in panel A, including only features with correlation coefficient above 0.8 with the mean of any latent factor. Significant features from all regression analyses show high correlation to the clusters’ mean (as the clusters are precisely constructed from them). Other non-significant features show equally high correlation with the clusters. c ) Volcano plots as in panels A and B, but showing only features with correlation coefficient below 0.3 to the mean of the clusters defining the latent factors. Only scattered features uncorrelated to the five latent factors appear significantly associated with CPI response or survival, indicating the absence of any other mutually orthogonal latent factor in the HMF-CPI cohort at the level of statistical significance set by the stringent False Discovery Rate used. Features of different nature are colored following the same legend as in the main Figures. The p-values and effect sizes shown result from logistic or Cox regressions. P-values shown in the plots were computed via logistic (response) or Cox (survival) regressions, as in Figs. 2 and 3 of the main manuscript. These are, by definition, two-sided. OS: overall survival; PFS: progression-free survival; BOR: best overall response according to RECIST.

Extended Data Fig. 6 Univariate analyses reveal the association of latent factors with CPI response across different tissues in the HMF-CPI cohort and six validation cohorts.

a ) Left panel: Forest plot illustrating the association (calculated through univariate regression models) of the five latent factors with CPI response and survival across groups of patients with different types of tumors in the HMF-CPI cohort. Right panel: Idem across six validation cohorts. Red or green dots denote clear association (regression coefficients estimate more than 1 (light) / 1.96 (dark) standard errors from 0) of a latent factor with response or survival, while gray dots denote lack of association. Dark color denotes significance of the association, while light color represents non-significant associations. In the forest plots, the dots represent the strength (coefficients estimated through multivariate logistic or Cox regression) of the association between the latent factor and response or survival across cohorts. The horizontal bars across dots denote the 95% confidence intervals. Gray dots represent latent factors whose estimates are within one standard error of 0, dots with light color (green or red) represent non-significant associations with coefficient estimates above (or below) one standard error of the 0, while dark colored dots represent significant associations. Green dots represent positive associations with improved outcomes (higher response odds or lower hazard ratio), while red dots represent negative associations (lower response or higher hazard ratio). b ) Stability of transcriptomics latent factors across validation cohorts. We computed the relationship between the distance of each feature to all the members of its cluster (defined in the HMF-CPI cohort) and all members of other clusters (silhouette score; Methods ). The silhouette scores thus computed for genes in the TGF-beta activity in the microenvironment across HMF-CPI and four validation cohorts are represented in the first five bar plots in the top panel. Two genes, one with relatively high silhouette score, and another showing more variability across all cohorts appear highlighted. The ranks of the genes (sorted according to their silhouette scores) are aggregated across all cohorts, and a significance score (reflecting genes that are ranked consistently better than expected) is computed (right-hand bar plot). The three graphs at the bottom of the panel represent the relationship between the silhouette score of the genes in each transcriptomics latent factor in the HMF-CPI cohort (x-axis) and their aggregated score (y-axis). Sample sizes for all datasets tested can be found in Supplementary Table 1 .

Extended Data Fig. 7 Relative importance of the five latent factors in the prediction of response or overall survival across patients in the HMF-CPI cohort.

The line plots represent the contribution of the values of each latent factor (Scaled feature values) across patients to the predictions cast by the response (BOR) and overall survival (OS) multivariate models. The effects are illustrated through the Shapley Values ( Methods and Supplementary Note 1 ). Thus, in each plot, the line corresponding to each latent factor follows the relative influence of the values of the feature used to measure the latent factor on the predictions obtained through the model across all patients. Lines with positive slope correspond to latent factors that increase either the probability of response or the hazards with the increase in their value. The bar plots below the line plots represent the overall importance of each latent factor (using the standard deviation of the Shapley values) across all predictions of each model in each cohort. a ) Representation of the relative importance of the latent factors in the prediction of response to CPI across the pan-cancer cohort and each tumor type separately within the HMF-CPI cohort. b ) Representation of the relative importance of the latent factors in the prediction of overall survival (hazards) to CPI across the pan-cancer cohort and each tumor type separately within the HMF-CPI cohort.

Extended Data Fig. 8 Comparison of response and survival models using Shapley values.

a ) Showing a comparison of response and survival hazard estimates. The points are color coded red for low responders (<10% probability response), yellow for medium responders (10-50% probability) and green for high responders(>50%). The estimates we obtained from XGboost models trained on representative biomarkers of the five latent factors across patients in the HMF-CPI cohort to predict CPI response and survival. b ) Exploring the determinants of the distribution of hazards across patients with low probability of response (scatterplot). The patients in this group have been subdivided into two smaller groups based on their predicted hazard, represented by dots of different shades of red separated by the horizontal line in the value of predicted hazard 1.5. The line plots represent the distribution (quantiles) of Shapley values (see Methods ) calculated for these two subgroups of patients for the five latent factors. The two lines appear more separated in the distributions of Shapley values of tumor proliferative potential and TGF-beta activity in the microenvironment. This indicates that it is the values of these two latent factors that contribute the most to the separation between these two groups of patients. c ) Example of the predicted CPI response and survival of one patient in the HMF-CPI cohort broken down by Shapley values.

Extended Data Fig. 9 Stratification of patients in validation cohorts using multivariate machine learning models.

a ) The histograms represent the distribution of the probability of response to CPI of patients across three of the validation cohorts (those with complete data on all five latent factors), either combined or separate. The bars are colored red (probability of response below 0.1, low), yellow (probability between 0.1 and 0.5, medium) or green (probability above 0.5, high). The absolute number of patients across the three cohorts in each group (low, medium, high) are shown in the horizontal bar below the combined histogram. The barplots below present the percentage of patients in each of the groups who actually showed response to CPI according to the data of each cohort. b ) Top panel: Kaplan-Meier curves resulting from the aforementioned stratification of patients across the three cohorts, either combined or separate. Bottom panel: Kaplan-Meier curves resulting from stratifying the patients across the three cohorts based on their predicted probability of survival according to the hybrid models trained on survival data. The p-value for each cohort (annotated in the plot) was calculated via a one-sided logrank test.

Extended Data Fig. 10 Application of multivariate machine learning models to identify patients with high probability to respond to CPI across the entire HMF cohort.

Bars represent the number of patients with metastatic tumors from different sites of origin in the HMF cohort who received (top) or did not receive (bottom) CPI as treatment. All patients with an annotation of having received a treatment (other than CPI) for the metastatic tumor and for which an annotation of the organ of origin of the primary tumor was available were included in this group (N = 2,497). The colored segments in the bars at the left represent the absolute number of patients with low (below 0.1), medium (between 0.1 and 0.5) or high (above 0.5) predicted probability of response. These bars have been separated based on the total number of patients of each tumor type, and x-axes representing the relative scales of each plot have been added. To the right side of the plot, the percentage of patients of each tumor type including more than 15 cases are represented as stacked bar plots, to facilitate comparability between tumor types. An important fraction of patients with tumors from the same origin as those in the HMF-CPI cohort (for example, in the lung) present high predicted probability of response to CPI. Interestingly, patients with tumors of other origins, who are not typically considered as candidates for CPI treatment also exhibit high predicted response probability.

Supplementary information

Supplementary information.

Supplementary Figs. 1–3 and Supplementary Note 1.

Reporting Summary

Peer review file, supplementary tables 1 and 2.

Supplementary Table 1. Description of all cohorts included in the study. Supplementary Table 2. Gene sets used to interpret transcriptomics clusters.

Supplementary Data 1

Supplementary Dataset 1. Results of the exhaustive analysis of the association of features with CPI response and survival.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Usset, J., Rosendahl Huber, A., Andrianova, M.A. et al. Five latent factors underlie response to immunotherapy. Nat Genet (2024). https://doi.org/10.1038/s41588-024-01899-0

Download citation

Received : 03 May 2024

Accepted : 07 August 2024

Published : 12 September 2024

DOI : https://doi.org/10.1038/s41588-024-01899-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Cancer newsletter — what matters in cancer research, free to your inbox weekly.

• Action/Adventure
• Children's/Family
• Documentary/Reality
• Amazon Prime Video

More From Decider

New Shows & Movies To Watch This Weekend: 'The Perfect Couple' on...

‘Bachelorette’ Finale Drama Spills Onto TikTok: Maria Georgas Admits...

'The Perfect Couple' Ending Explained: Who Was the Killer?

Fall TV Preview 2024: 'Yellowstone,' 'Outlander,' and So Much More

'The Bachelorette' Did Jenn Tran Dirty With Unbelievably Cruel Finale

Fall Streaming Movie Preview: 13 Movies We’re Hyped For, From...

11 Best New Movies on Netflix: September 2024's Freshest Films to Watch

Joey Chestnut Addresses Longstanding Takeru Kobayashi Rivalry Ahead Of...

Share this:.

• Click to share on Facebook (Opens in new window)
• Click to share on Twitter (Opens in new window)
• Click to share on WhatsApp (Opens in new window)
• Click to email a link to a friend (Opens in new window)
• Click to copy URL

‘The Office’ Spinoff Announces Additional Cast, Who Went Method By Shadowing Real Small-Town Newspaper Reporters

Where to stream:.

• Sabrina Impacciatore

John Krasinski Says Steve Carell Made Him Weep On The ‘IF’ Set After Giving Him “The Greatest Brother Speech”

Steve carell reveals whether he’ll be involved in new ‘the office’ series on peacock, peacock confirms new ‘the office’ series is a go — and will feature a ragtag group of midwestern newspaper reporters , ‘the office’ follow-up will star domhnall gleeson and ‘white lotus’ star sabrina impacciatore: report.

The Office spinoff cast just got bigger!

Chelsea Frei , Melvin Gregg and Ramona Young have all been cast in the new Peacock mockumentary series, which is rumored to be titled The Paper . They will be joining Domhnall Gleeson and Sabrina Impacciatore , who appear to be leading the Office spinoff cast.

Palisades Post , a local newspaper based in the California town of Pacific Palisades, announced the news while revealing that the actors shadowed members of the paper’s staff to “gain a deeper understanding of the inner-workings of a local publication.”

Their roles in the show remain unclear, but the actors reportedly shadowed newspaper staff who work in ad sales, reporting, circulation and graphic design.

The spinoff series, which is being helmed by The Office U.S. creator Greg Daniels and Michael Koman, is set in the same universe as the NBC classic, but is not a reboot or remake.

Per a release from Peacock, the logline for the project reads, “The documentary crew that immortalized Dunder Mifflin’s Scranton branch is in search of a new subject when they discover a dying historic Midwestern newspaper and the publisher trying to revive it with volunteer reporters.”

Since the show is set in the same universe as Dunder Mifflin, The Office fans may be wondering if that means there will be some crossover with America’s favorite paper company. But Steve Carell sadly dispelled those rumors .

“I will be watching but I will not be showing up,” Carell told The Hollywood Reporter . “It’s just a new thing and there’s really no reason for my character [Michael Scott] to show up in something like that.”

However, he praised Gleeson ahead of the new project. “I’m excited about it, it sounds like a great conceit. I love the idea — I guess it’s set in a family newspaper company,” he continued. “And I worked with Domhnall Gleeson, who is one of the leads. I did  The Patient  with him and he’s an excellent actor and a super nice guy so I think it’ll be great.”

It remains unclear when fans should expect the show to premiere, but Variety reported that production was slated to begin this summer.

And there could be more casting news coming down the line. Deadline reported in May that casting will continue “with a focus on the office staff and potentially one more series lead.”

'Jane The Virgin' Is Leaving Netflix — Where To Watch The Beloved Gina Rodriguez Romantic Dramedy Online

Joy Behar Flusters Gov. Josh Shapiro After Turning His Remark About Hooking Up With Ana Navarro Into Something Dirty: "Was That On-Air?"

'The Secret Lives Of Mormon Wives' Star Demi Engemann Says Her Husband Was Married To This 'RHOSLC' Star: "Utah Is Very Small"

Does 'Yellowstone' Return Tonight? 'Yellowstone' Season 5, Episode 9 Paramount Network Premiere Date

Joy Behar Repulsed As Dax Shepard Tells 'The View' What He Did When He First Met Kristen Bell: "That's Disgusting"

How To Watch Tonight's Packers-Eagles Friday Night Football Game: Channel, Free Live Stream Info