Meta-Psychology publishes theoretical and empirical contributions that advance psychology as a science through critical discourse related to individual articles, research lines, research areas, or psychological science as a field.
Replication Value Increases With Transparency, Test Severity, and Societal Impact
This comment argues that replications should not be prioritized as a function of citation count and sample size, but instead as a function of societal impact, test severity, and the information value of a replication.
Copyright (c) 2025 Rene Bekkers https://creativecommons.org/licenses/by/4.0/
Replication value as a function of citation impact and sample size: response to commentaries
The primary goal of our target article (Isager et al., 2025) is to give the research community an example of what a well-justified replication value metric could look like, and to encourage discussion of how replication value could be quantified in practice. Furthermore, in the target article we discuss practical hurdles to quantification and possible practical applications for RVCn and other metrics. As that article proposes a method for how to do research–in this case a method to select which claims in the literature need replication most–it is important to receive criticism, feedback, and viewpoints from a diverse range of authors interested in this topic. We are delighted to read the many thoughtful yet critical commentaries, several of which proposing adjustments or alternatives to the equations we have proposed in the target article. This is very encouraging to see, as our aim with initiating this call in Meta-Psychology was to create an open dialogue in the scientific record. RVCn is an efficient but limited metric. Its limitations should be laid bare, and we fully expect that improved metrics and selection procedures can be created in the future. We hope our target article and these commentaries together will inspire readers to continue the discussion of how to efficiently and transparently select studies for replication. In this rejoinder we will summarize what we see as the major themes touched on in the commentaries, and we will reply to some of the specific proposals and criticisms brought up by different commentary authors.
Copyright (c) 2025 Peder Isager, Anna E. van 't Veer, Daniël Lakens https://creativecommons.org/licenses/by/4.0/
Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology
Traditional metric indicators of scientific productivity (e.g., journal impact factor; h-index) have been heavily criticized for being invalid and fueling a culture that focuses on the quantity, rather than the quality, of a person's scientific output. There is now a wide-spread demand for viable alternatives to current academic evaluation practices. In a previous report, we laid out four basic principles of a more responsible research assessment in academic hiring and promotion processes (Schönbrodt et al., 2025). The present paper offers a specific proposal for how these principles may be implemented in practice: We argue in favor of broadening the range of relevant research contributions and propose a set of concrete quality criteria (including a ready-to-use online tool) for research articles. These criteria are supposed to be used primarily in the first phase of the assessment process. Their function is to help establish a minimum threshold of methodological (i.e., theoretical and empirical) rigor that candidates need to pass in order to be further considered for hiring or promotion. In contrast, the second phase of the assessment process focuses more on the actual content of candidates' research and necessarily uses more narrative means of assessment. The debate over ways of replacing current invalid evaluation criteria with ones that relate more closely to scientific quality continues. Its course and outcome will depend on the willingness of researchers to get involved and help shape it.
Copyright (c) 2025 Anne Gärtner, Daniel Leising, Nele Freyer, Philipp Musfeld, Jens Lange, Felix D. Schönbrodt https://creativecommons.org/licenses/by/4.0/
Responsible Research Assessment (Parts I and II): Responses to the Commentaries
We give a brief overview of our deliberations in responding to the commentaries on our two target papers presenting the RESQUE (Research Quality Evaluation) framework. While we were able to incorporate many suggestions for improvement directly, we acknowledge that other areas (e.g., quality in theorizing) will require further elaboration. In this paper, we specifically touch on the following issues: (a) eligible types of publication, (b) measurability, (c) quality criteria for software and datasets, (d) theoretical rigor, (e) quantity, (f) authorship, (g) potential bias (against certain methodologies, types of research contributions, or subdisciplines), (h) overall rigor score, (i) weighting of individual indicators, (j) types of data, (k) impact, (l) interdisciplinary value, (m) teaching, (n) expertise, (o) gaming the new metrics, and (p) representativeness. The RESQUE framework has met with largely positive reception so far, but continues to evolve and will thrive best when community involvement stays strong.
Copyright (c) 2025 Daniel Leising, Anne Gärtner, Felix Schönbrodt https://creativecommons.org/licenses/by/4.0/
Responsible Research Assessment I: Implementing DORA and CoARA for hiring and promotion in psychology
The use of journal impact factors and other metric indicators of research productivity, such as the h-index, has been heavily criticized for being invalid for the assessment of individual researchers and for fueling a detrimental "publish or perish" culture. Multiple initiatives call for developing alternatives to existing metrics that better reflect quality (instead of quantity) in research assessment. This report, written by a task force established by the German Psychological Society, proposes how responsible research assessment could be done in the field of psychology. We present four principles of responsible research assessment in hiring and promotion and suggest a two-phase assessment procedure that combines the objectivity and efficiency of indicators with a qualitative, discursive assessment of shortlisted candidates. The main aspects of our proposal are (a) to broaden the range of relevant research contributions to include published data sets and research software, along with research papers, and (b) to place greater emphasis on quality and methodological rigor in research evaluation.
Copyright (c) 2025 Felix D. Schönbrodt, Anne Gärtner, Maximilian Frank, Mario Gollwitzer, Malika Ihle, Dorothee Mischkowski, Le Vy Phan, Manfred Schmitt, Anne M. Scheel, Anna-Lena Schubert, Ulf Steinberg, Daniel Leising https://creativecommons.org/licenses/by/4.0/
The value of replications goes beyond replicability and is associated with the value of the research it replicates: Commentary on Isager et al. (2025)
Replications are essential for rigorous credible science yet are still grossly undervalued and very rare. The value of replications is directly tied to the value of the research they aim to replicate, and replications offer many benefits that go far beyond the mere testing of replicability, such as including verifications and error detection, promoting long-term reproducibility of all research outputs, clarifying theory, refining measurement, and testing generalizability. We need far more independent pre-registered well-powered direct replications to strengthen the credibility of scientific findings. Isager et al. (2025)’s aim to define a formula for the value of replications based on over-simplified metrics of citation count and sample size is misaligned, already misunderstood, and may backfire by hindering the pursuit and publication of replications.
Copyright (c) 2025 Gilad Feldman https://creativecommons.org/licenses/by/4.0/
This commentary is a response to Isager, P. M., van ’t Veer, A. E. & Lakens, D. (2025): Replication value as a function of citation impact and sample size. It argues that, in assessing "the value of of being correct about the truth status of a claim," it is important to try to capture nonscientific impact. This commentary focuses in particular on the impact that original research can have in a legal context.
Copyright (c) 2025 Edith Beerdsen https://creativecommons.org/licenses/by/4.0/
Replication Value Usage and its Performance for Large Sample Sizes - Commentary on Isager et al. (2025)
The Replication Value (RVCn) metric was introduced to help researchers prioritize studies for replication based on expected utility. While we welcome the introduction of this straightforward and systematic replication decision approach, we identify two limitations of the RVCn. First, when testing the “repeatability” of a study or systematically incorporating replication into a research workflow, the RVCn may not always be the most suitable metric to guide decisions. Use cases should consider the scope conditions of the metric. Second, the RVCn shows limited sensitivity in distinguishing between studies with large sample sizes. To address this, we propose a simple adjustment: a log transformation of the sample size component. This modification improves the metric’s discriminatory power for high-N studies and better aligns the (RVCn) with its intended purpose: guiding efficient and meaningful replication efforts.
Copyright (c) 2025 Linda Bomm, Delaney J. Peterson, Bert N. Bakker https://creativecommons.org/licenses/by/4.0/
Thinking Beyond RVCN: Addressing the Complexity of Replication Target Selection
Isager, van ’t Veer and Lakens (2025) proposed a quantitative operationalization of replication value (denoted RVCN), using average yearly citation count and sample size as proxies for value and uncertainty, respectively. In this commentary, we suggest that the approach of Isager et al., while a good theoretical departure point, oversimplifies the complex decision-making process that underpins replication target selection in practice. We present what we view as some issues with RVCN, notably the use of citation count and ambiguity as to whether RVCN is prescriptive or descriptive. We also present preliminary empirical evidence that RVCN diverges on its performance as a replication target selection method, compared with existing selection methods (such as those published by us in the past). We conclude with the recommendation that going forward, approaches should emphasize the multifaceted nature of replication target selection to maximize their practical utility.
Copyright (c) 2025 Merle-Marie Pittelkow, Sarahanne M. Field, Don van Ravenzwaaij https://creativecommons.org/licenses/by/4.0/
The Replication Dilemma: Potential Challenges in Measuring Replication Value—A Commentary on Isager et al. (2025)
The authors (Isager et al., 2025) start with the main assumption that researchers’ efforts toward replications are constrained by resources, and they propose a simple, practically scalable framework of research replication value that guides the researchers and the scientific community at large intending to achieve bigger bang for the buck. Specifically, the authors propose a framework that combines citation impact and sample size of the original articles as a metric for assessing replication value. This implies that original studies with higher scores on this metric can be prioritized for replication efforts. We thoroughly agree with the authors’ assumption and indeed support the view of working towards an optimal framework that helps the community achieve maximum research impact from the replication efforts. In this commentary, we propose to discuss three important limitations that have to be considered before using such metrics. We thoroughly agree with the authors' assumption, and indeed support the view of working towards an optimal framework that helps the community achieve maximum research impact from the replication efforts. In this commentary, we propose to discuss three important limitations that have to be considered before using such metrics.
Copyright (c) 2025 Adrien Fillon, Subramanya Prasad Chandrashekar https://creativecommons.org/licenses/by/4.0/
Psychology’s Theory Crisis, and why formal modelling cannot solve it
In light of psychology’s ‘theory crisis’, multiple authors have argued that adopting formalization and/or formal modelling would constitute a useful or even necessary step towards stronger psychological theory. In this article, I instead argue that formal modelling cannot solve the core problem the psychological ‘theory crisis’ refers to, which are the currently high degrees of contrastive and holistic underdetermination of our theories by our data. I do so by first introducing underdetermination as an explanatory framework for determining the evidential import of research findings for theories, and showing how both broader theoretical considerations and informal assumptions are key to this process. Then, I derive the aforementioned core problem from the current ‘theory crisis’ literature and tentatively explore its possible solutions. Lastly, I show that formal modelling is neither a necessary nor sufficient solution for either contrastive or holistic underdetermination, and that its uncritical adoption might instead worsen the crisis.
Copyright (c) 2025 Freek Oude Maatman https://creativecommons.org/licenses/by/4.0/
Use and Misuse of a Fast Approximation: Not a Criticism, but a Caution
P. Isager et al. (2025) propose a fast approximation of replication value, RVCn, that relies on citation count and sample size. This approximation is simple, transparent, and easy to implement across many studies. It can potentially help metascientists evaluate large collections of studies. However, RVCn is not a precise statement of fact; it should not substitute for detailed substantive and methodological arguments. I make two counterclaims: (1) studies with few citations might be worth replicating and (2) studies with large samples might be worth replicating. While RVCn can helpfully supplement researchers’ judgments, it should not substitute for researchers’ judgments.
Copyright (c) 2025 Carlisle Rainey https://creativecommons.org/licenses/by/4.0/
This is a commentary piece on the proposal by Isager et al. (2025) for a new metric, RVCn, designed to evaluate the replication value of psychological studies. We discuss the hope of utilizing the RVCn metric in undergraduate education and possible improvements on using some elements other than original sample size to derive this metric.
Copyright (c) 2025 Kaito Takashima, Yuki Yamada https://creativecommons.org/licenses/by/4.0/
Standalone Apps for Anxiety and Depression Show Promising Early Efficacy: A Commentary on Meta-Analytic Results
The current commentary is focused on the methods and conclusions drawn in a recent meta-analysis which evaluated the impact of standalone interventions in treating anxiety and depressive symptoms (Weisel et al., 2019). The current commentary discusses the large impact of methodological choices made to exclude transdiagnostic treatments and group heterogeneous treatments on study conclusions. Additionally, the current commentary evaluates these conclusions considering opposing from two additional meta-analytic findings. The current review concludes that more research is needed effects before drawing any definitive conclusions, but the current evidence base suggests that apps show promising early efficacy in treating anxiety and depressive symptoms.
Copyright (c) 2025 Nicholas Jacobson, William Chen, Raphael Huang https://creativecommons.org/licenses/by/4.0/
Reimagining Consumer Psychology through Open Science Principles
Consumer psychology is facing various challenges, including a lack of research integrity and unethical publishing practices. This commentary lists pivotal events and discusses related findings that point to the field's need for reform. Open Science principles are proposed as a transformative solution to promote transparency in data, methodology, access, and peer review. Consumer psychology can only be revitalized and regain credibility if it fully embraces these four pillars. Academic and professional associations with an impact on consumer psychology must set a good example by cultivating a culture of integrity and accountability in research and publishing. Consumer psychologists must educate future generations of researchers on research methodology and research ethics.
Copyright (c) 2025 Salim Moussa https://creativecommons.org/licenses/by/4.0/
In this commentary, we propose pre-replication as a necessary epistemic, analytic, and axiologic exercise to be carried out before each replication.
Copyright (c) 2025 Veli-Matti Karhulahti, Marcel Martončik, Matúš Adamkovič https://creativecommons.org/licenses/by/4.0/
Falsifying the Insufficient Adjustment Model: No Evidence for Unidirectional Adjustment From Anchors
After considering a more or less random number (i.e., an anchor), people’s subsequent estimates are biased toward that number. Such anchoring phenomena have been explained via an adjustment process that ends too early. We present a formalized version of the insufficient adjustment model, which captures the idea that decreasing the time that people have to adjust from anchors draws their estimates closer to the anchors. In four independent studies (N = 898), we could not confirm this effect of time on anchoring. Moreover, anchoring effects vanished in the two studies that deviated from classical paradigms by using a visual scale or a two-alternative forced-choice paradigm to allow faster responses. Although we propose that the current version of the insufficient adjustment model should be discarded, we believe that adjustment models hold the most potential for the future of anchoring research, and we make suggestions for what these might look like.
Copyright (c) 2025 Lukas Röseler, Lisa Incerti, Tobias R. Rebholz, Christian Seida, Frank Papenmeier https://creativecommons.org/licenses/by/4.0/
Many journals now require data sharing and require articles to include a Data Availability Statement. However, several studies over the past two decades have shown that promissory notes about data sharing are rarely abided by and that data is generally not available upon request. This has negative consequences for many essential aspects of scientific knowledge production, including independent verification of results, efficient secondary use of data, and knowledge synthesis. I assessed the prevalence of data sharing upon request in articles employing the Implicit Relational Assessment Procedure published within the last 5 years. Of 52 articles, 42% contained a Data Availability Statement, most of which stated that data was available upon request. This rose from 0% in 2018 to 100% in 2022, indicating a change in journals’ policies. However, only 27% of articles’ authors actually shared data. Among articles stating that data was available upon request, only 17% shared data upon request. The presence of Data Availability Statements was not associated with higher rates of data sharing (p = .55), indicating a lack of adherence with journals’ policies. Results replicate those found elsewhere: data is generally not available upon request, and promissory Data Availability Statements are typically not adhered to. Issues, causes, and implications are considered.
implicit relational assessment procedure
Copyright (c) 2025 Ian Hussey https://creativecommons.org/licenses/by/4.0/
Practicing Theory Building in a Many Modelers Hackathon
Scientific theories reflect some of humanity's greatest epistemic achievements. The best theories motivate us to search for discoveries, guide us towards successful interventions, and help us to explain and organize knowledge. Such theories require a high degree of specificity, which in turn requires formal modeling. Yet, in psychological science, many theories are not precise and psychological scientists often lack the technical skills to formally specify existing theories. This problem raises the question: How can we promote formal theory development in psychology, where there are many content experts but few modelers? In this paper, we discuss one strategy for addressing this issue: a Many Modelers approach. Many Modelers consists of mixed teams of modelers and non-modelers that collaborate to create a formal theory of a phenomenon. Here, we report a proof of concept of this approach, which we piloted as a three-hour hackathon at the Society for the Improvement of Psychological Science conference in 2021. After surveying the participants, results suggest that (a) psychologists who have never developed a formal model can become (more) excited about formal modeling + and theorizing; (b) a division of labor in formal theorizing is possible where only one or a few team members possess the prerequisite modeling expertise; and (c) first working prototypes of a theoretical model can be created in a short period of time. These results show some promise for the many modelers approach as a team science tool for theory development.
Copyright (c) 2025 Noah van Dongen, Adam Finnemann, Jill de Ron, Leonid Tiokhin, Shirley B. Wang, Johannes Algermissen, Elena C. Altmann, Štěpán Bahník, Li-Ching Chuang, Andrei Dumbravă, Jens H. Fünderich, Sandra J. Geiger, Daria Gerasimova, Aidai Golan, Judith Herbers, Marc Jekel, Anton Kunnari, Yih-Shiuan Lin, David Moreau, Yvonne Oberholzer, Hannah K. Peetz, Julia Rohrer, Adrian Rothers, Felix Schönbrodt, Yashvin Seetahul, Anna Szabelska, Natasha Tonge, Nicole Walasek, Marlene Werner, Denny Borsboom https://creativecommons.org/licenses/by/4.0/
Replication value as a function of citation impact and sample size
Researchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication effort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2023). In this article, we propose a quantitative operationalization of replication value. We first discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RVCn. Third, we suggest how RVCn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RVCn estimates. The goal of this article is to explain how RVCn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy.
Copyright (c) 2025 Peder Isager, Anna van 't Veer, Daniël Lakens https://creativecommons.org/licenses/by/4.0/
Overview on the Null Hypothesis Significance Test
For decades, waxing and waning, there has been an ongoing debate on the values and problems of the ubiquitously used null hypothesis significance test (NHST). With the start of the replication crisis, this debate has flared-up once again, especially in the psychology and psychological methods literature. Arguing for or against the NHST method usually takes place in essays and opinion pieces that cover some, but not all the qualities and problems of the method. The NHST literature landscape is vast, a clear overview is lacking, and participants in the debate seem to be talking past one another. To contribute to a resolution, we conducted a systematic review on essay literature concerning NHST published in psychology and psychological methods journals between 2011 and 2018. We extracted all arguments in defense of (20) and against (70) NHST, and we extracted the solutions (33) that were proposed to remedy (some of) the perceived problems of NHST. Unfiltered, these 123 items form a landscape that is prohibitively difficult to keep in one’s sights. Our contribution to the resolution of the NHST debate is twofold. 1) We performed a thematic synthesis of the arguments and solutions, which carves the landscape in a framework of three zones: mild, moderate, and critical. This reduction summarizes groups of arguments and solutions, thus offering a manageable overview of NHST’s qualities, problems, and solutions. 2) We provide the data on the arguments and solutions as a resource for those who will carry-on the debate and/or study the use of NHST.
Copyright (c) 2025 Noah Dongen, Leonie van Grootel https://creativecommons.org/licenses/by/4.0/
Comments on Cassese and Holman 2019 “Playing the Woman Card: Ambivalent Sexism in the 2016 U.S. Presidential Race”
During his campaign for the Republican Party nomination and for U.S. president, Donald Trump suggested that Hillary Clinton benefited from playing a “woman card”. The effect of exposure to Trump’s woman-card attack was investigated in the Cassese and Holman (2019) Political Psychology article “Playing the woman card: Ambivalent sexism in the 2016 U.S. presidential race”. However, neither Cassese and Holman (2019) nor a reanalysis of data analyzed in the article provided sufficient evidence for key claims in the article. Moreover, Cassese and Holman (2019) is unclear whether its Study 2 experimental data could be used to test claims made based on its Study 1 non-experimental data, providing an example of how journal policy requiring access to survey questionnaires could help peer reviewers and readers better assess reported research.
Copyright (c) 2025 Lawrence James Zigerell https://creativecommons.org/licenses/by/4.0/
The Evolution of Data Sharing Practices in the Psychological Literature
Sharing data has many benefits. However, data sharing rates remain low, for the most part well below 50%. A variety of interventions encouraging data sharing have been proposed. We focus here on editorial policies. Kidwell et al. (2016) assessed the impact of the introduction of badges in Psychological Science; Hardwicke, Mathur, et al. (2018) assessed the impact of Cognition’s mandatory data sharing policy. Both studies found policies to improve data sharing practices, but only assessed the impact of the policy for up to 25 months after its implementation. We examined the effect of these policies over a longer term by reusing their data and collecting a follow-up sample including articles published up until December 31st, 2019. We fit generalized additive models as these allow for a flexible assessment of the effect of time, in particular to identify nonlinear changes in the trend. These models were compared to generalized linear models to examine whether the non-linearity is needed. Descriptive results and the outputs from generalized additive and linear models were coherent with previous findings: following the policies in Cognition and Psychological Science, data sharing statement rates increased immediately and continued to increase beyond the timeframes examined previously, until reaching close to 100%. In Clinical Psychological Science, data sharing statement rates started to increase only two years following the implementation of badges. Reusability rates jumped from close to 0% to around 50% but did not show changes within the pre-policy nor the post-policy timeframes. Journals that did not implement a policy showed no change in data sharing rates or reusability over time. There was variability across journals in the levels of increase, so we suggest future research should examine a larger number of policies to draw conclusions about their efficacy. We also encourage future research to investigate the barriers to data sharing specific to psychology subfields to identify the best interventions to tackle them.
Copyright (c) 2025 Judith Neve, Guillaume A. Rousselet https://creativecommons.org/licenses/by/4.0/
Folk Conceptions of Free Will: A Systematic Review and Narrative Synthesis of Psychological Research
The existence of free will has been a subject of fierce academic debate for millennia, still the meaning of the term “free will” remains nebulous. In the past two decades, psychologists have made considerable progress in defining lay concepts of free will. We present the first systematic review of primary psychological evidence on how ordinary folk conceptualise free will, encompassing folk concepts, beliefs, intuitions, and attitudes about free will. A total of 1,384 records were identified following a pre-registered protocol. After abstract and full-text screening, 18 articles were eligible for inclusion, comprised of 36 studies and 10,176 participants from regions including the United States, Singapore, Hong Kong, India, Turkey, and Germany. A narrative synthesis of results showed that for ordinary folk, especially the more educated population from the United States, free will is a dynamic construct centred on the ability to choose following one’s goals and desires, whilst being uncoerced and reasonably free from constraints. Results suggesting metaphysical considerations regarding consciousness, dualism, and determinism were inconclusive. Our findings provided preliminary support for a psychological model of folk conception of free will, and elucidated potential pathways mediating the effects of consciousness and dualism on free will attributions. Further research is needed to explicate the distinction between having free will and having the ability to exercise free will, as well as the cross-cultural validity of findings on folk conceptions of free will.
Copyright (c) 2025 Alison Lam, Siu Kit Yeung https://creativecommons.org/licenses/by/4.0/
A Multi-faceted Mess: A Review of Statistical Power Analysis in Psychology Journal Articles
Many bodies recommend that a sample planning procedure, such as traditional NHST a priori power analysis, is conducted during the planning stages of a study. Power analysis allows the researcher to estimate how many participants are required in order to detect a minimally meaningful effect size at a specific level of power and Type I error rate. However, there are several drawbacks to the procedure that render it “a mess.” Specifically, the identification of the minimally meaningful effect size is very challenging, the procedure is not precision oriented, and does not guide the researcher to collect as many participants as feasibly possible. In this study, we explore how these three theoretical issues are reflected in applied psychological research in order to better understand whether these issues are concerns in practice. To investigate how power analysis is currently used, this study reviewed the reporting of 443 power analyses in high impact Psychology journals in 2016 and 2017 using Google Scholar. It was found that researchers rarely use the minimally meaningful effect size as a rationale for the chosen effect in a power analysis. Further, precision-based approaches and collecting the maximum sample size feasible are almost never used in tandem with power analyses. In light of these findings, we offer that researchers should focus on tools beyond traditional power analysis when sample planning, such as collecting the maximum sample size feasible.
Copyright (c) 2025 Nataly Beribisky, Udi Alter, Robert Cribbie https://creativecommons.org/licenses/by/4.0/