Evaluating the Replicability of Social Priming Studies
DOI:
https://doi.org/10.15626/MP.2022.3308Keywords:
priming, replication, preregistration, meta-analysisAbstract
To assess the replicability of social priming findings we reviewed the extant close replication attempts in the field. In total, we found 70 close replications, that replicated 49 unique findings. Ninety-four percent of the replications had effect sizes smaller than the effect they replicated and only 17% of the replications reported a significant p-value in the original direction. The strongest predictor of replication success was whether or not the replication team included at least one of the authors of the original paper. Twelve of the 18 replications with at least one original author produced a significant effect in the original direction and the meta-analytic average of these studies suggest a significant priming effect (d = 0.40, 95% CI[0.23; 0.58]). In stark contrast, none of the 52 replications by independent research teams produced a significant effect in the original direction and the meta-analytic average was virtually zero (d = 0.002, 95% CI[-0.03; 0.03]). We argue that these results have shifted the burden of proof back onto advocates of social priming. Successful replications from independent research teams will likely be required to convince sceptics that social priming exists at all.
Metrics
References
Armbruster, S. (2021). What makes a replication successful? An investigation of frequentist and Bayesian criteria to assess replication success [Doctoral dissertation, Ludwig Maximilian University of Munich].
Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J., Fiedler, K., & Wicherts, J. M. (2013). Replication is more than hitting the lottery twice. European Journal of Personality, 27, 138–138.
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554.
Bargh, J. A. (2006). What have we been priming all these years? On the development, mechanisms, and ecology of nonconscious social behavior. European Journal of Social Psychology, 36(2), 147–168.
Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71(2), 230–244.
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48.
Baumeister, R. F. (2016). Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology, 66, 153–158.
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Johnson, V. E., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2, 6–10.
Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., & Van’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for bias in psychology: A comparison of meta-analytic methods. Advances in Methods and Practices in Psychological Science, 2, 115–144.
Cesario, J. (2014). Priming, replication, and the hardest science. Perspectives on Psychological Science, 9(1), 40–48.
Cheung, I., Campbell, L., LeBel, E. P., Ackerman, R. A., et al. (2016). Registered replication report. Perspectives on Psychological Science, 11(5), 750–764.
Claesen, A., Gomes, S., Tuerlinckx, F., & Vanpaemel, W. (2021). Comparing dream to reality: An assessment of adherence of the first generation of pre-registered studies. Royal Society Open Science, 8(10), 211037.
Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. Psychological Review, 82(6), 407–428.
Cooper, H. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). SAGE Publications, Inc.
Dai, W., Yang, T., White, B. X., Palmer, R., Sanders, E. K., McDonald, J. A., Leung, M., & Albarracín, D. (2023). Priming behavior: A meta-analysis of the effects of behavioral and nonbehavioral primes on overt behavioral outcomes. Psychological Bulletin, 149(1-2), 67–98.
Dijksterhuis, A., & Bargh, J. A. (2001). The perception-behavior expressway: Automatic effects of social perception on social behavior. Advances in Experimental Social Psychology, 33, 1–40.
Dijksterhuis, A., & van Knippenberg, A. (1998). The relation between perception and behavior, or how to win a game of trivial pursuit. Journal of Personality and Social Psychology, 74(4), 865–877.
Doyen, S., Klein, O., Simons, D. J., & Cleeremans, A. (2014). On the other side of the mirror: Priming in cognitive and social psychology. Social Cognition, 32, 12–32.
Fiedler, K., & Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7, 45–52.
Francis, G. (2012). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19(2), 151–156.
Grimes, D. R., Bauch, C. T., & Ioannidis, J. P. A. (2018). Modelling science trustworthiness under publish or perish pressure. Royal Society Open Science, 5(171511), 171511.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23(5), 524–532.
Kühberger, A., Fritz, A., & Scherndl, T. (2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PloS One, 9(9), e105825.
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26.
Kvarven, A., Strømland, E., & Johannesson, M. (2019). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour, 4(4), 423–434.
Lakens, D., & Evers, E. R. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9(3), 278–292.
Molden, D. C. (2014). Understanding priming effects in social psychology: An overview and integration. Social Cognition, 32, 243–249.
Morey, R. D., & Lakens, D. (2016). Why most of psychology is statistically unfalsifiable. Unpublished manuscript.
Neely, J. H. (1977). Semantic priming and retrieval from lexical memory: Roles of inhibitionless spreading activation and limited-capacity attention. Journal of Experimental Psychology: General, 106(3), 226–254.
Newell, B. R., & Shanks, D. R. (2014). Unconscious influences on decision making: A critical review. Behavioral and Brain Sciences, 37, 1–19.
Nosek, B. A., & Errington, T. M. (2020). What is replication? PLoS Biology, 18(3), e3000691.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 1–8.
Pashler, H., Coburn, N., & Harris, C. R. (2012). Priming of social distance? Failure to replicate effects on social and food judgments. PLoS ONE, 7(8), e42510.
Patil, P., Peng, R. D., & Leek, J. T. (2016). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science, 11(4), 539–544.
Pecher, D., van Mierlo, H., Cañal-Bruland, R., & Zeelenberg, R. (2015). The burden of secrecy? No effect on hill slant estimation and beanbag throwing. Journal of Experimental Psychology: General, 144(4), e65–e72.
Peels, R., & Bouter, L. (2023). Replication and trustworthiness. Accountability in Research, 30(2), 77–87.
Rohrer, D., Pashler, H., & Harris, C. R. (2019). Discrepant data and improbable results: An examination of Vohs, Mead, and Goode (2006). Basic and Applied Social Psychology, 41(4), 263–271.
Schimmack, U. (2021). Replicability audit of Ap Dijksterhuis [Replicability-Index, April 23]. https://replicationindex.com/2021/04/23/r-audit-ap-dijksterhuis/
Shanks, D. R. (2017). Misunderstanding the behavior priming controversy: Comment on Payne, Brown-Iannuzzi, and Loersch (2016). Journal of Experimental Psychology: General, 146, 1216–1222.
Simons, D. J. (2014). The value of direct replication. Perspectives on Psychological Science, 9(1), 76–80.
Stroebe, W. (2019). What can we learn from many labs replications? Basic and Applied Social Psychology, 41, 91–103.
Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59–71.
Team, R. C. (2019). R: A language and environment for statistical computing.
van Elk, M., Matzke, D., Gronau, Q. F., Guan, M., Vandekerckhove, J., & Wagenmakers, E.-J. (2015). Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Frontiers in Psychology, 6, 1–7.
Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1–48.
Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2016). From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological Bulletin, 142(5), 472–497.
Wickham, H. (2016). Ggplot2: Elegant graphics for data analysis. Springer-Verlag New York. https://ggplot2.tidyverse.org
Published
Versions
- 2024-11-12 (2)
- 2024-11-11 (1)
Issue
Section
License
Copyright (c) 2024 Erik Mac Giolla, Simon Karlsson, David A. Neequaye, Magnus Bergquist
This work is licensed under a Creative Commons Attribution 4.0 International License.
