Testing the Usability of the Psychological Research Preregistration-Quantitative (PRP-QUANT) Template

Lisa Spitzer; Michael Bosnjak; Stefanie Mueller

doi:10.15626/MP.2023.4039

Downloads

PDF

Authors

Lisa Spitzer Leibniz Institute for Psychology (ZPID) https://orcid.org/0000-0002-4925-7291
Michael Bosnjak Trier University https://orcid.org/0000-0002-1431-8461
Stefanie Mueller Leibniz Institute for Psychology (ZPID) https://orcid.org/0000-0002-3611-6190

DOI:

https://doi.org/10.15626/MP.2023.4039

Keywords:

meta-research, open science, preregistration, reproducibility, replicability

Abstract

The Psychological Research Preregistration-Quantitative (PRP-QUANT) Template provides researchers with a comprehensive list of elements to consider when planning a psychological study. We assessed its usability and researchers’ intention to use it. We conducted a usability test (study 1) and surveyed researchers who submitted or reviewed a preregistration created with the template (study 2, authors: N = 19, reviewers: N = 29) regarding their impression of the template. For the usability test, we recruited participants via the mailing lists of the German Psychological Society, the American Psychological Association, and the British Psychological Society, and social media. Participants answered selected template and web probing items and provided an overall rating (N = 88). Based on the Unified Theory of Acceptance and Use of Technology (UTAUT), we expected that the intention to use the template is influenced by performance expectancy (moderated by age), effort expectancy (moderated by age and experience), and social influence (moderated by age, experience, and voluntariness, N = 60). The results suggest that the PRP-QUANT Template is suitable for different research areas within psychology, is evaluated as effective, and perceived positively. Performance expectancy and all predictors combined significantly predicted researchers’ intention to use the template. A plain language summary in English and German is available in Text S1 in the supplemental material.

Metrics

Metrics Loading ...

References

Bakker, M., Veldkamp, C. L. S., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18(12), e3000937.

Behr, D., Meitinger, K., Braun, M., & Kaczmirek, L. (2017). Web probing – Implementing probing techniques from cognitive interviewing in web surveys with the goal to assess the validity of survey questions. Mannheim, GESIS – Leibniz-

Institute for the Social Sciences (GESIS – Survey Guidelines).

Behrendt, S. (2014). Lm.beta: Add standardized regression coefficients to linear-model-objects [R package version 1.7-2]. https://CRAN.R-project.org/package=lm.beta

Beyer, F., Flannery, J., Gau, R., Janssen, L., Schaare, L., Hartmann, H., Nilsonne, G., Martin, S., Khalil, A., Lipp, I., Puhlmann, L., Heinrichs, H., Mohamed, A., Herholz, P., Sicorello, M., & Panagoulas, E. (2021). A fMRI pre-registration template. PsychArchives.

Blut, M., Chong, A., Tsiga, Z., & Venkatesh, V. (2021). Meta-analysis of the Unified Theory of Acceptance and Use of Technology (UTAUT): Challenging its validity and charting a research agenda in the red ocean. Journal of the Association for Information Systems, Forthcoming. https://ssrn.com/abstract=3834872

Bosnjak, M., Fiebach, C. J., Mellor, D., Mueller, S., O’Connor, D. B., Oswald, F. L., & Sokol, R. I. (2022). A template for preregistration of quantitative research in psychology: Report of the joint psychological societies preregistration task force. American Psychologist, 77(4), 602–615.

Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., & van ’t Veer, A. (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224.

Crüwell, S., & Evans, N. J. (2021). Preregistration in diverse contexts: A preregistration template for the application of cognitive models. Royal Society Open Science, 8(10), 210155.

Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160.

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191.

Hebbali, A. (2020). Olsrr: Tools for building ols regression models [R package version 0.5.3]. https://CRAN.R-project.org/package=olsrr

Heirene, R., LaPlante, D., Louderback, E. R., Keen, B., Bakker, M., Serafimovska, A., & Gainsbury, S. M. (2021). Preregistration specificity & adherence: A review of preregistered gambling studies & cross-disciplinary comparison (preprint). PsyArXiv.

Hope, R. M. (2022). Rmisc: Ryan miscellaneous [R package version 1.5.1]. https://CRAN.R-project.org/package=Rmisc

International Organization for Standardization. (2018). ISO 9241-11:2018: Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. https://www.iso.org/standard/63500.html

Jadil, Y., Rana, N. P., & Dwivedi, Y. K. (2021). A meta-analysis of the UTAUT model in the mobile banking literature: The moderating role of sample size and culture. Journal of Business Research, 132, 354–372.

Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLOS ONE, 10(8), e0132382.

Khechine, H., Lakhal, S., & Ndjambou, P. (2016). A meta-analysis of the UTAUT model: Eleven years later. Canadian Journal of Administrative Sciences / Revue Canadienne des Sciences de l’Administration, 33(2), 138–152.

Lakens, D. (2024). Examining the restrictiveness of the PRP-QUANT Template. Peer Community in Registered Reports.

Leiner, D. J. (2019). SoSci Survey [Computer software]. https://www.soscisurvey.de/

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 0021.

Neuwirth, E. (2022). Rcolorbrewer: Colorbrewer palettes [R package version 1.1-3]. https://CRAN.R-project.org/package=RColorBrewer

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences of the United States of America, 115(11), 2600–2606.

Ooms, J. (2021). Writexl: Export data frames to excel ’xlsx’ format [R package version 1.4.2]. https://CRAN.R-project.org/package=writexl

Parsons, S., Azevedo, F., Elsherif, M. M., Guay, S., Shahim, O. N., Govaart, G. H., Norris, E., O’Mahony, A., Parker, A. J., Todorovic, A., Pennington, C. R., Garcia-Pelegrin, E., Lazić, A., Robertson, O., Middleton, S. L., Valentini, B., McCuaig, J., Baker, B. J., Collins, E., … Aczel, B. (2022). A community-sourced glossary of open scholarship terms. Nature Human Behaviour, 6(3), 312–318.

Protzko, J., Krosnick, J., Nelson, L. D., Nosek, B. A., Axt, J., Berent, M., Buttrick, N., DeBell, M., Ebersole, C. R., Lundmark, S., MacInnis, B., O’Donnell, M., Perfecto, H., Pustejovsky, J. E., Roeder, S. S., Walleczek, J., & Schooler, J. (2020). High replicability of newly-discovered social-behavioral findings is achievable (preprint). PsyArXiv.

R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. https://www.R-project.org/

Revelle, W. (2021). Psych: Procedures for psychological, psychometric, and personality research [R package version 2.2.9]. Northwestern University. Evanston, Illinois. https://CRAN.R-project.org/package=psych

Shackel, B. (2009). Usability – Context, framework, definition, design and evaluation. Interacting with Computers, 21(5–6), 339–346.

Spitzer, L., & Mueller, S. (2023). Registered report: Survey on attitudes and experiences regarding preregistration in psychological research. PLOS ONE, 18(3), e0281086.

Spitzer, L., & Mueller, S. (2024). Stage 1 Registered Report: Restriction of researcher degrees of freedom through the Psychological Research Preregistration-Quantitative (PRP-QUANT) Template. PsyArXiv.

Swaen, G. G., Teggeler, O., & van Amelsvoort, L. G. (2001). False positive outcomes and design characteristics in occupational cancer epidemiology studies. International Journal of Epidemiology, 30(5), 948–954.

Van Den Akker, O. R., Bakker, M., Van Assen, M. A. L. M., Pennington, C. R., Verweij, L., Elsherif, M. M., Claesen, A., Gaillard, S. D. M., Yeung, S. K., Frankenberger, J.-L., Krautter, K., Cockcroft, J. P., Kreuer, K. S., Evans, T. R., Heppel, F., Schoch, S. F., Korbmacher, M., Yamada, Y., Albayrak-Aydemir, N., … Wicherts, J. M. (2023). The effectiveness of preregistration in psychology: Assessing preregistration strictness and preregistration-study consistency (preprint). MetaArXiv.

Van Den Akker, O. R., Van Assen, M. A. L. M., Bakker, M., Elsherif, M., Wong, T. K., & Wicherts, J. M. (2023). Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology. Behavior Research Methods, 56(6), 5424–5433.

Van den Akker, O. R., Weston, S., Campbell, L., Chopik, B., Damian, R., Davis-Kean, P., Hall, A., Kosie, J., Kruse, E., Olsen, J., Ritchie, S., Valentine, K., Van ’t Veer, A., & Bakker, M. (2021). Preregistration of secondary data analysis: A template and tutorial. Meta-Psychology, 5.

van ’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425.

Venkatesh, V., Thong, J. Y., & Xu, X. (2016). Unified theory of acceptance and use of technology: A synthesis and the road ahead. Journal of the Association for Information Systems, 17(5), 328–376. https://ssrn.com/abstract=2800121

Wei, T., & Simko, V. (2021). R package ‘corrplot’: Visualization of a correlation matrix [Version 0.92]. https://github.com/taiyun/corrplot

Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., … Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4(43), 1686.

Wickham, H., & Bryan, J. (2022). Readxl: Read excel files [R package version 1.4.2]. https://CRAN.R-project.org/package=readxl

Williams, M. D., Rana, N. P., & Dwivedi, Y. K. (2015). The unified theory of acceptance and use of technology (UTAUT): A literature review. Journal of Enterprise Information Management, 28(3), 443–488.