REGISTERED REPORTS AS A METHOD TO INCREASE CREDIBILITY OF SCIENCE – EXPERIMENTAL STUDY AMONG PSYCHOLOGICAL STUDENTS

Authors

DOI:

https://doi.org/10.15503/jecs20192.67.75

Abstract

Aims: The modern scientific publishing system suffers from many problems, amongst which one of the most important  is the pressure to publish positive results. A potentially simple way to mitigate publication bias are reviews of manuscript, prior to the data collection and analyses, as well as results presentation and discussion (i.e. Registered Reports).

Methods: To test this prediction we conducted a quasi-experiment: two groups of students (n=38), as a part of their academic classes, were asked to design and conduct research projects. They were divided in two groups – Registered Reports- and control group. In both groups students have been encouraged to publish their papers in a local scientific journal.

Results: Analyses revealed significant differences in p levels between groups, suggesting more reliable scores for Registered Reports group.

Conclusions: Our study partially confirmed the  stated hypothesis and suggested, that registered reports might mitigate publication bias. Future recommendations are advised.

Downloads

Download data is not yet available.

Author Biographies

Piotr Sorokowski, Institute of Psychology, University of Wrocław Dawida 1, Wrocław, Poland

Head of the Institute of Psychology at the University of Wroclaw, Poland. He has published over sixty research articles related to evolutionary, cultural and social psychology. His field studies in nonindustrial societies (Hadza of Tanzania, Tsimane’ of Bolivia, Yali of Papua) have shed new light on the interplay between evolution and culture on social and reproductive success.

Agata Groyecka, Institute of Psychology, University of Wrocław Dawida 1, Wrocław, Poland

The PhD student in Institute of Psychology at the University of Wroclaw. She is interested in social psychology and creativity psychology. She studies the link between creative cognition and intergroup relations.

Kamil Błaszczyński, Faculty of Psychology, University of Wrocław Ul. Dawida 1, Wrocław, Poland

Graduate of University of Wrocław. Ended master studies in pedagogy and defended his PhD thesis in Sociology. Employee in Institute of Psychology of University of Wrocław. His research interests concerne such topics as academic education, education of adults, statisticall analysis, IT usage in education.

Tomasz Frąckowiak, Faculty of Psychology, University of Wrocław Ul. Dawida 1, Wrocław, Poland

He works in Institute of Psychology at the University of Wroclaw. In his work, he focuses on psychological determinants of longevity, conditions of positive ageing, measures of older people’s quality of life and programs to increase their well-being. He authors many publications in the domain of psychology ageing, e.g. “Fenomen długowieczności. Perspektywa psychologiczna" [Phenomenon of longevity. Psychological perspective] (Scholar, 2019). Recently, he conducts research on cross-cultural differences in attitudes toward older people.

 

Aleksander Kobylarek, Institute of Pedagogy, University of Wrocław, , ul. Dawida 1/3, Poland

PhD in humanities, assistant professor at the University of Wrocław (Poland) in Departament of Pedagogy, manager of the University of the Third Age in the University of Wrocław up to 2016, author of more than 100 scientific publications, including articles, books, chapters, editor-in-chief of international scientific "Journal of Education Culture and Society" and "Ogrody Nauk i Sztuk (Gardens of Science and Arts).

References

Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531–533. https://doi.org/10.1038/483531a

Bem, S. L. (1974). The measurement of psychological androgyny. Journal of consulting and clinical psychology, 42(2), 155.

Callaway, E. (2011). Report finds massive fraud at Dutch universities. Nature, 479(7371), 15.

Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., ... & Altmejd, A. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637.

Enserink, M. (2012). Final Report on Stapel Also Blames Field As a Whole. Science, 338(6112), 1270–1271. https://doi.org/10.1126/science.338.6112.1270

Erlich, A. (2018). Pre-acceptance as a method to combat publication bias in area studies: a pilot in the Caucasus. Caucasus Survey, 6(3), 224-229.

Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE, 4(5), e5738.

Fanelli, D. (2010). “Positive” Results Increase Down the Hierarchy of the Sciences. PLoS ONE, 5(4), e10068. https://doi.org/10.1371/journal.pone.0010068

Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016). Can results-free review reduce publication bias? The results and implications of a pilot study. Comparative Political Studies, 49(13), 1667-1703.

Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The Extent and Consequences of PHacking in Science. PLOS Biology, 13(3), e1002106. https://doi.org/10.1371/journal.pbio.1002106

Jackson, C. J., Levine, S. Z., Furnham, A., & Burr, N. (2002). Predictors of Cheating Behavior at a University: A Lesson From the Psychology of Work. Journal of Applied Social Psychology, 32(5), 1031–1046. https://doi.org/10.1111/j.1559-1816.2002.tb00254.x

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

Kuczyńska, A. (1992). Inwentarz do oceny płci psychologicznej. Podręcznik. [The Psychological Sex Inventory] Warszawa: Pracownia Testów Psychologicznych PTP.

Lakens, D. (2015). On the challenges of drawing conclusions from p-values just below 0.05. PeerJ, 3, e1142. https://doi.org/10.7717/peerj.1142

Leimu, R., & Koricheva, J. (2004). Cumulative meta–analysis: a new tool for detection of temporal trends and publication bias in ecology. Proceedings of the Royal Society of London. Series B: Biological Sciences, 271(1551), 1961-1966.

Levine, T. R., Asada, K. J., & Carpenter, C. (2009). Sample Sizes and Effect Sizes are Negatively Correlated in Meta-Analyses: Evidence and Implications of a Publication Bias Against NonSignificant Findings. Communication Monographs, 76(3), 286–302. https://doi.org/10.1080/03637750903074685

Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537-542.

Morgan, G. A. (2000). Quasi-Experimental Designs. Journal of the American Academy of Child & Adolescent Psychiatry. 39. 794–796

McCabe, D. L., Treviño, L. K., & Butterfield, K. D. (2001). Cheating in academic institutions: A decade of research. Ethics & Behavior, 11(3), 219–232.

Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., Simonsohn U., Wagenmakers E.J., Ware J.J. & Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 21. https://doi.org/10.1038/s41562-016-0021

Nosek, B. A., & Lakens, D. (2014). Registered Reports: A Method to Increase the Credibility of Published Results. Social Psychology, 45(3), 137–141. https://doi.org/10.1027/1864-9335/a000192

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600-2606

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177/1745691612459058

Oleszkiewicz, A., Karwowski, M., Pisanski, K., Sorokowski, P., Sobrado, B., & Sorokowska, A. (2017a). Who uses emoticons? Data from 86 702 Facebook users. Personality and Individual Differences, 119, 289-295.

Oleszkiewicz, A., Frackowiak, T., Sorokowska, A., & Sorokowski, P. (2017b). Children can accurately recognize facial emotions from emoticons. Computers in Human Behavior, 76, 372-377.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716–aac4716. https://doi.org/10.1126/science.aac4716

Powell, K. (2016, November 4). Hard work, little reward: Nature readers reveal working hours and research challenges. Retrieved form http://www.nature.com/news/hard-work-little-reward-nature-readers-reveal-working-hours-and-research-challenges-1.20933

Probst, T. M., & Hagger, M. S. (2015). Advancing the Rigour and Integrity of Our Science: The Registered Reports Initiative: Editorial. Stress and Health, 31(3), 177–179. https://doi.org/10.1002/smi.2645

Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037/0033-2909.86.3.638

Saunders, R., & Savulescu, J. (2008). Research ethics and lessons from Hwanggate: what can we learn from the Korean cloning fraud? Journal of Medical Ethics, 34(3), 214–221. https://doi.org/10.1136/jme.2007.023721

da Silva, J. A. T. (2016). Does the removal of results from a submitted paper reduce publication bias?. Pacific Science Review B: Humanities and Social Sciences, 2(1), 29-30

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Simonsohn, U. (2013). Just Post It: The Lesson From Two Cases of Fabricated Data Detected by Statistics Alone. Psychological Science, 24(10), 1875–1888.https://doi.org/10.1177/0956797613480366

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014a). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. https://doi.org/10.1037/a0033242

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014b). p-Curve and effect size correcting for publication bias using only significant results. Perspectives on Psychological Science, 9(6), 666–681.

Sorokowski, P., Kulczycki, E., Sorokowska A., Pisanski, K. (2017). Predatory journals recruit fake editor. Nature, 543, 481-483.

Stahl, D., & Pickles, A. (2018). Fact or fiction: reducing the proportion and impact of false positives. Psychological Medicine, 48(7), 1084-1091

Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49(1), 108-112.

Stroebe, W., & Strack, F. (2014). The Alleged Crisis and the Illusion of Exact Replication. Perspectives on Psychological Science, 9(1), 59–71. https://doi.org/10.1177/1745691613514450

Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science, 4(3), 274–290. https://doi.org/10.1111/j.1745-6924.2009.01125.x

Wagenmakers, E., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011). Journal of Personality and Social Psychology, 100(3), 426–432. https://doi.org/10.1037/a0022790

Wicherts, J. M., Bakker, M., & Molenaar, D. (2011). Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results. PLoS ONE, 6(11), e26828. https://doi.org/10.1371/journal.pone.0026828

Wicherts, J., Veldkamp, C., Augusteijn, H., Bakker, M., van Aert, R., & van Assen, M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832

Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2017). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25, 1968-1972

Downloads

Published

2019-09-02

How to Cite

Sorokowski, P., Groyecka, A., Błaszczyński, K., Frąckowiak, T., & Kobylarek, A. (2019). REGISTERED REPORTS AS A METHOD TO INCREASE CREDIBILITY OF SCIENCE – EXPERIMENTAL STUDY AMONG PSYCHOLOGICAL STUDENTS. Journal of Education Culture and Society, 10(2), 67–75. https://doi.org/10.15503/jecs20192.67.75

Issue

Vol. 10 No. 2 (2019): Journal of Education Culture and Society

Section

TRANSGRESSION

License

CC-BY

Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal. All authors agree for publishing their email adresses, affiliations and short bio statements with their articles during the submission process.

Most read articles by the same author(s)

1 2 > >>