Reproducibility of research results is a fundamental property of high-quality research. (Royal Netherlands Academy of Arts and Sciences)
However, recent failures to replicate scientific work highlight the necessity for new initiatives to ensure reproducibility. CheckResearch.org is a community effort to validate and preserve scientific research by offering a platform to increase the reproducibility of research findings. It focuses on the domain of computer science, in which software and data are typically the main artifacts.
Both, the authors and the readers of scientific research benefit from an independent validation. The community checks if their approaches are described well enough for reproduction. Furthermore, their provided artifacts are tested for completeness and executability. In addition, varying setups and new data sources may deliver new insights to the authors’ approaches. Finally yet importantly, their approaches are presented to the public. Authors can also use CheckResearch.org to demonstrate step-by-step how they conducted their initial research, as template for the public. The public profits by independent validations of scientific research and particularly by a tested explanation how the scientific results can be reproduced thanks to the effort of individuals from the community. It hence fosters understanding and adoption of the scientific work. Moreover, we invite reviewers to share their evaluation results (of published work).
CheckResearch.org focuses on increasing the reproducibility of scientific results; it is not a mere artifact repository. Hence, the original artifacts can be gathered from existing repositories (such as findresearch.org), publishers or by contacting the authors directly.
The terms reproducibility and replicability are often used synonymous and the exact meaning of these terms is often subject to discussion. We define reproducibility and replicability according to the definition of Pellizzari et al.:
- A reproduction is an experiment targeting the same or a similar outcome. The more different the experiment conditions are (setting, environment, equipment), the more likely it is that the reproduction delivers new insights.
- A replication is a special form of a reproduction, namely one performed under identical conditions as the original experiment. We furthermore add the following clause for papers within the scope of computer science: If other people carry out the same experiment (i.e. using the same source code and data) we define it as replication as well.