Brown Bag Seminar

Topic: Enabling reproducibility through large-scale continuous scientific benchmarks for applications in Rosetta

Computational methods should be inherently reproducible. However, changes in the code, input options, and different test sets, as well as increasing complexities of applications and workflows lead to reporting of different accuracies. Further, tools are often developed through the use of benchmark sets and accuracies are often recorded on them. The challenge is that accuracies on a specific benchmark set are often higher for the tool which was trained on that set – and unfortunately these ‘advances’ are often published as true advances in the field. Moreover, benchmarking is often only accomplished when developing a new method or improving an established one. This ‘static’ benchmarking approach obscures which method or test set represents the true cutting edge of the field, especially since the performance depends on the benchmark set. Blind prediction challenges address these issues and allow critical assessments of methods, such as CASP for structure prediction, CAPRI for protein interactions, CAMEO for model evaluation, CAFA for functional annotation and CAGI for genome interpretation. However, testing methods in these prediction challenges typically requires months of commitment from the developers. Here we present a test server framework integrated with an HPC cluster that allows to continuously run scientific tests on a large scale for a multitude of Rosetta protocols. Our general setup addresses a number of challenges in research reproducibility and facilitates addition of future benchmarks with minimal requirements in time and expertise. Comprehensive documentation in both the setup of the benchmarks and accompanying each individual benchmark will aid in maintenance.

Join from PC, Mac, Linux, iOS or Android: https://simonsfoundation.zoom.us/j/92777740283?pwd=NTBpUUZQdTZXbGtYRWpOZnhKdHB5UT09