Towards a Specification for Recombinable Benchmarks and Software Tools

Simon Dierl  , Falk Howar

We consider research that is based on implementing software tools from novel ideas and evaluating them by executing the tools against benchmarks. Subsequent research project would benefit from freely recombining existing and novel tools and benchmarks to acquire data. However, existing approaches for distributing tools and artifacts do not allow recombination. We propose an approach for packaging and describing tools and benchmark that allows and supports recombination. We specify a tool packaging format, an approach to structuring benchmarks, and a descriptor file format to link the two. We applied these specifications in two case studies in Software Engineering and Environmental Engineering. For each case study, we considered the benefit gained and the overhead induced by introducing our approach, e.g., for file format conversions. While minor limitations became apparent, we showed both the viability of our proposed specification and its low overhead in terms of development effort.