Pairwise comparisons or constrained optimization? A usability evaluation of techniques for eliciting decision priorities

Decision support methodologies provide notations for expressing and communicating the priorities that inform a decision. Although a substantial literature has explored the theoretical merits of such notations and methodologies, much less work has investigated their usability in practice, which is of vital importance for their widespread adoption by users. In this paper, we explore the usability of two well-known preference elicitation techniques, pairwise comparisons and constrained optimization. The techniques were explored through two contrasting crowd worker experiments, a preliminary one evaluating recognition, that is, the ability to identify the most suitable formulation for a given task, and the other synthesis, that is, the ability to construct formulations for a given task. The tasks are based on a case study involving source selection, a well-known problem in the data integration domain. The results of the empirical evaluation show that, overall, pairwise comparisons resulted in significantly higher performance than constrained optimization, yet there is negligible difference between the usability appraisals for each technique. Furthermore, we observed that the technique that participants perform better with is not necessarily the one that they consider more usable.