Volume 68, Issue 4 p. 895-910
RESEARCH ARTICLE

Evaluation of context-aware recommendation systems for information re-finding

Maya Sappelli

Maya Sappelli

TNO, Anna van Buerenplein 1, 2595, DA, Den Haag, The Netherlands

Institute for Computing and Information Sciences, Radboud University, Heyendaalseweg 135, 6525 AJ, Nijmegen, The Netherlands

Search for more papers by this author
Suzan Verberne

Suzan Verberne

Institute for Computing and Information Sciences, Radboud University, Heyendaalseweg 135, 6525 AJ, Nijmegen, The Netherlands

Search for more papers by this author
Wessel Kraaij

Wessel Kraaij

TNO, Anna van Buerenplein 1, 2595, DA, Den Haag, The Netherlands

Institute for Computing and Information Sciences, Radboud University, Heyendaalseweg 135, 6525 AJ, Nijmegen, The Netherlands

Search for more papers by this author
First published: 03 August 2016
Citations: 7

Abstract

In this article we evaluate context-aware recommendation systems for information re-finding by knowledge workers. We identify 4 criteria that are relevant for evaluating the quality of knowledge worker support: context relevance, document relevance, prediction of user action, and diversity of the suggestions. We compare 3 different context-aware recommendation methods for information re-finding in a writing support task. The first method uses contextual prefiltering and content-based recommendation (CBR), the second uses the just-in-time information retrieval paradigm (JITIR), and the third is a novel network-based recommendation system where context is part of the recommendation model (CIA). We found that each method has its own strengths: CBR is strong at context relevance, JITIR captures document relevance well, and CIA achieves the best result at predicting user action. Weaknesses include that CBR depends on a manual source to determine the context and in JITIR the context query can fail when the textual content is not sufficient. We conclude that to truly support a knowledge worker, all 4 evaluation criteria are important. In light of that conclusion, we argue that the network-based approach the CIA offers has the highest robustness and flexibility for context-aware information recommendation.