Layered evaluation of adaptive learning systems

Abstract: This paper suggests an alternative to the traditional ‘as a whole’ approach of evaluating adaptive learning systems (ALS), and adaptive systems, in general. We argue that the commonly recognised models of adaptive systems can be used as a basis for a layered evaluation that offers certain benefits to the developers of ALS. Therefore, we propose the layered evaluation framework, where the success of adaptation is addressed at two distinct layers: • user modelling • adaptation decision making. We outline how layered evaluation can improve the current evaluation practice of ALS. To build a stronger case for a layered evaluation we re-visit the evaluation of the InterBook where the layered approach can provide a difference and provide an example of its use in KOD learning system.