Investigating the Impact of Online Homework Reminders Using Randomized A/B Comparisons

Procrastination by students may lead to adverse outcomes such as a focus on completion rather than learning or even a failure to complete learning tasks. One common method for motivating students and reducing procrastination is to send reminders with hints and study strategies, but it's not clear if these messages are effective or when is the best time to send them. Randomized A/B comparisons could be used to try different reminders or alternative ideas about how best to get students to start work earlier and, crucially, to measure the impact of these interventions on behaviour. This paper describes an A/B comparison of reminder emails set in a large CS1 course at a research-focused North American university. We found evidence that the email interventions caused a higher proportion of students to attempt the online homework but did not see evidence that these particular emails got students to start early, irrespective of changes to the timing of the reminder. More broadly, these findings illustrate how to use A/B comparisons in educational settings to test ideas about how to help students, and demonstrate the value of using randomized A/B comparisons, even when evaluating actions that seem obviously beneficial, such as reminder emails.