Evaluating Honeyfile Realism and Enticement Metrics

Deceptive files, often called honeyfiles, have become an established tool in cyber security. Advances in machine learning (ML) models for content generation now allow the synthesis of deceptive material automatically and at scale. Metrics to quantify honeyfile attributes are thus essential to creating and evaluating effective deceptions. The two critical aspects of honeyfiles for which metrics are useful are enticement and realism . Enticement is the ability to attract the attention of intruders or users with malicious intent. Realism measures the similarity of deceptive artefacts to the objects they mimic. In the honeyfile literature, metrics for these attributes have been proposed: the Common Token Count (CTC) [1], and Topic Semantic Matching (TSM) [2] scores for enticement, and coherence and cohesion [3] for realism. In this study, we compare these metrics to the perceptions of human users exposed to text samples in a simulated data breach scenario on a crowd-sourcing platform. We recruited participants to judge the realism and enticement of honeyfile text generated using several techniques. The main findings are: (i) for the enticement metrics, TSM is aligned with the perceived enticement (p-value<0.001), while for the CTC score, we find inconsistent and inconclusive results, and (ii) for the realism metrics, cohesion and coherence, do not consistently align with perceived realism.