Reverse-engineering expert similarity judgments: evidence from chess tactics

Experts in complex domains make similarity judgments they can demonstrate but cannot articulate. Chess tactics offer a natural laboratory for studying such judgments because they separate what a problem looks like (the board position) from what it requires (the solution). However, no study has used supervised learning on expert-labeled data to decompose which features drive an expert’s similarity criterion or how they are organized. We collected 2866 binary similarity annotations from a competitive chess player on tactical puzzle pairs and trained interpretable classifiers to recover the implicit decision structure. An ablation study reveals a near-total dissociation: dynamic features alone (solution properties) achieve an AUC of 0.901, versus 0.554 for static features—board appearance plays a negligible role. The expert’s implicit decision is well described by a dominant binary criterion with graded refinement: agreement on whether both solutions end in checkmate is near-necessary for a “similar” judgment, after which graded comparison of the solution sequences decides the rest. Inter-rater validation with two additional titled players confirms these criteria are shared, not idiosyncratic. Difference features alone nearly match the full model, supporting a feature-contrast representation of similarity (Tversky, 1977). At the methodological level, the combination of information retrieval, expert annotation, supervised classification, and qualitative re-evaluation constitutes a reusable pipeline for reverse-engineering implicit expert criteria. The qualitative step proved especially informative: the expert reversed 70% of disputed labels, every reversal converging with the model. For chess training, similarity-search systems should prioritize solution-level features over board appearance.