Exploring the Utility of Embedding Similarity for Contract Tasks

With the increasing use of text embeddings motivated by the adoption of Retrieval Augmented Generation (RAG) in applied domains, this work investigates whether the semantic aspects of text embeddings correspond to the colloquial understanding of semantic similarity in a legal domain. Using clauses from legal agreements, we find that embeddings and associated similarity measurements (e.g., cosine, L2) do not accurately reflect a legal understanding of ''semantically similar.'' More specifically, legal clauses are more similar to a minimally changed, negated version than those with identical legal meaning but worded differently across embedding sources and similarity measures. We demonstrate that discriminative classification can be an effective stop-gap solution with these two types of variants using either a zero-shot generative model prompt or a multi-layer perceptron trained on the embeddings. These results indicate that care should be taken when applying off-the-shelf semantic similarity tools in specialized domains and provides a basis from which further work can be conducted to determine cost effective methods for measuring nuanced notions of similarity.