This repository contains the code, data, and manuscript for a study on the semantic robustness of instruction-tuned large language models (LLMs) under controlled prompt perturbations. The work evaluates how consistently models preserve meaning when prompts are reformulated through paraphrasing or adversarial modifications that maintain underlying semantic intent. We compare a frontier API-based model (Gemini-2.5-Flash) and a small open-weight model (Qwen2.5-1.5B-Instruct) using sentence-transformer embeddings to compute cosine similarity between baseline and perturbed responses across 10 prompts, yielding 40 paired evaluations (2 models × 2 perturbation types × 10 prompts). The study focuses on distributional properties of semantic similarity rather than mean performance alone, highlighting variance, dispersion, and tail-risk behavior in model responses. Results show that while Qwen2.5-1.5B-Instruct exhibits smooth degradation under increasing perturbation severity, Gemini-2.5-Flash demonstrates a highly non-uniform response profile characterized by near-perfect stability in most cases alongside rare but severe semantic failures under adversarial conditions. Statistical analysis includes bootstrap confidence intervals and Mann-Whitney U testing to compare similarity distributions across models and perturbation types. Findings indicate that average similarity alone is insufficient to characterize robustness, and that dispersion and tail behavior provide critical additional insight into model reliability. All code, prompts, and evaluation pipelines are included to support full reproducibility of this low-cost, bootstrap-based evaluation framework.
LaDawn J Hall (Mon,) studied this question.