Token-Based Prompt Manipulation for Automated Large Language Model Evaluation

Evaluating the capabilities of text generation models requires approaches that can capture both the syntactic and semantic complexities inherent in language. Traditional evaluation methods often rely on human intervention or static benchmarks, which fail to provide the necessary scalability or depth needed for rigorous assessment. The novel concept of token-based prompt manipulation offers an automated, scalable alternative that systematically probes model behavior through controlled token-level variations. This method enhances the understanding of model sensitivity by targeting critical tokens within prompts, testing the robustness of language models without human bias. Experiments conducted on the Mistral LLM demonstrated significant impacts on model performance through token substitutions, removals, and syntactic reordering, particularly in relation to key syntactic structures. The results highlight the importance of token discrimination for evaluating both the fluency and coherence of model outputs across a wide range of linguistic tasks. Future research could extend the framework to multilingual models and more complex downstream tasks, offering deeper insights into how token manipulations affect performance across different linguistic contexts.