Comparative Study on Synthetic and Natural Error Analysis with BART & MarianMT

Text is essential for communication, information sharing, knowledge acquisition, and analysis. It shapes public opinion, supports education, and drives online content, making it crucial in various domains. While there are various language models utilized for text analysis and text correction, there is little to no survey conducted on these model's behavior and limitations. This work deals with studying BART and MarianMT language models behavior to an input dataset consisting of two types of errors, Synthetic and Natural. Synthetic errors are efficient to create and test, whereas Natural errors are more common and close to real world occurring errors. The models were trained and tested with the generated data, the results highlighted that BART exhibited consistent outputs towards both Synthetic and Natural errors and hence revealing a break-even point at the vicinity of 26% Synthetic error introduction. Conversely, the performance of MarianMT was comparatively diminished for Synthetic errors in contrast to Natural errors. These findings provide valuable insights into the behavior and capabilities of the models.