An important issue to consider is the training time, as it can have a considerable influence on the set of stories generated, due to factors such as uncertainty, diversity, and narrative coherence. This paper presents a systematic analysis of the dynamics of predictive entropy at different times and random seeds, studying the interaction of entropy with lexical diversity, repetition, semantic consistency, and entity continuity in probabilistic language generation models. A comparative evaluation of recurrent and attention-based architectures is performed using linguistic metrics. Predictive entropy was reduced by 32.4% (LSTM) and 28.7% (Transformer). LexDiv obtained 0.71 ± 0.03 and Self-BLEU obtained 0.42 ± 0.02, suggesting greater confidence in the model. However, it should be noted that a greater reduction in entropy may be associated with lower lexical diversity and higher Self-BLEU scores. This indicates a trade-off between confidence and expressiveness in probabilistic language models. The entropy term encourages smoother probability distributions and reduces premature mode collapse during Adam optimization. Ltotal=LCE−λH(p(y|x) aims to improve stability, reduce random initialization, and enable the generation of adaptable narratives, which may be relevant for neurodiversity-oriented narratives.
Alanís et al. (Tue,) studied this question.