Analyzing Finetuning Representation Shift for Multimodal LLMs Steering

Key Points

• Representation shift affects multimodal language model performance negatively, hindering task effectiveness.
• Key evidence shows that finetuning results in significant representation shifts in model outputs.
• Analysis utilizes a systematic approach to assess the impacts of finetuning on model representation.
• Implications may enhance future performance of multimodal language models, warranting further exploration.