Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models F i n d i n g s - E M N L P