Francesco Ortu @francescortu.bsky.social · 10/12/2024Additionally, blocking communication from this token significantly disrupts performance on standard benchmarks, while blocking image-text communication does not 110 1
Francesco Ortu @francescortu.bsky.social · 10/12/2024✅ This shows that, starting from the mid-layers, a single token effectively summarizes all 1024 image tokens! ❌ This does not occur in models fine-tuned for visual understanding (such as Pixtral). 110 1
Francesco Ortu @francescortu.bsky.social · 10/12/2024It was super fun to take our first step in interpreting multimodal LLMs, working closely with the brilliant @alexpietroserra.bsky.social and @EmanuelePanizon 100 1
