We found that LMs, large or small, text-only or multimodal, are susceptible to a shared problem -- 𝗼𝘃𝗲𝗿𝗰𝗼𝗻𝗳𝗶𝗱𝗲𝗻𝗰𝗲 𝗮𝗻𝗱 𝗼𝗯𝗹𝗶𝘃𝗶𝗼𝗻 𝘁𝗼 𝗱𝗲𝘁𝗮𝗶𝗹𝘀.
When presented with a scenario that is uncommon with overwhelming pretraining data that encourages a shortcut, they take the shortcut confidently all the time 1/
