iiinova — where your digital self keeps evolving

diegodoimo.bsky.social

@diegodoimo.bsky.social

diegodoimo.bsky.social @diegodoimo.bsky.social · 10/12/2024

This work is a collaboration with a team of talented researchers at the AreaSciencePark of Trieste, Italy. Special thanks to @alexpietroserra.bsky.social, Alessio Ansuini and @albecazzaniga.bsky.social ! If you are @neuripsconf.bsky.social don't miss our poster tomorrow, Dec 11, at 11am!! 🧵6/6

020

diegodoimo.bsky.social @diegodoimo.bsky.social · 10/12/2024

⚒️ We applied an advanced density-based clustering algorithm, showing its potential as an interpretability tool and in guiding novel strategies for the effective finetuning of LLMs. 🧵5/6

120

diegodoimo.bsky.social @diegodoimo.bsky.social · 10/12/2024

In fine-tuning, answer-focused modes rapidly emerge midway through the network, just after the intrinsic dimension peak. Early layers remain largely unchanged. 🧵4/6

120

diegodoimo.bsky.social @diegodoimo.bsky.social · 10/12/2024

In few-shot learning, the prompt topic defines the modes of data distribution early in the network, and density modes are hierarchically organized based on the similarity of the subjects. 🧵3/6

120

diegodoimo.bsky.social @diegodoimo.bsky.social · 10/12/2024

🎯 Key results: few-shot learning and fine-tuning show two distinct processing phases inside LLMs. These phases are separated by a peak of the data intrinsic dimension and a sharp decrease in the separation of the probability modes. Paper: arxiv.org/abs/2409.03662 🧵2/6

120

diegodoimo.bsky.social @diegodoimo.bsky.social · 10/12/2024

Just landed in Vancouver to present @neuripsconf.bsky.social the results of our new work! Few-shot learning and fine-tuning change the layers inside LLMs in a dramatically different way, even when they perform equally well on multiple-choice question-answering tasks. 🧵1/6

1100