iiinova — where your digital self keeps evolving

Reposted by Pierre Orhan

Lucas Benjamin @lucaswbenjamin.bsky.social · 24/08/2026

Our new paper is now out in @pnas.org! How does the (baby) brain extract regularities from sequences from what-follows-what up to large-scale network structure? What if it all came from a single mechanism? www.pnas.org/doi/abs/10.1...

15216

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

Do not hesitate to contact Pablo Diego or me if you have any questions about the work! This work was done while I was at @lsp-ens.bsky.social and finished at @institutducerveau.bsky.social ! Stay tuned for future works applying this approach to neurological recordings!

010

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

As a conclusion, artificial neural networks mirror some developmental steps and clarify sufficient conditions of their emergence. Yet models learn way too slowly: they require two orders of magnitude more words (tokens) than what children use to discover these linguistic structures!

130

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

The paper has some additional findings: - Simpler linguistic structures (phonemes) are learned first, but semantic classes are not learned in a particular order. - We invented a Topological Probe (enforcing topology rather than distance) to show the robustness of the approach.

110

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

Although the instantiation of the WordNet graph was strong in large models, it was small but significant in audio and small text models. We confirmed it with measurements based on semantic classes and visualized it through a nice coloring of the WordNet Graph, here for a 1B Pythia (text) model.

100

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

We found instantiation of these structures in an audio model: Wav2vec2 pretrained on English, but not when pretrained on other sounds! By exploring these instantiations during pretraining, we evidence the emergence of each linguistic structure: phonemes emerge before lexical semantics and syntax.

110

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

To detect the instantiation of linguistic structures, we adapt the structural probe of Hewitt and Manning to semantics and phonetics. Semantic is given by the WordNet graph (hypernym relationships "is contained in"). Example: mammal is a hypernym of placental, itself a hypernym of carnivor, equine…

100

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

Findings: - Linguistic structures are instantiated by a speech ANN. - Phonemes are instantiated before semantics and syntax, mirroring the acquisition stages of children. Methodological development: - Probing of speech models through checkpoints - Novel topological probe of WordNet

100

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

Novel study: Emergence of Phonemic, Syntactic, and Semantic Representations in Neural Networks. Joint work with P. Diego supervision: Y. Lakretz, E. Chemla, Y. Boubenec, @jeanremiking.bsky.social We explore the emergence of 3 linguistic structures in Neural Networks. Arxiv: arxiv.org/abs/2601.18617

1103

Reposted by Pierre Orhan

Joséphine Raugel @josephine-raugel.bsky.social · 26/02/2025

⚡️Check out our workshop tomorrow at @lpiparis.bsky.social, great speakers (@gabrielpeyre.bsky.social, @sdascoli.bsky.social, @samillingworth.com & many more) will cover Theory and Applications of Generative AI + Connexions with neuroscience 🧠 And there's food 🍰 ➡️ genai-conference-website.vercel.app

073

Reposted by Pierre Orhan

LSP-ENS @lsp-ens.bsky.social · 19/02/2025

Congratulations to Pierre Orhan (@pierreorhan.bsky.social) who successfully defended his PhD on "Learning dynamics in biological and artificial neural networks", conducted at @normalesup.bsky.social

053