iiinova — where your digital self keeps evolving

Pierre Orhan

@pierreorhan.bsky.social

37 followers 42 following 8 posts

Neuroscience Postdoc: theories of neural codes and their emergence Paris Brain Institute

Posts Replies Media

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

As a conclusion, artificial neural networks mirror some developmental steps and clarify sufficient conditions of their emergence. Yet models learn way too slowly: they require two orders of magnitude more words (tokens) than what children use to discover these linguistic structures!

130

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

Although the instantiation of the WordNet graph was strong in large models, it was small but significant in audio and small text models. We confirmed it with measurements based on semantic classes and visualized it through a nice coloring of the WordNet Graph, here for a 1B Pythia (text) model.

100

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

We found instantiation of these structures in an audio model: Wav2vec2 pretrained on English, but not when pretrained on other sounds! By exploring these instantiations during pretraining, we evidence the emergence of each linguistic structure: phonemes emerge before lexical semantics and syntax.

110

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

To detect the instantiation of linguistic structures, we adapt the structural probe of Hewitt and Manning to semantics and phonetics. Semantic is given by the WordNet graph (hypernym relationships "is contained in"). Example: mammal is a hypernym of placental, itself a hypernym of carnivor, equine…

100

Pierre Orhan @pierreorhan.bsky.social · 06/02/2026

Novel study: Emergence of Phonemic, Syntactic, and Semantic Representations in Neural Networks. Joint work with P. Diego supervision: Y. Lakretz, E. Chemla, Y. Boubenec, @jeanremiking.bsky.social We explore the emergence of 3 linguistic structures in Neural Networks. Arxiv: arxiv.org/abs/2601.18617

1103