iiinova — where your digital self keeps evolving

Mozes Jacobs

@mozesjacobs.bsky.social

PhD student at the Kempner Institute at Harvard University. Interested in Computer Vision and Theoretical Neuroscience. Advised by Demba Ba.

Posts Replies Media

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

Here are some examples of the wave dynamics used to segment Multi-MNIST images: 11/13

110

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

We also compared our model to U-Nets, which have global receptive fields via skip connections and bottlenecks. Incredibly, on Multi-MNIST, wave-based models outperformed similarly sized U-Nets, despite having fewer parameters and only local connectivity. 10/13

110

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

We see that more complex linear transformations of the hidden state timeseries are the best for extracting the global information, with a learned linear transformation performing the best (even better than the Fourier transform or the common technique of using the last RNN hidden state). 8/13

110

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

We then studied both our wave-biased model and a standard ConvLSTM (with no wave-based inductive bias). Incredibly, we found that both models learned to generate waves. The ConvLSTM’s emergent waves (shown below on a Tetrominoes image) suggest a degree of optimality for a wave-based solution. 7/13

120

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

To test this, we built a trainable RNN (the Neural Wave Machine/NWM) that generates traveling waves in its hidden states. We began by testing it on segmenting simple polygons. We find that wave-based models produce unique dynamics for each shape, resulting in distinct Fourier spectra. 6/13

120

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

We found that we could actually predict the area of the drums analytically by looking at the frequency of oscillations of each neuron (see below). This finding led us to wonder: can we actually learn (via trainable parameters) dynamics for more complex shapes? 5/13

120

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

The problem "Can One Hear the Shape of a Drum", posed by Mark Mac, is a classical example of spatial integration. Strike a drumhead, and its vibrations encode the boundary shape. We can see (with fixed RNNs that simulate drums) that different sized drumheads have different dynamics: 4/13

110

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

Spatial integration means that a neuron at one location can access signals from distant points. This could mean linking information together across an image to classify objects or linking words together in a sentence to derive meaning. 3/13

110

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

The act of vision is a coordinated activity involving millions of neurons in the visual cortex. How is information shared over these large distances? Evidence suggests traveling waves could carry this information across space, allowing neurons to “know” what’s happening far away. 2/13

130

Mozes Jacobs @mozesjacobs.bsky.social · 10/03/2025

Traveling waves of neural activity are observed all over the brain. Can they be used to augment neural networks? I am thrilled to share our new work, "Traveling Waves Integrate Spatial Information Through Time" with @andykeller.bsky.social! 1/13

23810