iiinova — where your digital self keeps evolving

Reposted by Felix Petersen

Alfredo Canziani @alfcnz.bsky.social · 13/12/2024

Convolutional Differentiable Logic Gate Networks @FHKPetersen

3453

Reposted by Felix Petersen

Alfredo Canziani @alfcnz.bsky.social · 12/12/2024

Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms @FHKPetersen

1103

Felix Petersen @petersen.ai · 12/12/2024

Join us at our poster session today, 11am-2pm, at East Exhibit Hall A-C *#1502*.

010

Reposted by Felix Petersen

Hendrik Strobelt @henstr.bsky.social · 12/12/2024

Most innovative paper at #NeurIPS imho. Can we create a network that becomes the physical chip instead of running on a chip? Inference speedups and energy preservation are through the roof ! Oral on Friday at 10am PT neurips.cc/virtual/2024...

neurips.cc

NeurIPS Poster Convolutional Differentiable Logic Gate NetworksNeurIPS 2024

021

Felix Petersen @petersen.ai · 10/12/2024

Join us on Wednesday, 11am-2pm for our poster session on Newton Losses in *West Ballroom A-D #6207*. neurips.cc/virtual/2024...

neurips.cc

NeurIPS Poster Newton Losses: Using Curvature Information for Learning with Differentiable AlgorithmsNeurIPS 2024

000

Felix Petersen @petersen.ai · 04/12/2024

Learn more in our paper (arxiv.org/abs/2410.23970) and check out our paper video on YouTube: youtu.be/ZjTAjjxbkRY

youtu.be

Computer Vision Models with LLM Training Dynamics (TrAct)

YouTube video by Felix Petersen

020

Felix Petersen @petersen.ai · 04/12/2024

...and it speeds up overall training by factors ranging from 1.25x (for large ViT pre-training) to 4x (for ConvNets). We benchmark TrAct on a suite of 50 experimental settings.

110

Felix Petersen @petersen.ai · 04/12/2024

Our implementation is efficient, only modifies the gradient in the backward, and is compatible with various optimizers. To use *TrAct*, just wrap your first layer in a "TrAct" module...

110

Felix Petersen @petersen.ai · 04/12/2024

Thus, we can effectively train the first-layer activations of a Vision model, with updates similar to those in the LLM Embedding layer.

110

Felix Petersen @petersen.ai · 04/12/2024

We close this gap by proposing TrAct: we conceptually *Tr*ain *Act*ivations. While we can't train activations directly bc only weights are trainable, we formulate an optimization problem to find the optimal weights to match a GD step on the activations, and in closed-form modify the gradients resp.

110

Felix Petersen @petersen.ai · 04/12/2024

This means that learning of the vision model, at the first layer, is much slower than in LLMs, and that learning is actually faster on higher contrast regions of the image than in low contrast regions due to a proportionality between gradients of weights and input pixel values.

110

Felix Petersen @petersen.ai · 04/12/2024

The big difference between LLMs and Vision models lies in the first layer: * in LLMs we update Embeddings (/activations) directly * but in Vision models we update the *weights* of the first layer, which causes indirect updates to the Activations (/embeddings)

110

Felix Petersen @petersen.ai · 04/12/2024

Have you ever wondered how training dynamics differ between LLMs 🖋️ and Vision 👁️ models? We explore this and close the gap between VMs and LLMs in our #NeurIPS2024 paper "TrAct: Making First-layer Pre-Activations Trainable". Paper link 📜: arxiv.org/abs/2410.23970 Video link 🎥: youtu.be/ZjTAjjxbkRY 🧵

192

Felix Petersen @petersen.ai · 28/11/2024

A big thanks to my co-authors Christian Borgelt, @tobiassutter.bsky.social @hildekuehne.bsky.social Oliver Deussen and Stefano Ermon. Also a shout-out to the authors of the methods we build on: @qberthet.bsky.social @mblondel.bsky.social @marcocuturi.bsky.social @bachfrancis.bsky.social ky.social

151

Felix Petersen @petersen.ai · 28/11/2024

Newton losses is easy to implement, and it's empirical Fisher extension can be added to existing pipelines with a single call of `InjectFisher` between the model and the loss.

130

Felix Petersen @petersen.ai · 28/11/2024

In Newton Losses, we merge SGD training of NNs with a Newton step on the loss. This is crucial for algorithmic losses like ranking and graph losses, esp. w/ vanishing+exploding grads. Intuition: if the loss is harder to optim. than the NN, we should use a stronger optimization method for the loss.

130

Felix Petersen @petersen.ai · 17/11/2024

Excited to share our #NeurIPS 2024 Oral, Convolutional Differentiable Logic Gate Networks, leading to a range of inference efficiency records, including inference in only 4 nanoseconds 🏎️. We reduce model sizes by factors of 29x-61x over the SOTA. Paper: arxiv.org/abs/2411.04732

29918