iiinova — where your digital self keeps evolving

Ameya P.

@bayesiankitten.bsky.social

601 followers 129 following 15 posts

Postdoctoral Researcher @ Bethgelab, University of Tübingen Benchmarking | LLM Agents | Data-Centric ML | Continual Learning | Unlearning drimpossible.github.io

Posts Replies Media

Ameya P. @bayesiankitten.bsky.social · 13/12/2024

Can better representation learning help? No! RanDumb recovers 70-90% of the joint performance. Forgetting isn't the main issue—the benchmarks are too toy! Key Point: Current OCL benchmarks are too constrained for any effective learning of online continual representations!

010

Ameya P. @bayesiankitten.bsky.social · 13/12/2024

Across a wide range of online continual learning benchmarks-- RanDumb consistently surpasses prior methods (even latest contrastive & meta strategies), often by surprisingly large margins!

110

Ameya P. @bayesiankitten.bsky.social · 13/12/2024

Continual Learning assumes deep representations learned outperform old school kernel classifiers (as in supervised DL). But this isn't validated!! Why might it not work? Updates are limited and networks may not converge. We find: OCL representations are severely undertrained!

100

Ameya P. @bayesiankitten.bsky.social · 13/12/2024

How RanDumb works: Fix a random embedder to transform raw pixels. Train a linear classifier on top—single pass, one sample at a time, no stored exemplars. Order-invariant, worst-case ready🚀 Looks familiar? This is streaming (approx.) Kernel LDA!!

100

Ameya P. @bayesiankitten.bsky.social · 13/12/2024

New Work: RanDumb!🚀 Poster @NeurIPS, East Hall #1910- come say hi👋 Core claim: Random representations Outperform Online Continual Learning Methods! How: We replace the deep network by a *random projection* and linear clf, yet outperform all OCL methods by huge margins [1/n]

110