iiinova — where your digital self keeps evolving

Reposted by Alessio Devoto

Daniel Scalena @danielsc4.it · 23/05/2025

💡 We compare prompting (zero and multi-shot + explanations) and inference-time interventions (ActAdd, REFT and SAEs). Following SpARE (@yuzhaouoe.bsky.social @alessiodevoto.bsky.social), we propose ✨ contrastive SAE steering ✨ with mutual info to personalize literary MT by tuning latent features 4/

142

Reposted by Alessio Devoto

Aryo Pradipta Gema @aryopg.bsky.social · 02/05/2025

MMLU-Redux just touched down at #NAACL2025! 🎉 Wish I could be there for our "Are We Done with MMLU?" poster today (9:00-10:30am in Hall 3, Poster Session 7), but visa drama said nope 😅 If anyone's swinging by, give our research some love! Hit me up if you check it out! 👋

01711

Reposted by Alessio Devoto

Pasquale Minervini @neuralnoise.com · 19/04/2025

My amazing collaborators will present several works at ICLR and NAACL later this month -- please catch up with them if you're attending! I tried to summarise our recent work in a blog post: neuralnoise.com/2025/march-r...

0175

Reposted by Alessio Devoto

Pasquale Minervini @neuralnoise.com · 13/03/2025

Please share it within your circles! edin.ac/3DDQK1o

0139

Reposted by Alessio Devoto

Nathan Godey @nthngdy.bsky.social · 06/03/2025

🚀 New Paper Alert! 🚀 We introduce Q-Filters, a training-free method for efficient KV Cache compression! It is compatible with FlashAttention and can compress along generation which is particularly useful for reasoning models ⚡ TLDR: we make Streaming-LLM smarter using the geometry of attention

1207

Reposted by Alessio Devoto

Lorenzo Loconte @loreloc.bsky.social · 04/03/2025

Live from the CoLoRAI workshop at AAAI (april-tools.github.io/colorai/) Nadav Cohen is now giving his talk on "What Makes Data Suitable for Deep Learning?" Tools from quantum physics are shown to be useful in building more expressive deep learning models by changing the data distribution.

1154

Reposted by Alessio Devoto

Naomi Saphra @nsaphra.bsky.social · 03/03/2025

2018: Saliency maps give plausible interpretations of random weights, triggering skepticism and catalyzing the mechinterp cultural movement, which now advocates for SAEs. 2025: SAEs give plausible interpretations of random weights, triggering skepticism and ...

Sanity Checks for Saliency Maps
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim
Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.

Sparse Autoencoders Can Interpret Randomly Initialized Transformers
Thomas Heap, Tim Lawson, Lucy Farnik, Laurence Aitchison
Sparse autoencoders (SAEs) are an increasingly popular technique for interpreting the internal representations of transformers. In this paper, we apply SAEs to 'interpret' random transformers, i.e., transformers where the parameters are sampled IID from a Gaussian rather than trained on text data. We find that random and trained transformers produce similarly interpretable SAE latents, and we confirm this finding quantitatively using an open-source auto-interpretability pipeline. Further, we find that SAE quality metrics are broadly similar for random and trained transformers. We find that these results hold across model sizes and layers. We discuss a number of number interesting questions that this work raises for the use of SAEs and auto-interpretability in the context of mechanistic interpretability.

29515

Reposted by Alessio Devoto

Kosta Derpanis @csprofkgd.bsky.social · 23/02/2025

Graphical tensor notation for interpretability www.lesswrong.com/posts/BQKKQi...

1163

Reposted by Alessio Devoto

hardmaru @hardmaru.bsky.social · 20/02/2025

Introducing The AI CUDA Engineer: An agentic AI system that automates the production of highly optimized CUDA kernels. sakana.ai/ai-cuda-engi... The AI CUDA Engineer can produce highly optimized CUDA kernels, reaching 10-100x speedup over common machine learning operations in PyTorch. Examples:

39017

Reposted by Alessio Devoto

Sebastian Raschka (rasbt) @rasbt.bsky.social · 15/02/2025

It's 2025, and I’ve finally updated my Python setup guide to use uv + venv instead of conda + pip! Here's my go-to recommendation for uv + venv in Python projects for faster installs, better dependency management: github.com/rasbt/LLMs-f... (Any additional suggestions?)

1115920

Alessio Devoto @alessiodevoto.bsky.social · 05/02/2025

Cool research on how models memorize data 📝 : The 'Manifold Memorization Hypothesis' by Brendan Ross, Hamidreza Kamkariet al. suggests memorization occurs when the model's learned manifold matches the true data manifold but with too small 'local intrinsic dimensionality'.

130

Alessio Devoto @alessiodevoto.bsky.social · 04/02/2025

Massive activations & weights in LLMs, two cool works 🤓: - The Super Weight: finds performance can be totally degraded when pruning a *single* weight - Mengxia Yu et al. - Massive Activations in LLM:finds some (crucial) activations have very high norm irrespective of context - Mingjie Sun et al.

151

Reposted by Alessio Devoto

Adina Yakup @adinayakup.bsky.social · 27/01/2025

On the last day before the Spring Festival holiday in China, DeepSeek released a NEW work on @hf.co 🤯 Janus-Pro🔥 autoregressive framework that unifies multimodal understanding and generation huggingface.co/deepseek-ai/... ✨ 1B / 7B ✨ MIT License

huggingface.co

deepseek-ai/Janus-Pro-1B · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

0103

Reposted by Alessio Devoto

Katharine Hayhoe @katharinehayhoe.com · 24/01/2025

Not only that, but much of the science community here is already stronger and larger than it was on X. On Twitter, my feed of scientists who study climate-related topics topped out at 3300. Here, we’re at 4500 already and it’s still growing. Pin here: bsky.app/profile/did:...

241581219

Reposted by Alessio Devoto

Simone Scardapane @sscardapane.bsky.social · 10/01/2025

*MoE Graph Transformers for Interpretable Particle Collision Detection* by @alessiodevoto.bsky.social @sgiagu.bsky.social et al. We propose a MoE graph transformer for particle collision analysis, with many nice interpretability insights (e.g., expert specialization). arxiv.org/abs/2501.03432

0124

Alessio Devoto @alessiodevoto.bsky.social · 22/12/2024

LLMs inner representations🔬 Llamas Work in English: LLMs default to English-based concept representations, regardless of input language @wendlerc.bsky.social et al Semantic Hub: Multimodal models create a single shared semantic space, structured by their primary language @zhaofengwu.bsky.social et a

2434

Reposted by Alessio Devoto

Jeremy Howard @howard.fm · 19/12/2024

I'll get straight to the point. We trained 2 new models. Like BERT, but modern. ModernBERT. Not some hypey GenAI thing, but a proper workhorse model, for retrieval, classification, etc. Real practical stuff. It's much faster, more accurate, longer context, and more useful. 🧵

19620147

Alessio Devoto @alessiodevoto.bsky.social · 19/12/2024

In Vision & Audio transformers, not all tokens need the same compute resources! We propose “modular learners” to control compute at token-level granularity (MHA & MLP): hard tokens get more, easy ones get less! w/ @sscardapane.bsky.social @neuralnoise.com @bartoszWojcik Soon #AAAI25 Link 👇

1193

Reposted by Alessio Devoto

Max Kleiman-Weiner @maxkw.bsky.social · 15/12/2024

Josh Tenenbaum on scaling up vs growing up and the path to human-like reasoning #NeurIPS2024

1826

Reposted by Alessio Devoto

Simone Scardapane @sscardapane.bsky.social · 11/12/2024

*Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference* with @alessiodevoto.bsky.social @neuralnoise.com Happy to share our work on distilling efficient transformers with dynamic modules' activation was accepted at #AAAI2025. 🔥 arxiv.org/abs/2312.10193

0102

Reposted by Alessio Devoto

Simone Scardapane @sscardapane.bsky.social · 06/12/2024

*Sparse Crosscoders for Cross-Layer Features and Model Diffing* by @colah.bsky.social @anthropic.com Investigates stability & dynamics of "interpretable features" with cross-layers SAEs. Can also be used to investigate differences in fine-tuned models. transformer-circuits.pub/2024/crossco...

084

Alessio Devoto @alessiodevoto.bsky.social · 06/12/2024

Very cool work! 👏🚀 Unfortunately, errors in the original dataset will propagate to all new languages 😕 We investigated the issue of existing errors in the original MMLU in arxiv.org/abs/2406.04127 @aryopg.bsky.social @neuralnoise.com

042

Reposted by Alessio Devoto

Aryo Pradipta Gema @aryopg.bsky.social · 06/12/2024

Super Cool work from Cohere for AI! 🎉 However, this highlights a concern raised by our MMLU-Redux team (arxiv.org/abs/2406.04127): **error propagation to many languages**. Issues in MMLU (e.g., "rapid intervention to solve ebola") seem to persist in many languages. Let's solve the root cause first?

193

Alessio Devoto @alessiodevoto.bsky.social · 04/12/2024

Cool take on straight-through-estimator (to backpropagate through discrete ops): during back-propagation, keep gradient's relative angle, not absolute direction. The authors call this the "rotation trick". From "Restructuring Vector Quantization With The Rotation Trick" @ChristopherFifty et al.

010

Reposted by Alessio Devoto

Emile van Krieken @emilevankrieken.com · 02/12/2024

Wish I knew about this much earlier: In VS Code (or Cursor 😉) you can install a 'data wrangler' extension to inspect the values of your PyTorch Tensors in a nice UI with summary statistics. Just right-click on a tensor in your variables view!

4254

Reposted by Alessio Devoto

Vicki @vickiboykis.com · 29/11/2024

Why don’t they just make the whole app out of L1 cache samwho.dev/numbers/

samwho.dev

Latency Numbers Every Programmer Should Know

An interactive exploration of how long things take.

810814

Reposted by Alessio Devoto

Thomas Wolf @thomwolf.bsky.social · 24/11/2024

It's Sunday morning so taking a minute for a nerdy thread (on math, tokenizers and LLMs) of the work of our intern Garreth By adding a few lines of code to the base Llama 3 tokenizer, he got a free boost in arithmetic performance 😮 [thread]

527134

Reposted by Alessio Devoto

Christian S. Perone @cperone.bsky.social · 19/11/2024

Hi, just sharing an updated version of the PyTorch 2 Internals slides: drive.google.com/file/d/18YZV.... Content: basics, jit, dynamo, Inductor, export path and executorch. This is focused on internals so you will need a bit of C/C++. I show how you can export and run a model on a Pixel Watch too.

28617

Reposted by Alessio Devoto

Simone Scardapane @sscardapane.bsky.social · 22/11/2024

*Building neural networks in Equinox* Third lab for the course, after JAX and Keras we see Equinox (from @patrickkidger.bsky.social) - including callable pytrees, stateful modules, and filtering. Stay tuned for part b with a text classification model. 😎 colab.research.google.com/drive/19NHkb...

0192

Reposted by Alessio Devoto

merve @merve.bsky.social · 22/11/2024

Apple released AIMv2 🍏 a family of state-of-the-art open-set vision encoders > like CLIP, but add a decoder and train on autoregression 🤯 > 19 open models come in 300M, 600M, 1.2B, 2.7B with resolutions of 224, 336, 448 > Loadable and usable with 🤗 transformers huggingface.co/collections/...

2465

Alessio Devoto @alessiodevoto.bsky.social · 20/11/2024

New and very cool library!👏 Our L2 Norm-based KV Cache compression is already implemented - ready to use! 🚀 Check out the method details in our EMNLP '24 paper: arxiv.org/abs/2406.11430

1132

Reposted by Alessio Devoto

Pasquale Minervini @neuralnoise.com · 20/11/2024

KVPress supports our state-of-the-art KV-Cache compression method from "A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression" (arxiv.org/abs/2406.11430, EMNLP'24 Oral) 🚀🚀🚀 You can read more on our EMNLP papers (and other projs) in this blog post: www.neuralnoise.com/2024/nov-res...

arxiv.org

A Simple and Effective $L_2$ Norm-Based Strategy for KV Cache Compression

The deployment of large language models (LLMs) is often hindered by the extensive memory requirements of the Key-Value (KV) cache, especially as context lengths increase. Existing approaches to reduce...

1183

Alessio Devoto @alessiodevoto.bsky.social · 18/11/2024

*Tokenformer: Rethinking Transformer Scaling With Tokenized Model Parameters* arxiv.org/pdf/2410.23168 Very cool transformer-inspired architecture where linear layers are replaced with token-parameter attention (Pattention). This allows for efficient scaling by adding new parameters to the model.

040

Reposted by Alessio Devoto

Aryo Pradipta Gema @aryopg.bsky.social · 18/11/2024

I’ll be travelling to London from Wednesday to Friday for an upcoming event and would be very happy to meet up! 🚀 I'd love to chat about my recent works (DeCoRe, MMLU-Redux, etc.). DM me if you’re around! 👋 DeCoRe: arxiv.org/abs/2410.18860 MMLU-Redux: arxiv.org/abs/2406.04127

0117

Reposted by Alessio Devoto

Eytan Adar @eytan.adar.prof · 13/11/2024

FWIW, blueark.app worked well. $5 to copy my tweet history over. Not sure *my* tweets were worth that... But that's not their fault.

2102

Reposted by Alessio Devoto

Simone Scardapane @sscardapane.bsky.social · 14/11/2024

Just landed here! 🔥 A nice news to start: *Alice in a differentiable wonderland* has gone over 1000 copies sold on Amazon and I am super happy about the feedback! If you happen to buy a copy feel free to drop a review and/or send me suggestions on the material: www.sscardapane.it/alice-book/

0272