iiinova — where your digital self keeps evolving

Replies

Pete Shaw @ptshaw.bsky.social · 01/10/2025

The Kolmogorov complexity of an object is the length of the shortest program that prints that object. Combining Kolmogorov complexity with the MDL principle provides an elegant foundation for formalizing Occam’s razor. But how can these ideas be applied to neural networks?

110

Pete Shaw @ptshaw.bsky.social · 01/10/2025

To address this question, we define the notion of asymptotically optimal description length objectives. We establish that a minimizer of such an objective achieves optimal compression, for any dataset, up to an additive constant, in the limit as model resource bounds increase.

100

Pete Shaw @ptshaw.bsky.social · 01/10/2025

We prove that asymptotically optimal objectives exist for Transformers, building on a new demonstration of their computational universality. We also highlight potential challenges related to effectively optimizing such objectives.

100