Reposted by my little fauxneme
hot take: software people are going to regret not unionizing when we could
my little fauxneme
@fauxnemic.com
406 followers 1.4K following 963 posts
industry and occasionally academic linguist (pencils & computers, generative & usage, mostly p-side), doing neural shit for literal decades, leftish, oldish, grumpy, don't do cop shit, will occasionally post stuff that makes you rethink following me
![once again, a snippet of Cosma's notebook on attention, transformers, and LLMs, this time reading:
"Now I bring all this up because, remember, once we have a source-coding scheme, we can "invert" it to get conditional probabilities; we could even sample from it to get a generator. (We'd need a little footwork to deal with some technicalities, but not a heck of a lot.) So something I'd really love to see done, by someone with the resources, is the following experiment:
Code up an implementation of Lempel-Ziv without the limitations built in to (e.g.) gzip; give it as much internal memory to build its dictionary as a large language model gets to store its parameter matrix. Call this "LLZ", for "large Lempel-Ziv".
Feed LLZ the same corpus of texts used to fit your favorite large language model. Let it build its dictionary from that. (This needs one pass through the corpus...)
Build the generator from the trained LLZ.
Swap in this generator for the neural network in a chatbot or similar. Call this horrible thing GLLZ.
In terms of perplexity (=
, basically), GLLZ will be comparable to the neural network, because Lempel-Ziv does, in fact, do universal source coding. (On the other hand, the neural network architecture may have been evolved over the last decade-plus to converge quickly on text-as-found-on-the-Web, without worrying about how well it does on arbitrary regular languages. So partly this is question of whether even Web-scale data has reached the asymptopia of Lempel-Ziv.) But it would be more interesting to compare the performance of GLLZ in other regards to that of more conventional language models.
[... snip to fit alt-text]
Update, 7 December 2024: Delétang et al. 2023 is the closest I have seen to this proposal, but (if I am reading correctly) it also uses off-the-shelf gzip, rather than an implementation of Lempel-Ziv which has the same capacity as the LLM they use for comparison."](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:y4dfkkhqyk7n7pbl3uzlzh2v/bafkreihzci43z2gf2qa4szobj5tshidida3swzfbf74weoco5ctg4j6zee)
