Hm. Very Bluesky, ignoring that others might also understand, but’re trying to make a different or orthogonal point.
I think the Phase 3 issue is confusing folks (& eliciting dunks like yours) when it isn’t what Tao meant. He was conducting thought experiment about AI gaming the trial process
![Sky posting:
look, i know that "sky [] fumbles everyone" is one of the site's favorite recurring bits, and im gonna be honest that i can look back and laugh at a lot of them
but i'm also gonna say that the last two weeks have been really fucking hard and having to come home and do them, alone, has sucked.](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:p37yvgssptkspdoanixz2fhh/bafkreihs5egfewpwfgoehajyitduamma36hqphazpliy4vk5knibnasdoa)
![series of posts:
Jonathan Weisberg @jweisber.bsky.social
It has tokens for longer chunks too, but it needs to have a token for each character as well.
Otherwise it wouldn't be able to parse arbitrary text
****
smith @protanope.bsky.social
I don't think that's correct, at least for most of the major foundational model LLMs these days.
There is a process for chunking-up/tokenizing that is dependent on context, language, surrounding words, etc, in order to avoid having to have a token for every character.
[link to IBM website describing tokenizatuon]
www.ibm.com/docs/en/wats…
Tokens and tokenization
A token is a collection of characters that has semantic meaning for a model. Tokenization is the process of converting the words in your prompt into tokens.
******
Jonathan Weisberg @jweisber.bsky.so.
Try putting the alphabet with each letter on its own line into tiktokenizer and see what happens
[link to TikTokenizer]](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:p37yvgssptkspdoanixz2fhh/bafkreib3h2i56r66hhdgjcwsexfs7qs6hvm6nht2v6pemdaw7n52vyiui4)
![smith @protanope.bsky.social
that is generally going to be around 51-52 because the LLM, from context, is determining that it needs to scope each next token for each next one-letter input & newline entry that is being entered. in that scenario, a well-trained LLM determines that.
compare "this code looks a bit fragiguous"
[screenshot of Tiktokenizer showing that, in o200k_base, the phrase
“this code looks a bit fragiguous”
has a
Token count of
7]](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:p37yvgssptkspdoanixz2fhh/bafkreicud6wpnsknebywb27rfmoon2kvxpnpe75lhiqpoa2co7ysw5nxuq)
