Congratulations to Marco and all the other co-authors on this groundbreaking 154-page survey. If you're at all interested in tokenization, give it a read.
Craig Schmidt
@craigschmidt.com
600 followers 2.4K following 67 posts
Interested in ML, AI, and NLP. Particularly interested in tokenization. Live in the Boston area and work in R&D at Kensho Technologies.
![Code: from minicons import scorer
lm = scorer.IncrementalLMScorer("gpt2-xl", "cuda:0")
stimuli = ["I was a matron in France", "I was a mat in France"]
# old way, no correction
# P.S. gpt2 does not automatically add a bos token at the beginning...
lm.token_score(stimuli, bos_token=True, surprisal=True, base_two=True, bow_correction=False)
'''Rounded Output
[[('<|endoftext|>', 0.0),
('I', 5.85),
('was', 4.28),
('a', 4.67),
('mat', 16.34),
('ron', 1.74),
('in', 2.12),
('France', 11.43)],
[('<|endoftext|>', 0.0),
('I', 5.85),
('was', 4.28),
('a', 4.67),
('mat', 16.34),
('in', 10.78),
('France', 10.71)]]
'''
# the new way! notice the surprisal of "mat" in both cases
lm.token_score(stimuli, bos_token=True, surprisal=True, base_two=True, bow_correction=True)
'''Rounded Output
[[('<|endoftext|>', 0.0),
('I', 6.30),
('was', 3.84),
('a', 4.68),
('mat', 16.34),
('ron', 2.11),
('in', 1.75),
('France', 11.42)],
[('<|endoftext|>', 0.0),
('I', 6.30),
('was', 3.84),
('a', 4.68),
('mat', 21.34),
('in', 5.80),
('France', 10.69)]]
'''](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:njnapclhkbrhe3hsq44q2e4w/bafkreif2efemwaqk4tlryxdbmo7b6si2ewhblzefmoxtmzys7bgjmgq5wm)
