Sign in

Isabelle Lee

@wordscompute.bsky.social
2.4K followers 534 following 56 posts

ml/nlp phding @ usc, currently visiting harvard; training & interpretability & reasoning iglee.me

PostsRepliesMedia
Isabelle Lee @wordscompute.bsky.social · 14/06/2026
Benchmarks can be superficial, but model explanations and evaluations are fundamentally intertwined. What if we used interpretability as principled, scientific evaluation? If it met scientific standards? arxiv.org/abs/2605.05508 coming to EvalEval at ACL as oral 🧵 1/6
The text lists authors associated with the topic of rigorous interpretation as evaluation, affiliated with various universities.
1141
Isabelle Lee @wordscompute.bsky.social · 11/02/2026
a harder task- last step prediction: ¬(¬Sunny(x) ∧ Breezy(x)) ↔ [MASK] or last two step prediction. Most LLMs only achieve <50% accuracy on both tasks. (n.b. since FOL is verifiable, we define correct as any generation that's equivalent to expression.) 7/9
Bar graph displaying the accuracy percentages of various models on two-step prediction tasks, with distinct colors for each step.Bar chart displaying accuracy percentages for various models across three complexity thresholds: 10-19, 20-29, and 30+.
110
Isabelle Lee @wordscompute.bsky.social · 11/02/2026
e.g. masked prediction. we mask an operator randomly and have LLMs guess: ¬(¬Sunny(x) ∧ Breezy(x)) ↔ (Sunny(x) [MASK] Breezy(x)). LLMs are correct ~45.7% on average: 6/9
Table displaying model accuracies for components, operators, and predicates prediction tasks, with various metrics for performance evaluation.
100
Isabelle Lee @wordscompute.bsky.social · 11/02/2026
So how do we strike a balance? We propose using First-Order Logic (FOL) as a middle ground. We 1. programmatically, randomly generate a bunch of FOL expressions 2. progressively simplify them, verifying their equivalence 3. chain them together 4. NL instantiate them w/ LLMs 4/9
Flowchart illustrating the process of using First-Order Logic, integrating human rules, symbolic generation, and LLM instantiation for reasoning examples.
100
Isabelle Lee @wordscompute.bsky.social · 11/02/2026
New dataset 🗂️ coming to #eacl What is (correct) reasoning in LLMs? How do you rigorously define/measure process fidelity? How might we study its acquisition in large scale training? We made a gigantic, verifiably correct reasoning traces of first order logic expressions! 1/9
Title highlights "FOL-Traces," a dataset for evaluating logical reasoning in language models, emphasizing rigorous testing and performance metrics.
140
Isabelle Lee @wordscompute.bsky.social · 11/02/2026
one of my new years "considerations" is to be less silent #onhere. so i guess i'll be #here and maybe also #there til february 15th
gemini summarized my google search when i was tryna look for an anti-new years resolution blog post. it says in highlight: "Approximately 80% to 88% of New Year's resolutions fail by mid-February, ..."
000
Isabelle Lee @wordscompute.bsky.social · 11/02/2026
Really excited to receive Coefficient Giving's Technical AI Safety Research Grant via Berkeley Existential Risk Initiative w/ @nsaphra.bsky.social! We aim to predict potential AI model failures before impact--before deployment, using interpretability.
161
Isabelle Lee @wordscompute.bsky.social · 29/03/2025
titled: peer review
a leaf falls on moo deng the pygmy hippo , blocking her visionmoo deng is upset presumably because she can’t see!
071
Isabelle Lee @wordscompute.bsky.social · 18/11/2024
made something with friends on friendsgiving
A bunch of strings!! The fabric kind, not the code kind. Many different shades of teal from light to dark.Great wave off kanagawa inspired little craft project, wave embroidered on a canvas
070