Is that an issue? We think yes: How can we robustly establish cognitive abilities in reasoning models if those abilities are so closely tied to the particular contexts they were evaluated in?
You can check your favorite model's systematicity using our code: github.com/smonsays/sys...
github.com
GitHub - smonsays/systematicity-eval: Offical code for the paper ''Thought without systematicity? Evaluating reasoning models on rule induction tasks"
Offical code for the paper ''Thought without systematicity? Evaluating reasoning models on rule induction tasks" - smonsays/systematicity-eval