My PhD advisor occasionally gives me a category of homework questions to ask LLMs and asks me to report what types of problems they can do easily and which they struggle with. He uses this info to fine tune his homework assignments and give conference talks to other professors on how to do the same.