This is exactly right, and students are often surprised to hear it. It’s one of the first things I explain in the “Should I use LLMs?” portion of my syllabus.
I saw a thing the other day that basically said “no matter what prompt you give it, the question an LLM is answering is ‘what would a plausible response to this prompt look like?’”