We do know how neural nets do all the math. We don’t know how the math encodes complex transformations between layers during generation (ie “reasoning”). It’s not as straight forward as ‘it’s just parroting what’s in the training data’ because LLMs generalize to tasks outside of the training data.
