Lots and lots of observability.
There was a great talk by honeycomb.io at Kubecon 2023 that went into detail about how they used o11y to check for the results of queries generated from A.I.
TLDR is you can't stop it hallucinating, but you can always tweak it if you observe it.