A being that acts like it has consciousness, but does not, is called a philosophical zombie. Philosophical zombies have long been a hypothetical trope, used to make a point about consciousness. But they're not hypothetical any more. We wrote a comment exploring this issue, published on Halloween...
nature.com
AI and the coming mental health zombie apocalypse - Molecular Psychiatry
Molecular Psychiatry - AI and the coming mental health zombie apocalypse