You are grossly exaggerating. I'll grant you that IaC is a far cry from what people wanted to accomplish through all the tooling.
But making it harder? Are you kidding me? Try to provision a VM with all the security locks and some software install. It may take a bit of time to do it once via Ansible, but the second time around it boils down to shifting around some packages / changing inventory.
Compare doing it fully manually every time, go spin up the VM, ssh into it and so on.
For sure, immutable infrastructure is a pipe dream as long as people can make manual changes. Drift? It sounds like it's a sort of thing that just happens naturally but in reality is people doing down and dirty changes.
Software itself can help only so much.