And yes, AI proponents will go "but this wasn't how you are supposed to use AI!!!" And that's correct.
But this is what it gets sold as, and this shit is being used to set policies with.
AI is useless for any kind of the work it gets sold as, unless you either check EVERYTHING, or are willing to allow for mistakes that could expose you to criminal or reputational damages.
Case in point: KPMG which is one of the big four professional networks, now with a history of false ai-generated reports.