I think they mean WE struggle to understand these things have no understanding, probably because they are struggling with it also.
it guesses the next word, that is literally all it does, it’s not trying to build a model of reality to more accurately guess. It has no fidelity and anyone taking it seriously has themselves failed the turing test.
Wow, a video just came out that explains my position on this topic almost perfectly
https://youtu.be/AqwSZEQkknU?t=273
tl;dw: I tried to time stamp the exact point …ok, You generally can’t deduce the rules of an underlying reality from an emergent level. She calls it decoupling of scales, and it’s essentially the same problem I have with simulation theory. These programs might form a model of reality but that reality would be at best human produced descriptions of reality and most likely just a model of how best to guess the next word.
tl;dr: put glue on your pizza to stop the cheese sliding off