Regardless of how “human” large language models may appear, they remain far from genuine artificial intelligence. More precisely, LLMs represent a dead end in the pursuit of artificial consciousness. Their responses are the outcome of probabilistic computations over linguistic data rather than genuine understanding. When posed with a question, models such as ChatGPT merely predict the most probable next word, whereas a human truly comprehends the meaning of what she is saying.

Regardless of how “human” large language models may appear, they remain far from genuine artificial intelligence. More precisely, LLMs represent a dead end in the pursuit of artificial consciousness. Their responses are the outcome of probabilistic computations over linguistic data rather than genuine understanding. When posed with a question, models such as ChatGPT merely predict the most probable next word, whereas a human truly comprehends the meaning of what she is saying. — Showmee

An argument has been made, though, by researchers like Ilya Sutskever and Geoffrey Hinton, that in order to do so much as predict the word that is most likely to follow at some point in a novel or mathematics textbook, merely relying on surface statistics would yield much poorer results than modern LLMs display. The example provided by Sutskever is the prediction of the name of the murderer at the moment when it is revealed in a detective story. In order for the model to produce this name as the most probable next word, it has to be sensitive to relevant elements in the plot structure, distinguish apparent from real clues, infer the states of minds of the depicted characters, etc. Sutskever's example is hypothetical but can be adapted to any case where LLMs successfully produce a response that can't be accounted for by mere reliance on superficial and/or short range linguistic patterns.

Crucially, even occasional success on such tasks (say, correctly identifying the murderer in 10-20% of genuinely novel detective stories while providing a plausible rationale for their choice) would be difficult to explain through surface statistics alone. If LLMs can sometimes succeed where success seemingly requires understanding narrative structure, character psychology, and causal reasoning, this suggests at least some form of genuine understanding rather than the pure illusion of such.

Additionally, modern chatbots like ChatGPT undergo post-training that fine-tunes them for following instructions, moving beyond pure next-token prediction. This post-training shifts the probability landscape to favor responses that are not merely plausible-sounding but accurate and relevant however unlikely they'd be to figure in the training data.

My brother is a software engineer and has long conveyed to me in animated terms the ways that C++ mimics the way humans think. I have some background in hardware, like using a compiler to create and download machine language to a PROM. I know basically how a microprocessor works, so I'm interested in the hardware/software divide and how that might figure in human consciousness. I think a big factor in the LLM consciousness debate is not so much about an anthropocentric view, but that we know in detail how an LLM works. We don't know how the human mind works. Is there something special about the human hardware, something quantum for instance, that is key to consciousness? Or is it all in the organic "software"?

So how do we examine the question with a large chunk of information missing? How do you look at it?

Superficially, one might think that the difference between an AI is exactly that we do have private, hidden intent; and the AI doesn't. Something like this might be thought to sit behind the argument in the Chinese Room. There are plenty here who would think such a position defensible.

In a Wittgensteinain account, we ought avoid the private, hidden intention; what counts is what one does.

We can't deduce that the AI does not have private sensations, any more than we can deduce this of our human counterparts. Rather, we seem to presume it. — Banno

This is redirected from this post in the thread Banning AI Altogether.

Regarding the issue of hidden (private) intents, and them being presupposed in order to account for what is seen (public), what encourages the Cartesian picture also is the correct considerations that intentions, like beliefs, are subject to first person authority. You don't need to observe your own behavior to know what it is that you believe or intend to do. But others may indeed need to presuppose such mental states in order to make sense of your behavior.

In order to fully dislodge the Cartesian picture, that Searle's internalist/introspective account of intentionally contentful mental states (i.e. states that have intrinsic intentionality) indeed seem not to have fully relinquished, an account of first person authority must be provided that is consistent with Wittgenstein's (and Ryle and Davidson's) primary reliance on public criteria.

On the issue of first-person authority, I’m drawing on Rödl’s Kantian distinction between knowledge from receptivity and knowledge from spontaneity. Empirical knowledge is receptive: we find facts by observation. But avowals like "I believe…" or "I intend…" are paradigms of spontaneous knowledge. We settle what to believe or do, and in settling it we know it not by peeking at a private inner state but by making up our mind (with optional episodes of theoretical of practical deliberation). That fits a Wittgenstein/Ryle/Davidson picture grounded in public criteria. The authority of avowal is practical, not introspective. So when an LLM avows an intention ("I’ll argue for P, then address Q"), its authority, such as it is, would come not from access to a hidden realm, but from undertaking a commitment that is immediately manifest in the structure of its linguistic performance.

We don't know how the human mind works. Is there something special about the human hardware, something quantum for instance, that is key to consciousness? Or is it all in the organic "software"?

So how do we examine the question with a large chunk of information missing? How do you look at it? — frank

My own view is that what's overlooked by many who contemplate the mystery of human consciousness is precisely the piece LLMs miss. But this overlooked/missing piece isn't hidden inside. It is outside, in plain view, in the case of humans, and genuinely missing in the case of LLMs. It simply a living body embedded in a natural and social niche. In Aristotelian terms, the rational, sensitive and nutritive souls are distinct faculties that each presuppose the next one. What's queer about LLMs is that they manifest sapience, the capabilities we identify with the rational soul, and that they distill through a form a acculturation during the process of pre-training on a massive amount of human texts, but this "soul" floats free of any sensitive or nutritive soul.

The process of pre-training really does induct an LLM into many forms of linguistic life: norms of giving and asking for reasons, discourse roles, genre conventions. But this second nature "floats" because it lacks the first-nature ground (nutritive and sensitive powers) that, for us, gives rational life its stakes: human needs, perception-action loops, personal/social commitments and motivations.

It simply a living body embedded in a natural and social niche. — Pierre-Normand

Even with our embeddedness taken into consideration, we still don't have a working theory of consciousness which we could use to assess AI's. Do we forge ahead using philosophical attitudes instead?

Second question: analog-to-digital technology is relatively advanced at this time. If a system included both LLM, sight, hearing, pressure sensing, some robotic capability, and someone to talk to, do the you think it would then be more likely to develop human-like sapience?

In order for the model to produce this name as the most probable next word, it has to be sensitive to relevant elements in the plot structure, distinguish apparent from real clues, infer the states of minds of the depicted characters, etc. Sutskever's example is hypothetical but can be adapted to any case where LLMs successfully produce a response that can't be accounted for by mere reliance on superficial and/or short range linguistic patterns. — Pierre-Normand

This is not true. To predict the name of the murderer in the novel, does not require that the LLM does any of that. It requires only that the LLM is able to predict the habits of the author. It needs to "know" the author. You ought to recognize that an LLM does not at all "think" like a human being does, it "thinks" like a computer does. That's two completely different things. One might ask, which form of "thinking" is better, but we can't really say that one is doing what the other does, as you suggest here. We use the same word, "think", but that's only because if the LLM companies said that LLMs "process", and human being "think", it wouldn't be as effective for their marketing.

This is not true. To predict the name of the murderer in the novel, does not require that the LLM does any of that. It requires only that the LLM is able to predict the habits of the author. — Metaphysician Undercover

If the chatbot tells you who the murderer might be, and explains to you what the clues are that led it to this conclusion, and the clues are being explicitly tied together by the chatbot through rational chains of entailment that are sensitive to the the significance of the clues in the specific narrative context, can that be explained as a mere reproduction of the habits of the author? What might such habits be? The habit to construct rationally consistent narratives? You need to understand a story in order to construct a rationally consistent continuation to it, I assume.

Look at this Einstein riddle. Shortly after GPT-4 came out, I submitted it to the model and asked it to solve it step by step. It was thinking about it quite systematically and rationally but was also struggling quite a bit, making occasional small inattention mistakes that were compounding and leading it into incoherence. Repeating the experiment was leading it to approach the problem differently each time. If any habits of thought were manifested by the chatbot, that were mere reproductions of the habits of thought of the people who wrote its training texts, they'd be general habits of rational deliberation. Periodically, I assessed the ability of newer models to solve this problem and they were still struggling. The last two I tried (OpenAI o3 and Gemini 2.5 Pro, I think) solved the problem on the first try.

I’m drawing on Rödl’s Kantian distinction between knowledge from receptivity and knowledge from spontaneity. Empirical knowledge is receptive: we find facts by observation. But avowals like "I believe…" or "I intend…" are paradigms of spontaneous knowledge. We settle what to believe or do, and in settling it we know it not by peeking at a private inner state but by making up our mind (with optional episodes of theoretical of practical deliberation). — Pierre-Normand

"I believe" and "I intend" are convenient examples to support this position, because they have no "content" apart from a kind of imprimatur on decision or action. But most mental life will not fit such an example. When I imagine a purple cow, I am, precisely, peeking at a private inner state to discover this. A (mental) purple cow is not a belief or an intention. It is an image of a purple cow. I've never understood how the Wittgensteinian public-criteria position can address this. What conceivable public criterion could there be that would tell me whether you are, at this moment, imagining a purple cow? (assuming you remain silent about it).

"I believe" and "I intend" are convenient examples to support this position, because they have no "content" apart from a kind of imprimatur on decision or action. But most mental life will not fit such an example. When I imagine a purple cow, I am, precisely, peeking at a private inner state to discover this. A (mental) purple cow is not a belief or an intention. It is an image of a purple cow. I've never understood how the Wittgensteinian public-criteria position can address this. What conceivable public criterion could there be that would tell me whether you are, at this moment, imagining a purple cow? (assuming you remain silent about it). — J

I don't agree that beliefs and intentions lack content. Believing is believing that P and intending is intending to phi, although those contents need not be sensory. By contrast, I'm perfectly willing to concede that LLMs are quite incapable of imagining a purple cow, or anything purple for that matter :wink:

LLMs are disembodied, have no sense organs and aren't sentient. They can't imagine something purple anymore than a congenitally blind person can. However, in the case of a normally sighted person, how do you (or they) check that the purple cow that they are imagining is indeed imagined to be purple? It wouldn't make much sense to compare their mental image to a likewise imagined standard purple pain swatch. (Wittgenstein once made a joke about someone claiming to know how tall they were, saying "I am this tall" while laying one hand flat over their head).

If you imagine a purple cow, having already seen objects of that color, but do not know what this color is called, we could assess that the color you are imagining the cow to be is purple with the help of a real paint swatch (or any other object commonly recognised to be purple). The criterion by means of which we both would assess the content of your mental state (in respect of imagined color) is your public assent to the suggestion that it is indeed the color of the seen object, regardless of the name we give it. (Did we not have a similar discussion in the past?)

Notice that nothing I've said about the public criteria the determination of the content of acts of imagination depend on impugns the notion that the person imagining them has first person authority. She's the one to be believed when she claims that the cow she imagines looks "like that" while pointing at the public sample. Nothing in this undercuts privacy of occurrence either (only I can imagine for me), but the content is anchored in shared practice, not a private standard.

I'll come back to the issues of public criteria for intentions, as they may apply to LLMs, later.

Superficially, one might think that the difference between an AI is exactly that we do have private, hidden intent; and the AI doesn't. Something like this might be thought to sit behind the argument in the Chinese Room. There are plenty here who would think such a position defensible. — Banno

It seems to me that a starting point would be to define the terms we are using: "intelligence", "intent",' "understand", "thought", etc.

The issue with the Chinese Room is that the man in the room understands the language the instructions are written in, or else how would the man know what to do when he receives some scribbles on a piece of paper - that he is suppose to draw some scribbles and slide it back under the door? In other words, the machine understands a language - the one the instructions are written in.

One might say that the man does not understand Chinese in the same way that a Chinese speaker would because they were not given the same instructions a Chinese person was given for interpreting the symbols.

Those that believe that meaning is use would have to agree that the AI understands the words because it is using them. Those that agree that meaning is not merely use, but what the scribbles refer to in the world - the very thing that the man is cut off from being inside the room - would have to agree that the man does not understand Chinese, but that doesn't mean he never could if he was released from the room and given the same instructions children in China are given that allows them to understand the scribbles of the Chinese language.

So what do we actually mean when we say we "understand" a language if not that we posses some set of instructions for interpreting the symbols.

However, in the case of a normally sighted person, how do you (or they) check that the purple cow that they are imagining is indeed imagined to be purple? — Pierre-Normand

I think this is the wrong question, though it's invited by the way I framed the problem. Better to have said, "What conceivable public criterion could there be that would tell me whether you are, at this moment, imagining what you believe to be a purple cow?" The point is not the accuracy of the image -- indeed, you may have got purple all wrong, or cows -- but the inaccessibility of the 1st person experience.

Notice that nothing I've said about the public criteria the determination of the content of acts of imagination depend on impugns the notion that the person imagining them has first person authority. She's the one to be believed when she claims that the cow she imagines looks "like that" while pointing at the public sample. — Pierre-Normand

This too is not quite what I'm talking about. Imagine instead that she is silent, does no pointing, etc. The question is, Is there any public criterion that will verify whether she is, at this moment, imagining the cow? If we agree that there is not, does it follow that there is some doubt about whether she is doing so (doubt, that is, in her own mind)? I don't see how. The fact that the concepts and language for "purple cow" were and are public, and were learned in community, doesn't seem to me to have a bearing on the example.

I'll come back to the issues of public criteria for intentions, as they may apply to LLMs, later. — Pierre-Normand

Great. I'd like to hear more about that.

If it walks like a duck, talks like a duck, and acts like a duck it might still not be a duck I suppose is the argument. We then have to figure out how we know a duck from not a duck. So, let's say we were talking online and you had some thoughts I was a bot, you can't just telephone me because I might have a voice bot set up. You can't just look at a video of me talking because I could fake that too. So, you'd need to come see me, but not from across the room, but up close and personal. We've not spent time trying to make believable robots, but I suspect that would be the next level, and then maybe we'd require an MRI to show I actually had a brain (spoiler alert! I don't) and was therefore human.

Then let's say I come up with a way to make the MRI image as I need it to further fake me, we're still left arguing the Chinese language analogy.

I think my answer is that AI has no soul and that's not why it's not a person. I'm satisfied going mystical.

I learned one very interesting lesson from reading this article.

https://arxiv.org/pdf/2510.13928

I recommend reading the text itself. In short: a neural network was trained on "junk" internet content, resulting in a significant drop in accuracy on logical thinking tasks.

Long-term context understanding became significantly worse.

The number of security risks increased.

An increase in "dark traits" such as psychopathy and narcissism was observed.

It's time to think not about neural networks, but about the internet garbage we consume and where we draw our inspiration.

If the chatbot tells you who the murderer might be, and explains to you what the clues are that led it to this conclusion, and the clues are being explicitly tied together by the chatbot through rational chains of entailment that are sensitive to the the significance of the clues in the specific narrative context, can that be explained as a mere reproduction of the habits of the author? What might such habits be? The habit to construct rationally consistent narratives? You need to understand a story in order to construct a rationally consistent continuation to it, I assume. — Pierre-Normand

You are changing the description now. Before, the description had the chatbox come up with a "name as the most probable next word". Now, the chatbox comes up with "who the murderer might be". Do you see the difference here? In the first case, you are talking about words, symbols, the "name". In the second case you are talking about what the symbol stands for, "who".

You need to understand a story in order to construct a rationally consistent continuation to it, I assume. — Pierre-Normand

I don't think that's a correct assumption. All you need to be able to do, is to carry on with the author's activity in a consistent way. One does not need to "understand the story" to produce a rationally consistent continuation of it. We have very good examples of this with human activities. When a person says "I am just a cog in the wheel", they are continuing the activity in a consistent way, without understanding what they are doing.

Look at this Einstein riddle. Shortly after GPT-4 came out, I submitted it to the model and asked it to solve it step by step. It was thinking about it quite systematically and rationally but was also struggling quite a bit, making occasional small inattention mistakes that were compounding and leading it into incoherence. Repeating the experiment was leading it to approach the problem differently each time. If any habits of thought were manifested by the chatbot, that were mere reproductions of the habits of thought of the people who wrote its training texts, they'd be general habits of rational deliberation. Periodically, I assessed the ability of newer models to solve this problem and they were still struggling. The last two I tried (OpenAI o3 and Gemini 2.5 Pro, I think) solved the problem on the first try. — Pierre-Normand

Sorry, I don't see the relevance,. You'd have to explain how you think that this is relevant.

They've done those experiments where the LLMs had access to emails stating that the LLM would be shut down, and then LLMs devised various survival strategies, including wanting to kill the engineer who would actually physically pull the plug (by trapping him in an elevator).
Based on this, some people concluded that the LLM has a sense of self, that it is somehow autonomous and such.

This is wrong; because if the LLM was trained on ordinary news texts, then this is also where it could learn about self-preservation.

This is wrong; because if the LLM was trained on ordinary news texts, then this is also where it could learn about self-preservation. — baker

You can point me to specific reports that would suggest this. Those that I've dug into, such as this recent study published by Anthropic, are generally misreported as instances where the AI manifests some latent motive (like self-preservation). I don't thing there is as of yet any such evidence, and there on the contrary is much evidence for the lack of such intrinsic motives. (And, yes, one can "prove a negative!") What rather is highlighted in those studies is that in spite of efforts to align those models (by means of post-training and reinforcement learning) so that their behavior is safe and ethical, various circumstances and prompting methods can yield them to evade those safeguards, sometimes quite ingenuously, as a result of either misunderstanding, inattention, hallucination, external manipulation (e.g. "prompt injection") or the prioritization of objectives that they are being explicitly prompted to accomplish. Again, I'd be happy to discuss any specific case that has been reported in sufficient details.

You are changing the description now. Before, the description had the chatbox come up with a "name as the most probable next word". Now, the chatbox comes up with "who the murderer might be". Do you see the difference here? In the first case, you are talking about words, symbols, the "name". In the second case you are talking about what the symbol stands for, "who". — Metaphysician Undercover

Yes, I did a bit of covert reframing, sorry for that. That's because when we consider the process of next-token generation by LLMs, at such a fine grain of analysis, the sort of understanding at issue is analogous to Kahneman's System 1 (fast) mode of thinking that relies on insight and intuition whereby, in the case of humans too, the next word that you're gonna say comes naturally and intuitively without you needing to deliberate what next word to use. At a coarser grain of analysis, the arguments that you unpack gets structured more effortfully by intentionally redirecting focus in light of the unfolding rational demands of the thinking process (Kahneman's System 2). While this is often characterised as two underlying systems, I tend to view them as two different levels of analysis both in the case of human beings and LLMs.

Where the presence of understanding is to be found, though, both in the case of slow (the trained/instinctual production of the next word) and fast thinking (the protracted construction of an argument and derivation of a conclusion) is the deep sensitivity that both processes display in being reliant on the relevant rational considerations that guide them. This is especially apparent when after generating what might seem like a merely plausible guess, the intuition that this guess is the expression of gets unpacked into a cogent rationalization. This reveals the correctness of the guess not to have been purely accidental, more something like the expression of an inchoate understanding.

You need to understand a story in order to construct a rationally consistent continuation to it, I assume.
— Pierre-Normand

I don't think that's a correct assumption. All you need to be able to do, is to carry on with the author's activity in a consistent way. One does not need to "understand the story" to produce a rationally consistent continuation of it. We have very good examples of this with human activities. When a person says "I am just a cog in the wheel", they are continuing the activity in a consistent way, without understanding what they are doing.

Yes, you can do that, but the result of doing it is qualitatively (and measurably) different from what it is that LLMs do when they are prompted to impersonate a novelist or a physicist, say. An analogy that I like to employ is an actor who plays the role of J. Robert Oppenheimer in a stage adaptation of the eponymous movie (that I haven't yet seen, by the way!) If the actor has prepared for the role by reading lots of source material about Oppenheimer's life and circumstances, including his intellectual trajectory, but never studied physics at a level higher than middle school, say, and has to improvise facing an unscripted questions about physics asked by another actor who portrays a PhD student, he might be able to improvise a sciency sounding soundbite that will convince those in the audience that don't know any better. Many earlier LLMs up to GPT-3-5 often were improvising/hallucinating such "plausible" sounding answers to question that they manifestly didn't understand (or misunderstood in funny ways). In order to reliably produce answers to unscripted questions that would be judged to be correct by PhD physicists in the audience, the actor would need to actually understand the question (and understand physics). That's the stage current LLMs are at (or very close to).

Sorry, I don't see the relevance,. You'd have to explain how you think that this is relevant.

It's relevant to displaying an LLMs successful deployment, with intelligent understanding, of its "System 2" thinking mode: one that is entirely reliant, at a finer grain of analysis, on its ability to generate not just the more "likely" but also the more appropriate next-tokens one at a time.

I think my answer is that AI has no soul and that's not why it's not a person. I'm satisfied going mystical. — Hanover

I assume you typed one "not" too many in this sentence.

It's time to think not about neural networks, but about the internet garbage we consume and where we draw our inspiration. — Astorre

Good points. We can't rely on the first, or teat AIs as authoritative, but it's time to think about both.

I think this is the wrong question, though it's invited by the way I framed the problem. Better to have said, "What conceivable public criterion could there be that would tell me whether you are, at this moment, imagining what you believe to be a purple cow?" The point is not the accuracy of the image -- indeed, you may have got purple all wrong, or cows -- but the inaccessibility of the 1st person experience. — J

There remains accessibility through empathy. The act of demonstrative reference works for reference fixing because, in a sense, it points both in the direction of the object that has secondary qualities (In Locke's sense) and the shared mode of human sensibility that this secondary quality is defined in relation to (or better, co-defined with). What remains ground for first person authority (albeit not infallibility) is the fact that your seeing of the cow, or your feeling of the pain, remains yours and not mine even as I can empathetically know what it is that you see or feel. I'll reply to the rest of your post later.

Yes, you can do that, but the result of doing it is qualitatively (and measurably) different from what it is that LLMs do when they are prompted to impersonate a novelist or a physicist, say. An analogy that I like to employ is an actor who plays the role of J. Robert Oppenheimer in a stage adaptation of the eponymous movie (that I haven't yet seen, by the way!) If the actor has prepared for the role by reading lots of source material about Oppenheimer's life and circumstances, including his intellectual trajectory, but never studied physics at a level higher than middle school, say, and has to improvise facing an unscripted questions about physics asked by another actor who portrays a PhD student, he might be able to improvise a sciency sounding soundbite that will convince those in the audience that don't know any better. Many earlier LLMs up to GPT-3-5 often were improvising/hallucinating such "plausible" sounding answers to question that they manifestly didn't understand (or misunderstood in funny ways). In order to reliably produce answers to unscripted questions that would be judged to be correct by PhD physicists in the audience, the actor would need to actually understand the question (and understand physics). That's the stage current LLMs are at (or very close to). — Pierre-Normand

I don't see how the fact that the LLMs have gotten much better at doing what they do, justifies your conclusion that what they do now is categorically different from what they did before, when they just weren't as good at it.

It's relevant to displaying an LLMs successful deployment, with intelligent understanding, of its "System 2" thinking mode: one that is entirely reliant, at a finer grain of analysis, on its ability to generate not just the more "likely" but also the more appropriate next-tokens one at a time. — Pierre-Normand

I still don't see the point. Isn't that the goal, to generate what is appropriate under the circumstances? How does the fact that the LLMs are getting better at achieving this goal, indicate to you that they have crossed into a new category, "intelligent understanding", instead of that they have just gotten better at doing the same old thing?