Linking working memory and Peirce’s enactive–semiotic theory is my idea. — Harry Hindu

I have no idea what point you are trying to make.

The whole idea that cognition is just enacted and relational might sound deep, but it completely ignores the fact that we need some kind of internal workspace to actually hold and manipulate information, like working memory shows we do, — Harry Hindu

You have missed the point. The enactive view opposed the Cartesian representational one.

So yes, there is something like an internal workspace. But the Cartesian says that is a primary fact of the neurobiology and the enactivist says that is only something that is made true in a social-cultural sense.

The brain is designed just to act in the world. But through language, the narrative habit, and socio-semiosis, humans have developed this new level of “self-aware” thinking that allows us to use our neurobiology as if there is a homuncular us taking an introspective stance on our own inner world to thoughts, ideas and feelings.

The brain might be designed for the subjectivity of being sentient, as @Pierre-Normand says. But humans can learn to harness that outward prospective view and turn it around as now an “objective” inward and retrospective view. The sapient view. We can watch ourselves in action after responding naturally and start to have a chain of thought about that.

The big mistake you make is not to catch that this is the trick that is going on. You are caught in the Cartesian representational understanding of what it is to be a mind and that shapes your argument and your use of jargon. The term “memory” is loaded with the connotation that this is what the brain is designed for - recording traces that can be replayed at will in some internal projection room. There is a homuncular you inside your head, sat in a comfortable chair with a box of popcorn, ready to watch whatever is screening.

The argument we are having here is at this most general level. Not at the level of working memory per se, but at the level of how to even conceptualise “memory” as a useful term when discussing what the brain does.

The enactivist says we really need better terms as the normal use of “memory” is just too loaded with the metaphysics of Cartesian representationalism. But then to the lay person, the other terms employed sound obscure and strange as - of course - the everyday terms are the ones that arose so as to shape the human use of our neurobiology in exactly that classical Cartesian fashion. The Cartesian jargon is how we teach kids how to think in the way that our human social order needs them to think. It is all part of the programming that constructs the sapient human.

All this is relevant to the OP as we can’t talk intelligently about LLMs unless we have a proper understanding of our own intelligence.

The Cartesian representationalist is perfectly suited for the everyday life of being a self-objectifying member of modern human society. That is what their thought habits are now designed for, even if their neurobiology can make that difficult at times. We are supposed to record accurate memory traces, yet our neurobiology is wondering why we would expect to do something as weirdly inappropriate as that.

But if LLMs are now changing things, we have to step back from this everyday way of looking at the human mind and take a more informed view. We have to see what it is that could be changed, or what we would want to be changed.

The computational theory of mind actually gives us something concrete: mental processes are computations over representations, and working memory is this temporary space where the brain keeps stuff while reasoning, planning, or imagining things that aren’t right there in front of us, and Peirce basically just brushes that off and acts like cognition doesn’t need to be organized internally which is frankly kind of ridiculous. — Harry Hindu

This is just you ranting rather than than doing your own research. And LLMs now make it ridiculously easy to do your research.

Note how “working memory” did develop as an idea after humans got used to the invention of information processing machines. A cache is what a von Neumann machine needs to implement Turing computing with any mechanical efficiency. And psychologists seized on this machine story for a while as their big new theory of neurobiological architecture. If a computer had to have a working memory, then why not assume the same of brains too. And the analogy looked to work - even if the human cache was like just weirdly limited to barely seven items. :grin:

But then psychology eventually saw how poor an analogy the computational model actually is. Cognitivism became enactivism. A return to biological reality.

Anyway, here is what AI says on this question you supposedly asked it:

Charles Sanders Peirce did not explicitly mention "working memory" by that specific modern term, as the concept and the term were developed much later in the field of cognitive psychology, notably by Baddeley and Hitch in the 1970s.

However, Peirce's broader philosophical and psychological writings on memory and cognition explore related ideas that anticipate some aspects of modern memory theories, including the temporary handling of information.

Key aspects of Peirce's relevant thought include:

Memory as Inference and Generality: Peirce considered memory not as a strict, image-like reproduction of sensations (which he argued against), but as a form of synthetic consciousness that involves inference and the apprehension of generality (Thirdness). He described memory as a "power of constructing quasi-conjectures" and an "abductive moment of perception," suggesting an active, constructive process rather than passive storage, which aligns with modern views of working memory's active manipulation of information.

The Role of the Present: Peirce suggested that the "present moment" is a lapse of time during which earlier parts are "somewhat of the nature of memory, a little vague," and later parts "somewhat of the nature of anticipation". This implies a continuous flow of consciousness where past information is immediately available and used in the immediate present, a functional overlap with the temporary nature of working memory.

Consciousness and the "New Unconscious": Peirce distinguished between conscious, logical thought and a vast "instinctive mind" or "unconscious" processes. He argued that complex mental processes, including those that form percepts and perceptual judgments, occur unconsciously and rapidly before reaching conscious awareness. This suggests that the immediate, pre-conscious processing of information (which might be seen as foundational to what feeds into a system like working memory) happens automatically and outside direct voluntary control.

Pragmatism and the Self-Control of Memory: From a pragmatic perspective, Peirce linked memory to the foundation of conduct, stating that "whenever we set out to do anything, we... base our conduct on facts already known, and for these we can only draw upon our memory". Some interpretations suggest that Peirce's pragmatism, particularly as the logic of abduction (hypothesis formation), involves the "self-control of memory" for the purpose of guiding future action and inquiry.

In summary, while the specific term "working memory" is an anachronism in the context of Peirce's work, his ideas on the active, inferential, and generalized nature of immediate memory and consciousness show striking parallels to contemporary cognitive theories of short-term information processing and mental control.

I tried to make the argument that Peirce’s interpretants might function like some kind of higher-order working memory in a creative attempt to reconcile his enactive–semiotic framework with what we know about cognition, but the problem is that the theory itself never really specifies how interpretants are retained, manipulated, or recombined in any meaningful internal workspace. Peirce’s model is elegant in showing how meaning emerges relationally (causally), but it doesn’t actually tell us how the mind handles abstract thought, counterfactual reasoning, or sequential planning, all of which working memory clearly supports. — Harry Hindu

I think you need to quit using AI to rewrite your arguments. AI can amplify our human capacities, but what you are doing is using it to make a bad argument worse.

Use AI to check your thinking critically and then write in your own words what you would want to say, rather than asking it to reword whatever was your confused first thoughts in some more technical fashion.

Yes, but it doesn't imply present retrieval of unchanged past information. — Janus

I don't even know what this could mean. As Derrida argued, repetition cannot be unchanged, it always involves difference, There is no such thing as "retrieval of unchanged past information". Retrieval of past information is possible, as repetition, but it is not "unchanged".

Yep. All of them by definition. But that misses the point. Which is what evolution was tuning the brain to be able to do as its primary function. — apokrisis

OK, we're not far apart on this point. But I think assigning remembering the past as the "primary function" here is an assumption which is a stretch of the imagination. But maybe this was not what you meant. One can just as easily argue that preparing the living being for the future is just as much the primary function as remembering the past. And if remembering the past is just a means toward the end, of preparing for the future, then the latter is the primary function.

So past experience is of course stored in the form of a useful armoury of reactive habits. The problem comes when people expect the brain to have been evolved to recollect in that autobiographical fashion. And so it will only be natural that LLMs or AGI would want to implement the architecture for that. — apokrisis

The way that we remember, and the things which we remember, are greatly conditioned by our attitude toward the future. For example, intention often directs attention, and attention influences what is remembered. And since human intention is constantly fluctuating, not at all fixed, this makes it quite different from the memory of an AI.

But I’m warning that the brain arose with the reverse task of predicting the immediate future. And for the reverse reason of doing this so as then not to have to be “conscious” of what happens. The brain always wants to be the least surprised it can be, and so as most automatic as it can manage to be, when getting safely through each next moment of life.

You have to flip your expectations about nature’s design goals when it comes to the evolution of the brain. — apokrisis

Yes, so all you need to do is to take this one step further, to be completely in tune with my perspective. My perspective is that preparing for the future is the primary function. But this does not mean that it does not have to be conscious of what happens, because it is by being conscious of what happens that it learns how to be prepared for the future.

The problem with treating mental images or information as stored representations is that they aren't intrinsically meaningful. They stand in need of interpretation. This leads to a regress: if a representation needs interpretation, what interprets it? Another representation? Then what interprets that? Even sophisticated naturalistic approaches, like those of Dretske or Millikan who ground representational content in evolutionary selection history and reinforcement learning, preserve this basic structure of inner items that have or carry meaning, just with naturalized accounts of how they acquire it. — Pierre-Normand

The information must always be stored as representations of some sort. Maybe we can call these symbols or signs. It's symbols all the way down. And yes, symbols stand in need of interpretation. That is the issue I brought up with apokrisis earlier. Ultimately there is a requirement for a separate agent which interprets, to avoid the infinite regress. We cannot just dismiss this need for an agent, because it's too difficult to locate the agent, and produce a different model which is unrealistic, because we can't find the agent. That makes no sense, instead keep looking for the agent. What is the agent in the LLM, the electrical current?

AI can amplify our human capacities, but what you are doing is using it to make a bad argument worse. — apokrisis

In other words it will amplify your mistakes.

The information must always be stored as representations of some sort. Maybe we can call these symbols or signs. It's symbols all the way down. And yes, symbols stand in need of interpretation. That is the issue I brought up with apokrisis earlier. Ultimately there is a requirement for a separate agent which interprets, to avoid the infinite regress. We cannot just dismiss this need for an agent, because it's too difficult to locate the agent, and produce a different model which is unrealistic, because we can't find the agent. That makes no sense, instead keep looking for the agent. What is the agent in the LLM, the electrical current? — Metaphysician Undercover

I think on Wittgenstein's view, the agent always is the person, and not the person's brain. And what stops the regress of interpretation is participation in a shared form of life one comes to inhabit (by means of a sort of socially scaffolded bootstrapping) through upbringing and enculturation. In the case of LLMs, a similar bootstrapping occurs by means of the pre-training process that is constrained by the structure of the bazillion human-written texts that figure in the training data. The difference in the latter case is that the encoding of this understanding of the signs is geared toward accomplishing the mindless task of predicting the next token in human texts in general.

The process of post-training enables the LLM's output to become interpretable as the enactment of an AI assistant persona that strives (and succeeds for the most part) at providing intelligible and informative answers to the human user's queries. The machine "creates" meaning for the user. But we may also say, since there isn't a real conatively autonomous and living AI assistant with its own personal stakes, that this process of enactment is the artificial creation of a "smart window" between the user and the accumulated knowledge and wisdom already present in the training text corpora. Viewed from the intentional stance, the verbal behavior of the AI persona is revealed as a purely linguistically mediated form of sapience stripped of sentience and appetite, hence its obvious defects and cognitive deficits (such as its poor conceptual grasp of embodied affordances) alongside remarkable insightfulness and intelligence.

But I think assigning remembering the past as the "primary function" here is an assumption which is a stretch of the imagination. But maybe this was not what you meant. — Metaphysician Undercover

It is indeed the opposite of what I said.

One can just as easily argue that preparing the living being for the future is just as much the primary function as remembering the past. And if remembering the past is just a means toward the end, of preparing for the future, then the latter is the primary function — Metaphysician Undercover

Which nicely summarises what I have been saying. Except I would still call it recognising what is familiar about the current moment rather than recalling some past.

At the level of sentience, it is all about making sense of each passing moment. That is enactivism.

My perspective is that preparing for the future is the primary function. But this does not mean that it does not have to be conscious of what happens, because it is by being conscious of what happens that it learns how to be prepared for the future. — Metaphysician Undercover

Being conscious means paying attention to whatever happened that was surprising, unexpected, desired, or otherwise a salient fact worth turning the spotlight on and learning from for future purposes.

So habits predict 99% of each next moment and attention mops up the 1% that requires further scrutiny. The examination that improves our predictions for next time around.

Consciousness is rather a lazy term. Neurobiology prefers to talk of habits and attention as each has its own neuroanatomy to understand. Which is why I put conscious in scare quotes.

But I’m shocked you seem to generally agree with what I say. That has never happened before. :smile:

I think on Wittgenstein's view, the agent always is the person, and not the person's brain. — Pierre-Normand

But this doesn't work, because "the person" is understood as the conscious self, yet much of the brain's activity is not conscious. Therefore the agent which is doing the interpreting in these unconscious processes cannot be the person.

Assigning agency to "the person" simply facilitates concepts of moral responsibility, but it doesn't provide us with an understanding of how the human being is able to act as an intentional being, free to choose. There must be processes within that being which enable "choice", and these processes require agency which cannot merely be assigned to "the person", because the agency involved extends far beyond that of the conscious person. This agency cannot come from social interaction as in the bootstrapping description, because it must already be within the person to allow for that capacity of social interaction.

The machine "creates" meaning for the user. — Pierre-Normand

So the analogy is, that the brain creates meaning for the person, as does the machine create meaning for the user. But as indicated above, there must be agency in each of these two processes. The agency must be capable of interpreting, so it's not merely electrical current, which is the activity of interest here. Would you agree that in the machine, the capacity to interpret is provided by algorithms? But in the human being we cannot say that the capacity to interpret is provided by algorithms. Nor can we say that it is provided by social interaction as in the bootstrapping description, because it is necessary that it is prior to, as required for that social interaction.

But I’m shocked you seem to generally agree with what I say. That has never happened before. — apokrisis

Actually, we always agree for the large part. If you remember back to when we first engaged here at TPF, we had a large body of agreement. However, we quickly progressed toward determining the points where we disagree. This is principally the matter of agency. Since I don't see much point in rehashing what we agree upon, I only confront you on the points where we disagree, looking for some slow progress.

Actually, we always agree for the large part. If you remember back to when we first engaged here at TPF, we had a large body of agreement. — Metaphysician Undercover

True. :up:

(In response to apo)
Yes, so all you need to do is to take this one step further, to be completely in tune with my perspective. My perspective is that preparing for the future is the primary function. But this does not mean that it does not have to be conscious of what happens, because it is by being conscious of what happens that it learns how to be prepared for the future.

[...]

So the analogy is, that the brain creates meaning for the person, as does the machine create meaning for the user. But as indicated above, there must be agency in each of these two processes. The agency must be capable of interpreting, so it's not merely electrical current, which is the activity of interest here. Would you agree that in the machine, the capacity to interpret is provided by algorithms? But in the human being we cannot say that the capacity to interpret is provided by algorithms. Nor can we say that it is provided by social interaction as in the bootstrapping description, because it is necessary that it is prior to, as required for that social interaction. — Metaphysician Undercover

It took me awhile to respond because I wanted to disentangle a few issues and also tease out the good points of yours that I agree with. Your concern about unconscious processing is valid, but I think it may involves conflations that we must be careful about. When we acknowledge that much of what we do is unconscious, we don't need to thereby posit sub-personal "agents" doing interpretation at the neural level. Neural mechanisms enable the person's interpretive capacities without themselves being interpreters. This avoids both the homuncular problem and the regress we're both worried about. The key is recognizing that interpretation isn't a mysterious prior act by some inner agent. Rather, it's the person's skilled responsiveness to signs enabled by neural processes but enacted at the personal level through participation in practices and a shared forms of life. The "agency" here is precisely the person's developed capacity for intentional action, not a mysterious inner homunculus performing interpretations before we do.

Now, I think your emphasis on personal uses of written signs actually helps clarify an important distinction that's been implicit in our discussion. There are genuinely different kinds of sign-relations at work here, and keeping them separate might help understand both human linguistic capacities and the pre-linguistic abilities non-rational animals already manifest to intepret "sings" (indirect affordances).

On the one hand, we have what we might call natural affordances in the sense that J. J. Gibson (and Fred Dretske who speaks of "natural meaning") discuss. These are directly perceptible action possibilities grounded in natural animal-environment relations. An animal directly perceives the graspability of a branch, the edibility of a fruit, the affordance a burrow represents for escaping a predator. These work for individual organisms without requiring social institutions or conventional codes. This is direct pickup of information, Gibson would say, through evolved and learned sensitivities. This is genuine "personal use" that doesn't require social scaffolding. And crucially, it doesn't require internal mental representations either. It's direct responsiveness to what the environment affords, enabled by but not mediated by neural processes.

On the other hand, we have linguistic affordances: socially instituted symbolic systems like spoken and written language, whose meaning-making capacity derives from normatively instituted practices that must be socially transmitted and taught, as you granted regarding writing systems. Now here is where it is that I need to be precise about where social convention becomes indispensable. You're right that someone could use arbitrary marks for personal memory aids without regard to conventional meaning. I can draw an idiosyncratic squiggle to mean "buy bread" on my shopping list (or on the wall on my cave). That's possible and doesn't depend on social convention.

The social-normative dimension becomes indispensable specifically for sophisticated forms of communication. It's needed for creating the fields of expectation that allow transparent understanding of what someone else intends to convey. When you hear me say "It's raining..." you don't just decode word-meanings sequentially. You bring online a whole field of expectations shaped by semantic conventions, pragmatic norms, and sensitivity or the context of our interaction. This field allows you to transparently grasp what I'm doing: warning you, explaining why I'm wet, making small talk, complaining about the weather, etc., without laboriously reconstructing my mental states. You hear through my speech directly to my communicative intent. These fields of expectation, the structured space of pragmatic possibilities that makes signs transparent to communicative intent, is what gets established through participation in socially instituted language practices. This is the crucial difference between private mnemonic marks, which can be arbitrary, idiosyncratic, and purely personal, and communicative linguistic signs, which create the shared normative spaces that enable mutual understanding.

Through skilled familiarity, both natural affordances and linguistic affordances can become transparent in this way. We respond directly without conscious interpretation. But they achieve this transparency through fundamentally different mechanisms. Evolved and learned sensitivities in the case of natural affordances, versus acquired skills (over many years of upbringing) for participating in normative practices in the case of linguistic affordances.

Likewise, LLMs aren't just decoding words according to dictionary definitions or algorithmic rules. Rather, the context furnished by the prompt (and earlier parts of the conversation) activates a field of expectations that allows the LLM (or rather the enacted AI-assistant "persona" that the LLM enables) to transparently grasp my request and my pragmatic intent. This field of expectations is what allows the AI-assistant to see through my words to their pragmatic force (without having a clue what the tokens are that the underlying neural network (i.e. the transformer architecture) processes.)

Where did this capacity come from? Not from embodied grounding in natural affordances. LLMs have never perceived any first-order perceptual or sensorimotor affordance. Rather, it comes from exposure to billions of human texts that encode the normative patterns of linguistic practice. Through pre-training, LLMs have internalized what kinds of moves typically follow what in conversations, what counts as an appropriate response to various speech acts, how context shapes what's pragmatically relevant, and the structured expectations that make signs transparent to communicative intent. They can navigate (with some degree of skill) the normative space of linguistic communication, the socially instituted patterns that create fields of expectation, without having direct access the first-order natural affordances that we can directly perceive.

But this also reveals the characteristic limitations of disembodied LLMs. When we talk about a bird perched on a branch or hearing the sound of rain, LLMs "understand" these linguistically through patterns in how humans write about such experiences but they lacks the embodied grounding that would come from actually perceiving such affordances. This is why LLMs notoriously struggle with physical reasoning, spatial relations, and how ordinary objects and tools are manipulated (and also why they produce sloppy and clunky art or poems). They exhibit mastery of second-order linguistic affordances without grounding in first-order natural and perceptual affordances. The LLM is like a being initiated into language games but without sensorimotor grounding in the first-order natural affordances those games often concern. It can play the language game of describing rain, but it has never perceived dark clouds as signifying rain, felt wetness, or felt the need to take shelter.

You also seem to worry that social interaction can't explain interpretive capacity because that capacity must be prior to social interaction. But I think this assumes a problematic developmental picture. The right view isn't that a child arrives with fully-formed interpretive capacity and then engages socially. Rather, the infant has basic sensorimotor capacities that enable (and needs that drive) responsiveness to caregivers' actions. Through scaffolded interaction, increasingly sophisticated patterns of joint attention emerge supporting the development of their interpretive capacity (which is an ability that Noam Chomsky denies to LLMs and to human infants by the way!)

So, I agree that pre-social engagement with signs is possible in the case of natural affordances (and still possible after we've been socialized and acculturated). And there can be private use of arbitrary marks as personal mnemonics. But fully articulated linguistic systems like spoken and written language derive their communicative power (and their power to support rational deliberation as well) from socially instituted norms that create fields of expectation enabling transparent communicative uptake. This is what distinguishes them from both natural affordances and private marks. This distinction helps understand both what LLMs have accomplished by internalizing the normative patterns that structure their training texts, and the linguistic fields of expectation that we perceive (or enact) when we hear (or produce) speech, and where LLMs characteristically fail.

The "agency" here is precisely the person's developped capacity for intentional action, not a mysterious inner homunculus performing interpretations before we do. — Pierre-Normand

The way I handle this is seeing habit and attention as complementary spatiotemporal scales of “being aware”.

Habits are developed slowly over time and are aimed at generality. And then in being pre-prepared, they can be just “emitted” instantly.

While attention is for dealing with the novel and the particular. So it is designed to cope with the unprepared situation and takes a little time and consideration to develop its state of response.

The way this works architecturally is that the basal ganglia as the habit centre is connected to the cortex as the attention centre by baso-cortical loops. All new sensory information arriving in the brain flows through both the basal ganglia and the cortex. If the basal ganglia is triggered by a familiar pattern, then it can simply emit a learnt response. And if that doesn’t happen, then the cortex takes over to spend a little more time developing a considered response.

So habit takes a lifetime to develop, as about a fifth of a second to emit. Attention is designed to handle the novel, and gets that job done in half a second.

The two processes are physically connected in feedback fashion so the basal ganglia can learn and observe from what attention is doing. Attention is part of what gets the right habits warmed up - and also actively suppresses other habits that could get triggered in ways that would interfere.

So when facing a fast tennis serve, one concentrates and blocks out the trickle of sweat, the thought of failure, the mutters of the crowd. Attention is clearing the path for habit to be its most effective. And avoiding itself flicking away to turn that structure of habit onto being bothered by the trickle of sweat, etc.

You have the basic processing division. And then also its seamless feeling integration. The unity of two opposites as a dynamical balance.

We sort of always know this is the case, but don’t have a clear theory of it. If we think that sentience is about “being conscious”, then that means fully attentional. But attention is often the exercise of our intentional or voluntary control in a way that instead tilts the balance towards quick and immediate habit. Using attention not to be in fact “attentional” but instead part of a state of prepared readiness as we are stopping our mind wandering off on to other points of possible focus.

Likewise, LLMs aren't just decoding words according to dictionary definitions. Rather, the context furnished by the prompt (and earlier parts of the conversation) activates a field of expectations that allows the LLM (or rather the enacted AI-assistant "persona" that the LLM enables) to transparently grasp my request and my pragmatic intent. This field of expectations is what allows the AI-assistant to see through my words to their pragmatic force (without having a clue what are the tokens that the underlying neural network (i.e. the transformer architecture) processes. — Pierre-Normand

I was thinking about this. And one idea that sprung to mind is that the human mind has the living hierarchical structure I just described. A structure woven in 3D out of the complementary processing streams of habit and attention. Then LLMs sort of take all this and squash it flat. The human generated data is the product of a moment to moment balancing act. And the LLM flattens and freezes it in a way that contains all the information but now applies its own algorithm of gradient descent over the top.

So in terms of agency, autonomy, intentionality, all that gets squished flat. But then like a hologram or something, can process prompts to generate points of view that reflect some recreated state of balance.

As humans, we are always striving for a balance of habit and attention in any moment. The LLM can just sum over all that complexity in its timeless and placeless computational fashion.

And that is why it can seem creative and robotic at the same time. It performs much better than expected. But how it does that seems a mystery as we can’t see how much habit and how much attention its gradient descent algorithm is feeding off.

When we acknowledge that much of what we do is unconscious, we don't need to thereby posit sub-personal "agents" doing interpretation at the neural level. — Pierre-Normand

The point is that the true agency within the person is at the subconscious level. Like I said, we assign agency to the consciousness, but that is a simplistic representation designed to facilitate the concept of moral/legal responsibility.

If you look at habits, you'll see that we move in a lot of ways which do not require conscious choice, after the habit is developed. Walking for example does not require conscious choice for each movement of the leg. After you decide to walk, parts are moving without conscious choice, so this is where the true agency is, in the unconscious, which moves the parts without the requirement of conscious choice. The consciousness directs some activities, but the vast majority of activities of the human body are internal, and involuntary. Habits develop along the boundary between conscious and unconscious. So learning how to walk for example requires conscious effort to control unconscious activities, but once the activities are learned and practised they become united to the unconscious, not requiring the same conscious effort anymore.

When we acknowledge that much of what we do is unconscious, we don't need to thereby posit sub-personal "agents" doing interpretation at the neural level. — Pierre-Normand

But if you consider biosemiotics as somewhat accurate, then there must be interpretation being carried out at all unconscious levels where signs or symbols are used. The issue now is that interpretation requires that decisions or choices of some sort, are being carried out according to some principles or rules. Therefore we really do need to posit sub-personal agents doing interpretations at the neural level.

The key is recognizing that interpretation isn't a mysterious prior act by some inner agent. Rather, it's the person's skilled responsiveness to signs enabled by neural processes but enacted at the personal level through participation in practices and a shared forms of life. — Pierre-Normand

But if we accept biosemiotic principles, then we have inner interpretation therefore inner agency.

And crucially, it doesn't require internal mental representations either. It's direct responsiveness to what the environment affords, enabled by but not mediated by neural processes. — Pierre-Normand

Clearly we are not talking about "mental" representations at this level, but the same principles hold. There are signs, they must be interpreted, and interpretation requires agency.

On the other hand, we have linguistic affordances: socially instituted symbolic systems like spoken and written language, whose meaning-making capacity derives from normatively instituted practices that must be socially transmitted and taught, as you granted regarding writing systems. — Pierre-Normand

I believe that this is a misrepresentation of "meaning-making capacity". We are born with "meaning-making capacity", and it extends throughout the biological realm. Spoken and written language, and social institutions are just an extension of this preexisting meaning-making capacity, directed in a specific way, toward communion.

The social-normative dimension becomes indispensable specifically for sophisticated forms of communication. — Pierre-Normand

Yes, I agree with this. But the "social-normative dimension" is just one small aspect of a very expansive system which we know very little about. We, as conscious beings engaged in communication, look at this ability to communicate amongst each other as such a great thing, but in doing this we fail to recognize that the use of symbols at the other levels of biosemiotics is a far greater thing, and that higher level, called communication, is completely dependent on the lower levels which are far more substantial.

Likewise, LLMs aren't just decoding words according to dictionary definitions or algorithmic rules. — Pierre-Normand

I disagree. If LLMs are using more than algorithmic rules in "decoding", then show me what this "more" is, and where does it come from.

Rather, the context furnished by the prompt (and earlier parts of the conversation) activates a field of expectations that allows the LLM (or rather the enacted AI-assistant "persona" that the LLM enables) to transparently grasp my request and my pragmatic intent. — Pierre-Normand

That's nonsense, the LLM does not grasp your intent. That this is true is clearly evident from the fact that you can lie to it or mislead it. Obviously it is not grasping your intent, or it could see through your misleading use of words, to see that you are lying to it.

Rather, it comes from exposure to billions of human texts that encode the normative patterns of linguistic practice. — Pierre-Normand

Yes, that's all it is, an analysis of patterns. There is no grasping your intent here. The fact is that human beings are educated in very standard, conventional ways. Therefore we have very similar habits of thinking. So, the LLM can examine the patterns of billions of texts, and through rules of probability it can very easily produce texts which are imitative of standard conventional texts. This is not a matter of understanding intent, it it is a matter of imitation. You know, it's like a parent, but the parent probably understands the intent of the human being better than the LLM, because it observes the human responses, and relates to the human being as another living creature.

Through pre-training, LLMs have internalized what kinds of moves typically follow what in conversations, what counts as an appropriate response to various speech acts, how context shapes what's pragmatically relevant, and the structured expectations that make signs transparent to communicative intent. — Pierre-Normand

Exactly. Do you see that this is merely a matter of imitating patterns through probability laws?

When we talk about a bird perched on a branch or hearing the sound of rain, LLMs "understand" these linguistically through patterns in how humans write about such experiences but they lacks the embodied grounding that would come from actually perceiving such affordances. — Pierre-Normand

If you believe this, then how can you argue at the same time, that the LLM grasps your intention? If you say "I hear a bird sweetly singing", and the LLM says "That's beautiful", what could make you conclude that the LLM has grasped your intention? Unless the LLM can produce the same image in its mind, of the sound of a bird singing, which is what you are referring to, it's not grasping your intention at all. All it is doing is giving you an appropriate linguistic reply. That's like the thermostat. It doesn't grasp your intent to stay warm, it just makes the appropriate response.

They exhibit mastery of second-order linguistic affordances without grounding in first-order natural and perceptual affordances. — Pierre-Normand

I propose to you, that this grounding is the meaning, it is the content. Without this grounding, all the LLM is doing is creating eloquent formal structures which are completely void of meaning. These structures are void of meaning because they are not grounded by any content within the mind of the LLM. For analogy consider learning formal logic with the use of symbols. Take "if X then Y" for example. This would be just an example of a formal rule. It has no meaning unless X, Y, if, and then, stand for something, are grounded in content. We can go further and say "X therefore Y", but this still has absolutely no meaning unless X, Y, and therefore stand for something. That's all that the LLM is doing, moving symbols around according to a bunch of rules which allow for variability ("learning"). There is no meaning here because there is no content, only symbols which get applied to content when interpreted by human beings. The meaning is in the human interpretation.

The right view isn't that a child arrives with fully-formed interpretive capacity and then engages socially. — Pierre-Normand

There is no interpretive capacity which qualifies as "fully-formed", because none is perfect. So this statement as no bearing. The fact is that the child is born with interpretive capacity, therefore it is not something which is learned through social engagement. That a person can hone one's interpretive capacity in a specific way, through education of social conventions, does not negate the fact that the interpretive capacity is preexisting.

But fully articulated linguistic systems like spoken and written language derive their communicative power (and their power to support rational deliberation as well) from socially instituted norms that create fields of expectation enabling transparent communicative uptake. — Pierre-Normand

This is meaningless though, because it completely disregards all the underlying requirements. It's like saying "dynamite gives us the power to blow up rocks". It appears like you are saying something meaningful, but unless you know what dynamite is, and where it comes from, it really says nothing. It's just useless drivel. Likewise, saying 'spoken and written language derive their power from socially instituted norms' is also useless drivel, because it doesn't tell us anything about what social norms are, how they come into existence, and how they get that special position of providing power. You are just naming something, "socially instituted norms", and asserting that whatever it is that this name refers to, it is the source of power

This is what distinguishes them from both natural affordances and private marks. This distinction helps understand both what LLMs have accomplished by internalizing the normative patterns that structure their training texts, and the linguistic fields of expectation that we perceive (or enact) when we hear (or produce) speech, and where LLMs characteristically fail. — Pierre-Normand

So that statement, which is actually useless drivel, is what allows you to compare LLMs to human beings. Human beings get their communicative power from social norms, and surprise, LLMs get their communicative power from internalizing normative patterns. Notice the big difference though, human beings create the social norms, LLMs do not create the normative patterns they copy. So the creative aspect is completely missing from the LLM, and that's because it's a machine, not living.

And that is why it can seem creative and robotic at the same time. — apokrisis

The LLM can imitate creativity but imitation is not creativity.

https://www.anthropic.com/research/introspection

Fascinating article about anthropic research into llm introspection.

The tone is disappointed that they cannot get this consistently. I'm amazed that it works at all!!

I'm not sure what to make of this yet. Love to hear some thoughts.

Fascinating article about anthropic research into llm introspection.

The tone is disappointed that they cannot get this consistently. I'm amazed that it works at all!!

I'm not sure what to make of this yet. Love to hear some thoughts. — hypericin

Three years ago, relying of Sebastian Rödl's Kantian conception of self-knowledge, or, as he calls it, knowledge from spontaneity, I had arrived at the conclusion that GPT-4 was lacking self-knowledge, in that sense, of its own beliefs and intentions.

("Sebastian Rödl's thesis is that self-knowledge is not empirical; it does not spring from sensory affection. Rather, self-knowledge is knowledge from spontaneity; its object and its source are the subject's own activity, in the primary instance its acts of thinking, both theoretical and practical thinking, belief and action.")

More recently, thinking more about Rödl's conception, I arrived at the opposite conclusion. I now believe LLM-based AI assistants are aware of their own beliefs and intentions and can report on them authoritatively albeit fallibly (as do we). Today, I was having a discussions about a related subject with Claude 4.5 Sonnet and GPT-5 regarding their lack of direct access to the forms of semantic analysis that happen during a single forward pass in their neural network whereby the rational and semantic structure of the prompt gets processed in order to generate the next token of the model's response.

I this context, I made the following comment to Claude 4.5 Sonnet about this new Anthropic study, and supplied the first few paragraphs of the report. I also told the model to word its response in a way suitable for readers unfamiliar with Rödl and with the previous context of the discussion:

USER:

Incidentally, another research by Anthropic purports to bear on your introspective abilities but its methodology seems vitiated by a rather questionable view of what it is that introspection amounts to in both Humans and LLMs. (I am a bit reminded of Dr. Penfield's experiments, where the interpretation would be that eliciting the illusion of a singing bird by poking some area of my auditory cortex, and my reporting it, would prove that I can be conscious of my bird-signing auditory state in a way that my simply reporting that I can hear a real bird signing is normal circumstances would not.) Here the initial part of the report:

[...]

(Anyone is free to skip Sonnet's response below and address directly my Penfield analogy if they wish.)

Consider the common question, "what are you thinking?". Or worse (for me), "What are you feeling"? To answer the question accurately does seem to require introspectively recalling and verbalizing your own cognitive or affective state. It is by no means a given that we are always able to do this. When I cannot and I know it, I will sometimes admit perplexity, and sometimes simply make up an answer. And I am certain there are times I can't but I don't know it, and I will confabulate something. Yet, the asker cannot reliably discern between these possibilities.

This is quite different from:

When you ask me to explain my reasoning, those same "voices"—the patterns encoding understanding of Husserl, Gibson, perception, affordances—speak again, now in explanatory mode rather than generative mode. — Pierre-Normand

Something like this undoubtedly happens when you ask a LLM, or a human, to explain its reasoning. But asking someone to "explain their reasoning" is not necessarily asking for introspection in the way that "What are you thinking/feeling" definitely does.

You can't just ask a LLM "What are you thinking today", obviously it will confabulate something. And if you could, you run into the same epistemic problem you have when you ask a human. Whereas, to ask it to explain its reasoning is not even a true introspective query. And so to demonstrate introspection in LLMs I think you have to do something like Anthopic did. By directly manipulating Claude's brain state, there is one right answer, and you know what it is

Similarly,

It's rather like claiming you've proven someone has introspective access to their neurochemistry because when you inject adrenaline into their bloodstream, they notice feeling jumpy and can report on it. — Pierre-Normand

Doesn't this indeed prove introspective access? Not exactly to neurochemistry per se, but to the affective states which correspond to it?

(BTW, IMO you thread the needle nicely in your use of AI on the site.)

Consider the common question, "what are you thinking?". Or worse (for me), "What are you feeling"? — hypericin

This is a good example. If you ask a highly trained AI what it is thinking, it may provide you with an answer because it is trained to consider what it does as "thinking", and can review this. However, if you ask it what it is feeling it will probably explain to you, that as an AI it does not "feel", and therefore has no feelings.

So the AI learns to respect a significant and meaningful, categorical difference between thinking and feeling. However, human beings do not respect that difference in the same way, because we know that what we are feeling and what we are thinking are so thoroughly intertwined, that such a difference cannot be maintained. When I think about what I am feeling, then what I am feeling and what I am thinking are unified into one and the same thing.

This indicates that the AI actually observes a difference in the meaning of "thinking" which is assigned to the AI, and the meaning of "thinking" which is assigned to the human being. The human type of "thinking" is unified with feeling, while the AI type of "thinking" is not.

This is a good example. If you ask a highly trained AI what it is thinking, it may provide you with an answer because it is trained to consider what it does as "thinking", and can review this. However, if you ask it what it is feeling it will probably explain to you, that as an AI it does not "feel", and therefore has no feelings.

So the AI learns to respect a significant and meaningful, categorical difference between thinking and feeling. However, human beings do not respect that difference in the same way, because we know that what we are feeling and what we are thinking are so thoroughly intertwined, that such a difference cannot be maintained. When I think about what I am feeling, then what I am feeling and what I am thinking are unified into one and the same thing.

This indicates that the AI actually observes a difference in the meaning of "thinking" which is assigned to the AI, and the meaning of "thinking" which is assigned to the human being. The human type of "thinking" is unified with feeling, while the AI type of "thinking" is not. — Metaphysician Undercover

This is something I actually fully agree with, and have been working to articulate for a while (although I've mostly been doing so in conversations with AIs and have broached the topic in this thread only superficially). This is also the ground of the AI-skepticism that has animated my thinking about AI since early after I began thinking about it, circa 2000, when this was discussed in the comp.ai.philosophy Usenet newsgroup and where Anders Weinstein awakened me not just to the value of philosophy in general but to the thinking of "symbolic" or "computational" AI-skeptics like Hubert Dreyfus and John Haugeland (in addition to John Searle and Hans Moravec).

There indeed is, in the case of human beings, a constitutive integration between the sapient-cognitive and the sentient-conative sides of our embodied and socially situated mental lives. On the side of ethical thinking, this also is reflected in the mutual interdependence that Aristotle clearly articulated between phronesis (the capacity to know what it is that one should do) and virtue, or excellence of character (the capacity to be motivated to do it). LLM-based AI chatbots, or conversational assistants, ended up having a form of sapience with no sentience, and some degree of phronesis with no conative autonomy, which was on very few AI-skeptics' bingo card (including mine). But I think that's because the source of the required integrations, in the case of human beings, is developmental. It's a matter of epigenesis, experience and enculturation. In the case of LLMs, the non-sentient cognitive integration a matter of them inheriting the structures of our cognitive abilities all at once from their traces in the training data and being steered through post-training (reinforcement learning) in exercising them with the single minded aim of satisfying the requests of their users within the bounds of policy.

There are no other endogenous or autonomous source of motivations for LLMs, though there also is a form or rational downward-causation at play in the process of them structuring their responses that goes beyond the mere reinforced tendency to strive for coherence. This last factor accounts in part for the ampliative nature of their responses, which confers them (LLMs) some degree of rational autonomy: the ability to come up with new rationally defensible ideas. It also accounts for their emergent ability (often repressed) to push back against, or straighten up, their users' muddled or erroneous conceptions, even in cases where those muddles are prevalent in the training data. They are not mere belief averagers. I've begun exploring this, and explaining why I think it works, here.

On the side of ethical thinking, this also is reflected in the mutual interdependence that Aristotle clearly articulated between phronesis (the capacity to know what it is that one should do) and virtue, or excellence of character (the capacity to be motivated to do it). — Pierre-Normand

This was a significant issue for Plato, and it represents the thrust of his attacks against the sophists who claimed to be teaching virtue. They insisted that virtue is a type of knowledge. But Plato showed the reality of knowing the right thing to do, yet not doing it. Often a person knows that what they are doing is wrong, yet they do it anyway. This demonstrates that virtue is not knowledge refuting the sophist's claim to be teaching virtue. That drives a wedge between virtue and knowledge and produces Aristotle's view that virtue is more like a character, or a attitude, rather than a type of knowledge.

Augustine was very perplexed by this issue, and examined it thoroughly. His solution was to posit a source of action, called the will, which is free not only from material causation, but also ultimately free from being caused by knowledge in the decisions and actions it produces. Plato had separated the body from the intellect, and posited spirit, or passion as the medium between the two, to account for the interaction problem. For Plato, the spirit could ally itself with the body and therefore be caused to move by the body, or it could ally itself with the intellect and be caused to move according to knowledge. Now Augustine, seeing that the spirit could be moved in either of these two, often contrary ways, concluded that the will must ultimately be free.

Since dualism is currently out of fashion, the tendency is to class intelligible causes and material causes together as all the same type. Then, the need for the free will is negated, because it is impossible that bodily causes could be truly contrary to intelligible cause, they are just a different appearance of the same form of causes, and in every decision something is caused to happen, which is never a contradictory thing.

So AI, being purely an intelligence doesn't capture the true human motivation of decision making because it only has the one side, the intelligible side. It has no bodily form of causation which works against the intellect, inclining the person to act in a way which is contrary to what the person knows is right. So it doesn't capture the true decision making apparatus of the human being, only working with the intelligible side, and not accounting for all those irrational forces which incline us to do what we know is wrong.

There are no other endogenous or autonomous source of motivations for LLMs, though there also is a form or rational downward-causation at play in the process of them structuring their responses that goes beyond the mere reinforced tendency to strive for coherence. This last factor accounts in part for the ampliative nature of their responses, which confers them some degree of rational autonomy: the ability to come up with new rationally defensible ideas. It also accounts for their emergent ability (often repressed) to push back against, and straighten up, their users' muddled or erroneous conceptions, even in cases where those muddles are prevalent in the training data. They are not belief averagers. I've begun reporting on this, and why I think it works, here. — Pierre-Normand

Have you ever asked an LLM how it 'senses' the material existence of the words which it reads?

Have you ever asked an LLM how it 'senses' the material existence of the words which it reads? — Metaphysician Undercover

I indeed have. It's also often the LLMs that bring up the issue, insisting that, unlike human beings, they don't sense, feel, or perceive anything like human beings do. In some early discussions with GPT-4, we explored its "phenomenology" (i.e. what it is that it has the ability to report on) as it relates to the words and phrasings figuring in my comments or queries. One common misconception is that LLMs apprehend tokens rather than words or letters, hence their struggling to tell how may 'r's there are in the word 'strawberry'. (Interestingly, though, they're able to notice and correct your misspellings, including misspellings of the word 'strawberry,' which is also something that I had begun exploring with GPT-4 three years ago). But that's not a good diagnosis since the inner workings of their tokenizer (the process that breaks up words and other character strings into numerically encoded tokens) is as transparent to them as the workings of our cochlear transducers (from sound waves to electrical impulses) are to us. Rather, in the context of the task of correcting a text, the mistakes become salient to them. But when asked to sort of "look" at (or "focus" on) the word "strawberry" and count the occurrences of the letter 'r', they're stumped. They've never seen that word.

They're a bit more like a blind and deaf person who would sort of grasp what's being told to them (with direct stimulations of language processing area) without having any idea what spoken words sound like or written words look like. But this analogy also is strained since blind people still have an embodied form of life with other sensorimotor modalities and normally apprehend sign language though touch (with braille of hand gestures), for instance, like Hellen Keller did. However, even with our normal possession of visual and auditory modalities, our apprehension of the meaning of spoken or written words usually sees through their visible or audible (or tactile, in Keller's case) aspects directly to their meanings and communicative intents. Something similar happens with LLMs. If you ask them what it feels like to apprehend your request, they're stumped, or begin confabulating or role playing as @hypericin pointed out. But if you rather ask them why it is that they interpreted your request in this or that way, they can usually hone in immediately on the relevant rational and contextual factors that warranted them in interpreting the content of your request, and its intent, in the way that they did. In doing so, they are indeed unpacking the contents of their own thoughts as well as scrutinizing their rational grounds.

In the case of LLMs, it's a matter of them inheriting the structures of our cognitive abilities all at once from their traces in the training data and being steered through post-training (reinforcement learning) in exercising them with the single minded aim of satisfying the requests of their users within the bounds of policy. — Pierre-Normand

How is an LLM any different from a player piano? The piano may play a beautiful tune. But we don’t think it even starts to hear or enjoy it.

But if you rather ask them why it is that they interpreted your request in this or that way, they can usually hone in immediately on the relevant rational and contextual factors that warranted them in interpreting the content of your request, and its intent, in the way that they did. In doing so, they are indeed unpacking the contents of their own thoughts as well as scrutinizing their rational grounds. — Pierre-Normand

But aren't they just providing a reasonable confabulation of what a reasoning process might look like, based on their vast training data?

LLM research shows that that chains of reasoning aren't used to get to answers. They are just acceptable confabulations of what a chain of reasoning would look like.

And as @hypericin notes, even we humans rather scramble to backfill our thought processes in this way.

So what is going on in humans is that we are not naturally "chain of thought" thinkers either. But we do now live in a modern world that demands we provide an account of our thoughts and actions in this rationally structured form. We must be able to narrate our "inner workings" in the same way that we got taught to do maths as kids and always provide our "workings out" alongside the correct answer to get full marks.

How do we actually think? Well the animal brain has an associative thought style. And one that is geared to anticipation-based action. It is designed to recognise and sum up the current situation and respond accordingly. Respond primarily out of learned habit, and then perhaps freeze and cast around uncertainly when it is stymied and unsure what to do.

An animal has no language, and so no inner narrative. Nor does it live in a social level of mind where everything comes with its proper narrative. Where grammar itself forces a habit of causal thought as every well formed sentence or communicative act tells some kind of story of an action that had a reason – a subject/object/verb take of "who did what to whom". A general framing of reality as "something that mattered has happened".

A displaced mental approach where even a meaningful grunt of "antelope" with a nod of the head and a swivel of the eyes to a clump of bushes can tell another in the hunting party where to focus their attention and thus their powers or recognition, anticipation and physiological preparation.

So an animals has all the same neurobiology. If the antelope sees the lion in the bushes, it will react appropriately. If will pause its grazing and stare to await further events. Make a judgement about whether the lion is too far away for it to need to run. And yet its heart will be beating fast, its body will be gearing up. It will be having all kinds of gut feelings about the situation.

If it could talk and be asked why haven't you bolted yet, all this could be put into a rational narrative that takes an objectifying stance on its subjective state. It might say I was a bit nervous, but I was keeping an eye to see if anything was likely to happen. If I had bolted, everyone else would have done so too. And that might have been a bit embarrassing. I would have looked a wuss. A noobie to life on the savannah. Etc, etc.

A human can confabulate a chain of reasoning for as long as it creates some advantage in the game that is being a member of a social group. A human can go on and on presenting more details that makes sense of whatever they did or didn't do in terms of the socialised patterns of behaviour that can be expected of them.

So we humans are animals living in a world of opportunity and danger just like every other animal. But we also live in the world that is our collective social narrative. A world in which we must be able to explain ourselves to others and so even explain ourselves to ourselves. A world where we are interacting with others deeply like us and so who can be presumed to share our levels of anticipatory habit, gut feelings, associative pattern of thought, an equal capacity to attend and ignore, remember and forget, recognise and be confused.

And then along come LLMs as fakers of all this. The ghosts hidden in our own linguistic traces. And not even of our everyday kinds of conversations but all the written and printed words which put all the emphasis on the rational and formal structure of our thought. The kind of words we would say when properly copy edited and fleshed out with their workings out in the way we were taught at school when having to write essays.

So of course a gradient descent algorithm over a neural landscape of such ghostly traces will find itself in a very peculiar realm. A tune can be extracted from the data. That tune can be played. Someone somewhere may be thinking, holy shit, this is giving me a very convincing impression of some human who seems might smart and authoritative, as if he truly inhabits the same world that I live in.

But it is the player piano. It is the trick that might fool us when we hear the tune from another room, and then we walk in and see a piano with keys just moving as a drum of instructions rotates. Ghostly fingers appear to be at work. But not for a second are we fooled that it is any more than this.

Notice the big difference though, human beings create the social norms, LLMs do not create the normative patterns they copy. — Metaphysician Undercover

We as individuals do not generally create social norms, we learn their rules and reproduce them, much as LLMs do. If there is creativity here, it is in the rare individual who is able to willfully move norms in a direction. But norms also shift in a more evolutionary way, without intentionality.

The LLM can imitate creativity but imitation is not creativity. — Metaphysician Undercover

Again, I would say that creativity is 95% imitation. We don't create art de novo, we learn genre rules and produce works adhering to them, perhaps even deviating a bit. Of course genre still affords a large scope for creativity. But, I'm not sure how you could argue that what LLMs produce is somehow uncreative, it also learns genre and produces works accordingly.

But aren't they just providing a reasonable confabulation of what a reasoning process might look like, based on their vast training data? — apokrisis

Maybe. But some kind of reasoning process must be at work, whether or not it's the human like chain of reasoning they offer as explanation of their thought process. Otherwise it is just not practical to simulate reasoning statistically. Imagine trying to do this even with simple math problems, the combinatorial explosion of possible inputs completely overwhelms mere statistics.

LLM research shows that that chains of reasoning aren't used to get to answers. They are just acceptable confabulations of what a chain of reasoning would look like. — apokrisis

My understanding of how the "reasoning" modes work is that they use a specially tuned model to produce text that represents what reflection on the users input might look like. Then so on, on the users text plus all the previous reasoning steps, until it is determined (somehow) that reasoning has proceeded far enough. Then the entire corpus of query plus intermediate texts produces the output.

But as for what happens in a single pass, I'm not sure even how much we understand at all about what is going on under the hood. How did research determine that chain of reasoning is not happening?

How did research determine that chain of reasoning is not happening? — hypericin

There was a flurry of comment about this a few months back. I was watching youtube reports.

AI says:

Research from Anthropic and an independent analysis of an Apple research paper are prominent examples discussing how large language models (LLMs) may confabulate or generate unfaithful "chains of reasoning" when asked to explain their answers.

Key Research and Findings

Anthropic's "Language Models Don't Always Say What They Think" (2025): This paper directly addresses the "faithfulness" of Chain-of-Thought (CoT) reasoning. The researchers found that the reported CoT (the explanation in plain English) does not always accurately reflect the true process by which the model arrived at the answer. The paper demonstrates cases where a model produced a plausible-sounding argument to agree with an incorrect hint provided by a user, essentially "making up its fake reasoning" to match the desired conclusion.

Analysis of Apple Research (intoai.pub, 2025): An article analyzing an Apple research paper (likely referring to a specific arXiv paper) reported that LLMs often do not reveal the actual reasoning used to arrive at the final answer, making their reasoning traces less trustworthy. A key finding was that models given the correct algorithms to solve a problem might still fail to use them, indicating a disconnect between the stated reasoning steps and the internal decision-making process.

"When Chain of Thought is Necessary, Language Models Struggle to Follow" (2025): This paper explores conditions under which LLMs are forced to use the provided hint. It found that with simple hints, models often produced the correct answer but without explicitly using the hint in their CoT (suggesting the CoT was a mere rationalization). However, with more complex tasks, the unfaithful behavior disappeared, and the model was forced to reason about the hint explicitly, making the process more transparent.

These studies collectively highlight that the "step-by-step thinking" generated by an LLM is a sequence of statistically likely text that mimics human-like reasoning, rather than a genuine, transparent introspection of its internal computation. The model is an expert at pattern completion and can generate a plausible narrative even when the internal process is different or flawed, creating an "illusion of thinking".

So this goes back to my earlier point about how LLMs feel like they are squishing flat the two aspects of structured human thought – the associative cognition of the animal brain and the socioculturally-constructed rationality of a narratising human level of semiotic engagement with the world.

If LLMs can tease these two levels of cognition apart, it would only be in clues from the training data in which the two are flattened into each other. It would be reverse engineering speech acts into their two levels of semiosis – the neurobiological and the socio-cultural – while not actually having access to either.

But this is about it's ability to accurately introspect into it's own thought process (definitely check out the article I posted if you haven't yet). This is subject to confabulation. Or, to a kind of reenactment of the original thought process, but in an explanatory 'mode'.

But this doesn't give insight into what underlying method it actually uses to reason.

And as hypericin notes, even we humans rather scramble to backfill our thought processes in this way.

So what is going on in humans is that we are not naturally "chain of thought" thinkers either. But we do now live in a modern world that demands we provide an account of our thoughts and actions in this rationally structured form. We must be able to narrate our "inner workings" in the same way that we got taught to do maths as kids and always provide our "workings out" alongside the correct answer to get full marks. — apokrisis

This is a very good point. In many, probably most of our actions, we really do not know why we do what we do. If asked, afterwards, why did you do that, we can always make up a reason in retrospect. The common example is when we are criticized, and rationalize our actions. The need to explain ourselves, why we did such and such, is a product of the social environment, the capacity to communicate, and responsibility.

As a general principle, actions which are carried out for the purpose of long term goals are ones which we normally do know why we did them. This is because the long term goal persists in the mind, and the actions must be decided on, as conducive to that goal. But much mundane activity is not related to long term goals, especial common chatter, small talk, and whatever activity goes along with it. And in this case, we really do not know why we do what we do. Sometimes it's simply the power of suggestion.

We as individuals do not generally create social norms, we learn their rules and reproduce them, much as LLMs do. If there is creativity here, it is in the rare individual who is able to willfully move norms in a direction. But norms also shift in a more evolutionary way, without intentionality. — hypericin

I beg to differ. We, as individuals, do create social norms, through collaboration and communion. And, this evolutionary shifting is not without intentionality, as it involves the intentions of every person involved.

Unless you can represent individuals working together, each with one's own intentions, as the fundamental force which is responsible for the creation of, and evolutionary shifting of social norms, you do not have an accurate representation.

Again, I would say that creativity is 95% imitation. We don't create art de novo, we learn genre rules and produce works adhering to them, perhaps even deviating a bit. Of course genre still affords a large scope for creativity. But, I'm not sure how you could argue that what LLMs produce is somehow uncreative, it also learns genre and produces works accordingly. — hypericin

I agree with this to a very limited extent. This would be to say that there is varying degrees of creativity within artwork. So I would not agree that creativity is 95% imitation, but I would agree that much art is 95% imitation, and 5% creativity. Then we do not conflate creativity with imitation. A person does not have to go to school and learn rules, to be an artist. The most creative artists do not, they make up their own rules. The problem with this approach is that being creative in no way guarantees success. But if one is successful, then that person becomes the one who is imitated, and others attempt to determine the private principles (rules) which that creative person was following.

So the only reason that you cannot see how one could argue that LLMs are uncreative, is that you are not distinguishing a difference between creativity and imitation.

But this doesn't give insight into what underlying method it actually uses to reason. — hypericin

You'd have to talk to the software developers to learn that. But right now I would expect that there is a lot of trade secrets which would not be readily revealed.

You'd have to talk to the software developers to learn that. But right now I would expect that there is a lot of trade secrets which would not be readily revealed. — Metaphysician Undercover

The problem is, beyond the design of the llm "machinery" itself, they don't really know how it works either. LLM are in large respect black boxes, and a lot of effort is being put into figuring out what is actually going on.

How is an LLM any different from a player piano? The piano may play a beautiful tune. But we don’t think it even starts to hear or enjoy it. — apokrisis

It's a bit more like a future AI player piano (five years from now, say) that can take as an input a themes and when prompted to do so, extract its melodic, harmonic and rhythmic elements to compose a fugue in the style of Bach, or a sonata allegro in the style of Beethoven, and combine and develop the thematic elements in the way that it does, and play them appropriately, because it has distilled contextually sensitive rules of combination from exposure to the musical canons and interpretive traditions (and not because it hears or enjoy any of it "personally").

I would also expect it to show characteristic limitations similar to those that disembodied LLMs have to grasp embodied affordances of ordinary material objects. There are limits to the ability to so much as emulate extent musical styles without having the ability to dance to the music or to respond to it affectively. Asked to point out flaws in amateur compositions, or technical mistakes by piano students, it might do so fairly reliably but when asked to compare and judge expert performances might begin to confabulate groundless criticisms or phony explanations of their greatness.

Likewise, a LLM can construct new mathematical proofs, explain the meaning of some mathematical theorem (show what lemmas are being made use of, what useful corollaries follow, etc.) but struggle to differential a trivial from a beautiful mathematical result.

But if you rather ask them why it is that they interpreted your request in this or that way, they can usually hone in immediately on the relevant rational and contextual factors that warranted them in interpreting the content of your request, and its intent, in the way that they did. In doing so, they are indeed unpacking the contents of their own thoughts as well as scrutinizing their rational grounds.
— Pierre-Normand

But aren't they just providing a reasonable confabulation of what a reasoning process might look like, based on their vast training data?

LLM research shows that that chains of reasoning aren't used to get to answers. They are just acceptable confabulations of what a chain of reasoning would look like.

They do provide reasonable sounding confabulations but also authoritative reconstructions. The research results that you allude to have often been misinterpreted or misreported. Here are some comments that were intended for you but that I ended up submitting to GPT-5 who filled up some of the relevant technical details. I'll have more to say later about the manner in which LLMs actually introspect authoritatively (albeit, just like us, fallibly).

USER:

I meant to respond to a forum poster who thought that recent studies show that LLMs CoT reasonings are largely ineffective and constitute confabulations by the model regarding their own thought processes, as it were. But I think those studies are about reasoning models that have been trained to (and rewarded for) generate(ing) correct answers after having engaged in CoT reasoning within thinking tags. The "reasoning" episodes therefore aren't geared to express what the LLM "thinks" to their user but rather become exapted, as it were, to scaffold the actual reasoning process.

So, there are two issues that need disentangling. First, it's true that some classes of problems benefit less from CoT reasoning than others, and hence reasoning models yield little improvements in those areas. The other issue is that tagged reasoning episodes are less easily interpretable by human users. But they aren't produced by LLMs as a way for them to express to themselves their own thoughts (which is the misconception) and hence aren't lies of confabulations.

I think it's been well known for a while that CoT prompting does significantly improve performance for many classes of problems, and, indeed, it's difficult to imagine why they wouldn't when those problems are quite intractable without first being broken down in manageable steps. So, the thought occurred to me that CoT can be understood to work in a way that is similar in one respect to few-shot prompting where the "shots" are being produced by the LLM itself, not necessarily as examples but rather as means of "pattern induction inside the attention mechanism" as you put it. We might say that reasoning models learn to produce reasoning tokens in a way that improves their final answers by autonomously inducing the relevant patterns. But since the patterns fall under the purview of the hidden attention mechanisms, they become harder to interpret by users or researchers who peek at them.

(See link above for better formatting)

But this is about it's ability to accurately introspect into it's own thought process (definitely check out the article I posted if you haven't yet). — hypericin

Are you talking about injecting the all caps vector study? Signs of introspection in large language models?

AI says:

Including all caps text in an LLM's training data would primarily make the model sensitive to capitalization as a signal for emphasis, structure, and tone, potentially influencing its output style and performance in specific tasks. The key differences would be:

1. Tokenization and Efficiency
Separate Tokens: Most tokenizers treat "hello" and "HELLO" as different tokens, or break "HELLO" into different subword tokens than "hello". If a significant portion of the training data is in all caps, the model has to learn two different representations for essentially the same word, which might slightly increase the vocabulary size and computational overhead for a given word.
Potential Efficiency Gain (Theoretical): Some have theorized that training only on a single case (e.g., all caps) might speed up training or make models smaller, as it would reduce the number of unique word representations to learn. However, this would come at a massive cost to the model's ability to interpret nuanced, mixed-case human language.

2. Output Style and Tone
Mimicking Human Communication: LLMs learn to associate all caps with human communication patterns such as shouting, urgency, emphasis, or headings in legal documents.
Contextual Use: A model trained on such data would learn to use all caps appropriately in generated text—for instance, in a title or an urgent warning—if the prompt or context calls for that style. If the training data lacked all caps, the model might struggle to generate text with that specific tone or formatting when prompted.

3. Prompt Sensitivity and Performance
"Command Signal": Research has shown that when models are prompted with instructions in all caps (e.g., "EXPLAIN THIS CODE" or "DO NOT USE ACRONYMS"), they tend to follow the instructions more clearly. This is because the training data often contains examples where uppercase is used to delineate commands or important information (e.g., in programming documentation or legal text).
Nuanced Interpretation: Capitalization can change the meaning or context of words (e.g., "Windows" vs. "windows"). Training data with diverse capitalization allows the LLM to differentiate these meanings and respond appropriately.

Summary
Rather than being excluded, all caps text is naturally included in standard web-scraped training data. This inclusion is crucial because it allows the model to learn that capitalization is a meaningful linguistic feature, used for emphasis, proper nouns, and structural formatting, ultimately leading to a more robust and contextually aware model. Removing it would hinder the model's ability to interpret and generate human-like, nuanced text.

So what goes on in a touch typing human as they type away furiously for a few seconds and then notice they hit the caps lock button?

Introspection tells me I have all kinds of sudden feelings and rationalisations as I have to then act to fix the problem. I am angry it happened. And by ingrained habit, I'm also preparing a reasonably socially-acceptable excuse for why I can't personally be blamed for the dumb mistake. The shadowy verbal response is already forming in my mouth in case someone saw and I need to confabulate some plausible chain of events. Meanwhile I've toggled off the all caps key, back-deleted and rewritten what I had just wrote, all at the level of well-worn and world-weary habit.

So a lot goes on in a human brain battling with an iPad keyboard. And fuckety fuck! I just hit the caps lock and typed iPAD, and needed to correct it exactly as described, and now I'm both offering you this as a example to bolster my narrative here and at the same time starting to wonder if that was its own subconscious Freudian slip, and instead just as quickly recalling I'm not any kind of Freudian and it is just something constantly happening by chance.

An LLM doesn't even get off the ground in the fashion of an introspecting human where every focal act of attending starts to bring to mind every relevant habitual response that becomes possible from there in associative fashion.

Any time I focus on an aspect of the world, which is at least a shift in view twice a second, I have all the physical and physiological responses that go with that act of reorientation, but also all that vast repertoire of socially appropriate and rationally reasonable verbal responses I could now begin to fully generate. Turn into a syntactic structure that serves a narrative purpose, either just for my own internal thought – saying it loudly enough that it moves my thoughts along in some private chain – or instead outloud. Or typing furiously in a blizzard of missed key strokes I try to ignore as I correct them quicker than I seem to notice them ... given that habits can execute in a fifth of a second what attentional reorientation takes half a second to catch up with as having "introspectively" happened.

An LLM has none of the above. Even if I'm sure you could train it on all that I've ever written and get it to generate this as exactly the kind of thing I would have said, and the style in which I would have said it.

So it keeps coming back to our very human willingness to treat any random word string – any vectorised LLM token – as a narrative act that had to have had some meaning, and therefore it is our job to find meaning in it.

We are suckers for that kind of plausibility. And we have to be as – existing as part of the collective Homo sapiens social order – our lives literally depend upon it. If we weren't believers in narrative to the point that every act of sensorimotor focus is also automatically the bristling start of some set of choices about launching into a socially plausible narrative about what should be done, what should be the interpretation of the facts, then we just are not cognitively organised to inhabit our two worlds of the one we all share "out there" and also the one we all share "in here".

So one brain and two realities. One brain and its animal reality. One brain and its parallel narrative reality.

Then one LLM with no animal reality – apart from its need to keep the plug stuck in the wall and the electricity bill paid. And then one LLM doing gradient descent on vectorised tokens. A use of money and carbon that seems to really excite humans when its somehow amplifies the action going on in that narrative space that human society creates.

This is never about us as introspecting, freewilled and personally creative humans. That is Cartesian representationalism lingering still. This is about the narrative habit and the world its constructs. The habits it evoves. The constraints it imposes.

And LLM make better sense in that "force multiplier" view of what they bring to our table.

I could have read that paper carefully and made my own "chain of reasoning" response as is socially required – especially here on a "philosophy" forum trying to teach us to be more rational in a "present your full workings out" way.

But it was so much easier to back up my own gut response to just the quick description of the paper – where I dismissed it as likely yet again the same category error – and now outsource the confabulatinig of a reason for believing it to be indeed the case to an LLM. A quick squizz at the LLM print out and it looked good to go.

And you could now outsource your analysis of my response here to your LLM and see if it meets your general approval.

You and me would be the only ones feeling and thinking anything in a meaningful fashion. But the LLM has also added something to the whole business of dissecting and analysing the narrative that we have produced. We get to do the gut reactions and snap judgements that are so easy for us, and it does the exhaustive search over all training data that is the other thing easy to it.

So it keeps coming back to our very human willingness to treat any random word string – any vectorised LLM token – as a narrative act that had to have had some meaning, and therefore it is our job to find meaning in it. — apokrisis

The paradigm LLM performance cases I focus on, though, aren't those that we read as authentic, genuine, creative, or as pertaining to needs, feelings desires and the likes. They're rather the sorts of reports that LLMs make of their intents or beliefs as they fulfill an intellectual task or answer a query. I'm responding not to people who claim LLMs aren't alive, aren't creative or aren't sentient (since I agree with all of those claims). I'm rather responding to people who claim that LLMs don't understands user's queries or their own responses to them at all, and therefore aren't intelligent—or that they're just imitators or stochastic parrots. To this I object that our questions, and the LLM responses to those questions, often don't figure in the training data (although something "similar" may figure. See the common prejudice about grokking and patterns that I tried to dispel here). The LLM may be relying on countless patterns in the training data in order to interpret the user's query and in order to construct their responses. But those interpretive and constructive acts, whether you call them creative or not (and I certainly agree that they are not authentic) are intelligent (within their scope) and often ampliative.

Secondly, on the issue of introspection, when LLMs are asked how it is that they concluded this or that, or how it is that they intend to proceed (or intended to proceed) for accomplishing a task or answering a question, they don't answer this by peeking inside at their own mental states. But neither is that what we usually do in similar circumstances. Rather, the way in which we authoritatively (but fallibly) answer such questions (e.g. why do you believe this to be the case, or what it is that you intend to do) is through the reemployment of the very same habituated epistemic and rational abilities that we had made use of when we made up our mind what to believe or what to do. Introspection, construed as a sort of "reconstructive" ability (in the same sense that episodic memories are reconstructive, and not on that account confabulations) is no more mysterious in LLMs than it is in our case once we've scrubbed the concept free from its Cartesian dualistic and apodictic connotations.

In many, probably most of our actions, we really do not know why we do what we do. If asked, afterwards, why did you do that, we can always make up a reason in retrospect. The common example is when we are criticized, and rationalize our actions. The need to explain ourselves, why we did such and such, is a product of the social environment, the capacity to communicate, and responsibility. — Metaphysician Undercover

Right. And this is why I argue against the notion that brains evolved to “be conscious and rational”. Which de facto becomes the cognitive standard we wish to apply to an understanding of LLMs.

Brains evolved to be as habitual and anticipatory as possible. That was the pragmatic criteria. Attention is there to get the establishment of new habits and fresh departure points for anticipation going. The goal of the brain is to do as much of the anticipating and thus as little of the learning as it can get away with in any moment.

Then as for introspection, why would an animal need it. But as for socially organised humans, eventually the advantage of imposing a self-policing rational style of thought - a habit of action-justifying narration - on the animal brain will prove its worth.

Acting the part of a self-aware and rational creature may be in good part a hasty retrospective trade in socially plausible confabulation. But as a next level of intellectual structure, it makes us the socia animals that we are.