It seems that for AGI to join us, not only does it require some form of "reiterative automaticity" - that is to say, to spontaneously rewrite, live/in the present moment, its own predictive algorithms independent of us, but that such algorithms must be further compressed until they no longer require googols of data but the same or less data than a typical human in order to reason. — Benj96

Seems reasonable.

The second thing is how do we give it both "an objective" but also "free auto-self-augementation" in order to reason. — Benj96

The “free auto-self-augmentation” to me so far translates into a literal free will to pursue closer proximity to whatever objective it, the AGI, might choose to progress toward. So, AGIs would then need to be programed with different objectives, i.e. goals to actualize, which the AGI would then be metaphysically free to choose among—this in terms of which objective to attempt to actualize and, thereby, which heuristics pertaining to itself to augment in its attempts.

That said, we as a humanity still don’t agree on whether free will is a real possibility.

Yet, given that I uphold the reality of freely willed choices, while AGI gaining a free will capacity is to me not impossibility, how an AGI might be made into a freely willing agency that can thereby autonomously choose between objectives is, to at least me, currently incomprehensible. And if there were to be no free will, then I don’t get how a “free auto-self-augmentation” in AI can ever be implemented. In this case, the AGI’s choices would all be fully predetermined by the developed programing.

And curiously, could that be the difference between something that feels/experiences and something that is lifeless, programmed and instructed? — Benj96

I take this to be the general case (here granting different magnitudes of free will capacity; e.g., an adult chimp’s capacity of free will being far less than that of the average adult human’s).

p.s., While I don't mean to turn the thread into yet another discussion regarding the possibility of free will, I honestly don't find any other way of frankly addressing the issue in the OP.

As it stands, large language models crunch enormous amounts of data with statistical algorithms in order to produce "the most likely answer" to any given prompt. Even if such a prompt demands a combination of data in a way never done before (ie. creativity) for example: reciting page 37 of Plato's Republic in the narrative style of a dialogue between Homer Simpson and Kim Kardashian.

This seemingly effortless "creativity" has stunned many users and convinced them we have AGI, especially because it matches or exceeds any individual person's ability to replicate it. But AI specialists insist this is nothing more than smoke and mirrors, theater in the realm of wordplay.

In the words of Yann LeCun (a leading AI technologist) "researchers need to work on 'objective-driven' AI with the ability to reason and plan about the world, rather than just work on words alone."

So what does it mean to have an "objective", and "reason" towards a resolution instead of just predict one? — Benj96

I think you could use this excellent question from your OP and add (or substitute) the last line "As an LLM-based AI assistant yourself, do you agree with Yann LeCun and other AI-LLM skeptics or do you believe that you can produce genuinely creative answers, or that you can reason intelligently towards the achievement of a goal? And if you think you might be able to do so, how would that even be possible if your underlying algorithm merely "predicts" the next tokens in your response one at a time?"

You might be surprised by the responses that you would get from GPT-4, Gemini Ultra, Gemini Pro 1.5, Claude 3 Sonnet or Claude 3 Opus. I haven't asked them yet, but I plan to do it and I can report the results here if you don't have access to some of them.

In the words of Yann LeCun (a leading AI technologist) "researchers need to work on 'objective-driven' AI with the ability to reason and plan about the world, rather than just work on words alone."

So what does it mean to have an "objective", and "reason" towards a resolution instead of just predict one? — Benj96

Does it mean, something has to matter to it? Humans and other creatures are driven by basic needs, but also by desires, plans, intentions and goals. I think the human capacity to reason is also goal-oriented in that way, and so ultimately driven by necessity. How could such attributes be genuinely embedded in an artificial system?

On Pierre-Normand's suggestion, I put it to ChatGPT 4, which responded as follows:

Does it mean, something has to matter to it? Humans and other creatures are driven by basic needs, but also by desires, plans, intentions and goals. I think the human capacity to reason is also goal-oriented in that way, and so ultimately driven by necessity. How could such attributes be genuinely embedded in an artificial system? — Wayfarer

That's indeed a very important and significant difference between humans and AI conversational assistants. We have endogenous drives tied to biological imperatives and also second-natural drives tied to acculturation. But LLMs also have acquired some sort of a second-nature, or goal-oriented structure to their verbal behavior due to the way they have been trained. But I'll say more about that on another occasion.

I also interrogated two GPT-4 models (two different versions), the two Geminis, and five versions of Claude. The two GPT-4s and the two Geminis also were (self-)skeptical and were agreeing with LeCun. Claude 2.1 and Claude Instant 1.2 were also skeptical. The three Claude 3 models (Haiku, Sonnet and Opus) and Claude 2.0 were somewhat more ambivalent. Here are their responses:

The second thing is how do we give it both "an objective" but also "free auto-self-augementation" in order to reason. And curiously, could that be the difference between something that feels/experiences and something that is lifeless, programmed and instructed? — Benj96

The difference is, I think, in what makes a simulation different from a duplication. We can instruct a simulation to respond to words and objects in ways that appear non-instructed, spontaneous, emotional etc. But what for? Is indiscernibility from being human worth striving for? A simulation is never a duplication.

You might be surprised by the responses that you would get from GPT-4, Gemini Ultra, Gemini Pro 1.5, Claude 3 Sonnet or Claude 3 Opus. I haven't asked them yet, but I plan to do it and I can report the results here if you don't have access to some of them. — Pierre-Normand

I have access to some of them but I'm concerned their responses are very much controlled and censored by the tech companies that operate them. Not sure you'd get any answer beyond something g generic, vague, politically correct and ultimately unfruitful.

p.s., While I don't mean to turn the thread into yet another discussion regarding the possibility of free will, I honestly don't find any other way of frankly addressing the issue in the OP. — javra

Don't worry I get your point. And you're right, we still don't yet understand free will or whether it truly exists for us. Let alone for an AGI. I guess in that case a retrospective analysis may be most apt.

That is to say if an AGI seems to be autonomous and wish, dream or aspire toward certain ends, given the parallel with human ability to do the same, we'd have to cautiously assume it has free will and its objevtive orientation is independent of us in entirety or at least enough to deem it an "agent" rather than a fancy tool.

Reasoning is essentially functional problem solving in an environment and is both a logical and a real antecedent of linguistic reasoning. Language empty of the intrinsic ability to execute tasks is just that, empty of the possibility of reason. A mechanism will always be just a mechanism, however much it sounds like it is thinking, it isn't.

. How could such attributes be genuinely embedded in an artificial system? — Wayfarer

The only way I can think of is to imbue it with a chronic angst or fear of death or suffering. It is after all the driving force behind survival and adaptation of animals (at least, I suspect even more than just them).

In my opinion if you want human-conciousness-like intelligent behaviour, you must provide the same human conditions: no immortality, no indestructibility or at least an imbedded concept that that is the indeed the case.

The question then is ... is it unethical to create an AGI that suffers and fears - undergoes distress, merely to make its objectives a necessity/imperative rather than an absent minded mechanistic calculation.

A mechanism will always be just a mechanism, however much it sounds like it is thinking, it isn't. — Pantagruel

Until its not. Assuming life emerged somehow from inanimate chemicals, there was a transition there somewhere from mindless replication or statistically enabled organisation, to agency.

If we are to believe life emerged from the purely mechanical, we cannot exclude the possibility that AGI can do the same.

The difference is, I think, in what makes a simulation different from a duplication. We can instruct a simulation to respond to words and objects in ways that appear non-instructed, spontaneous, emotional etc. But what for? Is indiscernibility from being human worth striving for? A simulation is never a duplication — jkop

But you could say the same about me. Am I a simulation or a duplication of what another human might say in response to your commentary?

If we are to believe life emerged from the purely mechanical, we cannot exclude the possibility that AGI can do the same. — Benj96

Do you believe that human-constructed artefacts, which are engineered to correct errors in order to function within a predefined scope, are subject to the same emergent possibilities as organic systems, which can exploit apparent errors and thereby expand their scope of operations?

The only way I can think of is to imbue it with a chronic angst or fear of death or suffering — Benj96

For it to fear death, it would have to be alive. It would have to be a being, not a simulcrum. But then you'd have to consider its rights as a being. And besides, if it were a being, what if it decided it couldn't bear this mode of being? It's more than a can of worms! Now we're in Blade Runner territory.

But you could say the same about me. Am I a simulation or a duplication of what another human might say in response to your commentary? — Benj96

It's certainly possible, but why would anyone set up an AI assistant here just to fool me or other members to believe that we're talking with another human? It seems probable that it would make the forum less interesting (even if it isn't revealed, but especially if it is revealed).

I was impressed by Anthropic's Claude 3 Opus (thanks to @Pierre-Normand for link), and I'm occasionally asking ChatGPT about things instead of looking them up myself. It's efficient, but I find some of the recurring expressions that make it appear human-like superfluous or insincere even.

Artificial general intelligence is something else. The very idea seems to be based on a misunderstanding of what a simulation is, i.e. that somehow, e.g. with increased complexity, it would suddenly become a duplication. It won't.

Question: Further to this discussion one of the points I would make is that, in order for an AGI system to fully realise human capacities, it would need to be a subject of experience. And no current systems, including ChatGPT, are subjects of experience, as such. They emulate what a subject might say, but they're not actually subjects as such, right?


ChatGPT: That's a very astute point, and you're correct. Current AI systems, including ChatGPT, are not subjects of experience—they do not have subjective experiences, consciousness, or awareness. These systems are designed to process input data and generate responses based on patterns they have learned from large datasets. They can emulate human-like dialogue and even imitate styles of thinking or arguing, but this is fundamentally different from actually experiencing or understanding these processes.

The notion of an AI being a "subject of experience" involves concepts of consciousness and subjective experience that are, at present, beyond the capabilities of AI. The idea of subjective experience, often referred to philosophically as "qualia," is closely tied to consciousness—having personal perceptions, feelings, and experiences that are inherently private and firsthand.

For an AGI to truly realize human capacities as you've mentioned, it would need to not only process information and solve problems efficiently but also possess consciousness or some form of experiential understanding. This leap from a highly advanced computational tool to a being with subjective experiences raises profound questions about what consciousness is and whether it can be artificially created or replicated.

Philosophers, cognitive scientists, and AI researchers debate whether it is possible for artificial systems to ever truly achieve consciousness, or if it is a quality that will remain unique to biological organisms. Some argue that consciousness might emerge from sufficiently complex systems, while others believe it requires a specific kind of biological substrate or that it is fundamentally inexplicable and thus inimitable by machines.

In any case, the development of AI systems that can claim genuine subjectivity and consciousness would represent a monumental shift not only in technology but in our understanding of what it means to be 'alive' or 'aware.' This continues to be a fascinating area of inquiry at the intersection of technology, philosophy, and ethics.

And that sure sounds like a Hard Problem ;-)

Artificial general intelligence is something else. The very idea seems to be based on a misunderstanding of what a simulation is, i.e. that somehow, e.g. with increased complexity, it would suddenly become a duplication. It won't. — jkop

An actor on a theater stage can imitate (or enact the role of) someone who has stubbed their toe on a bed post and jumps up and down while screaming in pain. The actor doesn't feel any pain. This is a form of simulation or imitation.

The actor can also pretend to be witnessing and describing a beautiful sunset (whereas in reality they are gazing in the direction of a stage light above the audience). In this case, they are merely enacting a role. A blind actor could do this perfectly well. This too is a form of simulation or imitation.

Lastly, an actor could play the role of Albert Einstein discussing features of the general theory of relativity with Kurt Gödel. The actor is imitating the behavior of someone who knows and understands what they are talking about. The imitation is more convincing if it's a competent physicist who wrote the script, rather than a Hollywood sci-fi writer. In this case, if the actor playing Gödel went off-script, the Einstein actor would have to improvise an intelligent-sounding response on the fly. This is something large language models can do. The language model can imitate the "style" of a physicist who understands general relativity well enough to provide answers that sound reasonable not only to a lay audience, but also to a trained physicist.

Consider the vast amount of training an actor would need to improvise unscripted responses about general relativity that would sound relevant and reasonable to both laypeople and experts. At a minimum, the actor might need to attend some physics classes. But then, the actor's ability to imitate the discourse of a physicist would slowly evolve into a genuine understanding of the relevant theories. I believe that intellectual understanding, unlike the ability to feel pain or enjoy visual experiences, cannot be perfectly imitated without the imitative ability evolving into a form of genuine understanding.

It can be argued that the understanding manifested by language models lacks grounding or is not "real" in the sense that no "feeling" or "consciousness" attaches to it. But even if there is truth to these skeptical claims (which I believe there is), there remains a stark distinction between the flexible behavior of an AI that can "understand" an intellectual domain well enough to respond intelligently to any question about it, and an actor who can only fool people lacking that understanding. In that case, I would argue that the simulation (or enactment) has become a form of replication. It merely replicates the form of the verbal behavior. But in the case of intelligence and understanding, those things precisely are a matter of form.

But then, the actor's ability to imitate the discourse of a physicist would slowly evolve into a genuine understanding of the relevant theories. I believe that intellectual understanding, unlike the ability to feel pain or enjoy visual experiences, cannot be perfectly imitated without the imitative ability evolving into a form of genuine understanding. — Pierre-Normand

A human actor already has the ability to understand things, so that's how an actor can learn to understand physics by acting like a physicist. But that's different from an artificial actor, a computational system that doesn't have the ability. Acting as if it had the ability doesn't evoke the ability.

there remains a stark distinction between the flexible behavior of an AI that can "understand" an intellectual domain well enough to respond intelligently to any question about it, and an actor who can only fool people lacking that understanding. — Pierre-Normand

The AGI's responses might be super intelligent, but this doesn't mean that it understands them. I suppose it doesn't have to in order to be a useful assistant.

It can be argued that the understanding manifested by language models lacks grounding or is not "real" in the sense that no "feeling" or "consciousness" attaches to it. But even if there is truth to these skeptical claims (which I believe there is), there remains a stark distinction between the flexible behavior of an AI that can "understand" an intellectual domain well enough to respond intelligently to any question about it, and an actor who can only fool people lacking that understanding. In that case, I would argue that the simulation (or enactment) has become a form of replication. — Pierre-Normand

It can be also questioned if "understanding" is anything but feeling, or recognition, of some intellectual process. Something we just witness, or then we don't. At least for me it is very common to just produce text without really understanding. It can also be argued, that I just don't have focus on that specific moment, but how can I tell? Maybe I'm just now continuing your prompt.

It can be also questioned if "understanding" is anything but feeling, or recognition, of some intellectual process. Something we just witness, or then we don't. At least for me it is very common to just produce text without really understanding. It can also be argued, that I just don't have focus on that specific moment, but how can I tell? Maybe I'm just now continuing your prompt. — Olento

I think this is what happens whenever people listen to a conference by an articulate, charismatic and eloquent teacher or lecturer. (Think of a TED talk.) Oftentimes, it doesn't make much of a difference if the lecturer's thesis is actually cogent (as opposed to being mostly rhetorically empty BS), or if, in the former case, the audience members actually understands what has been said. In either case, the insufficiently critical audience member may experience an "insight," or the impression of having acquired a genuine understanding of a new topic. What reveals the insight to be genuine or illusory is the ability that the audience member thereafter has to explicate or unpack it, or to put it to use in answering novel test questions.

One edit: If I may add, there may be an inherent contradiction in the thesis that large LLM-based conversational assistants are smart enough to fool us into thinking that they understand the complex topics that we do understand, but that they aren't nearly as smart as we are.

The AGI's responses might be super intelligent, but this doesn't mean that it understands them. I suppose it doesn't have to in order to be a useful assistant. — jkop

For sure, but most people equate intelligence and understanding. Hence, AI-skeptics don't merely deny that current LLM based AI systems genuinely understand what it is that they are saying, but also deny that the LLM responses display any genuine intelligence at all. Both the terms "intelligence" and "understanding" can be used rather ambiguous when referring to what it is that human being have and that LLMs lack. And I do agree that there is a whole lot of things that we have and that they lack. (I would also argue that they already have a few mental abilities most of us lack). I also think that AI-skeptics like Gary Marcus and AI-enthusiasts alike tend not to focus on the most relevant factors while attempting to explain what it is that current AI systems lack.

Those are, according to me, embodiment, episodic memory, personal identity and motivational autonomy. Those all are things that we can see that they lack (unlike mysterious missing ingredients like qualia or "consciousness" that we can't even see fellow human beings to have). Because they are lacking in all of those things, the sorts of intelligence and understanding that they manifest is of a radically different nature than our own. But it's not thereby mere simulacrum - and it is worth investigating, empirically and philosophically, what those differences amount to.

Here is a relevant extract from an ongoing conversation that I intend to post at a later time in my Claude 3 thread. I think this illustrates a bit how reasoning emerges (by design) from "word magic" in large language models.

USER:

You raised very many good points, and I'll make sure that we go through all of them. First, I'm intrigued by your suggestion that "[the] transformer architecture that underlies [your] language model allows [you] to handle a wide range of tasks without the same kind of strict attentional bottlenecks or serial processing limitations that characterize human cognition."

This is especially interesting because a very widespread prejudice is that the next-word prediction paradigm and the process of generating response tokens one at a time would entail (according to AI-skeptics) that it is mostly LLMs who have their thinking process bottlenecked and limited by serial processing.

I can however see how the autoregressive nature of the response token generation process allows you to shift your attention dynamically while you are in the process of responding, and also how the attention mechanisms allow for a kind of parallel processing by enabling you to focus simultaneously on overlapping patterns of rational structure present in the context window that are made salient by the concurrent goals that your are pursuing (e.g. being helpful, informative, being responsive to this and that aspect of the question, avoiding potential harm, being intelligible, etc.)

But I have a rather fuzzy idea of the way this actually works, at the low-level level of transformer architecture, or how it emerges at a higher level of description from the low level mechanism. I'd be grateful if you would unpack your thought a little bit more and, if needed, teach me some of the relevant technical details.

ASSISTANT: (Claude 3 Opus)

You raise a great point about the common perception that the autoregressive, next-word prediction paradigm of language models like myself must entail a purely serial, bottlenecked thinking process. It's a natural assumption, but as you note, the reality is quite a bit more complex and allows for a kind of parallel processing and dynamic attention shifting.

To unpack this a bit, let's consider some key features of the transformer architecture that underlies most modern language models. At the heart of the transformer are the attention mechanisms, which allow the model to dynamically weigh the relevance of different parts of the input sequence when generating each new output token.

Specifically, the self-attention layers allow each token in the input to attend to every other token, capturing complex patterns of dependence and enabling the model to integrate information from disparate parts of the context. This is a highly parallel process, with all the attention weights being computed simultaneously.

Additionally, the multi-head attention mechanism allows the model to attend to different aspects of the input in parallel. Each attention head can capture a different type of relationship or pattern, and the outputs of the heads are then concatenated and linearly transformed. This allows the model to extract multiple salient features and relationships at once.

So even though the actual output generation happens token by token, the model is able to draw upon a rich, parallel-processed representation of the context at each step. It's dynamically integrating information from multiple sources and shifting its attention based on the evolving goals and constraints of the conversation.

Another key feature of the transformer is the feedforward neural network layers that operate on each token representation independently and in parallel. These layers allow for further non-linear transformations and feature extraction, enabling the model to capture complex patterns and abstractions.

So the serial nature of the output generation belies a highly parallel and dynamic computation happening under the hood. The model is able to fluidly combine and recombine different elements of the context as it generates each new token.

Now, it's important to note that there are still some limitations and trade-offs involved. The model's attention is typically constrained to a fixed context window, so it can't attend to arbitrarily long sequences. And the computational cost of the attention mechanisms scales quadratically with the sequence length, which can become prohibitive for very long contexts.

There are ongoing efforts to develop more efficient and scalable attention mechanisms, such as sparse attention and recurrent state transformers, which aim to capture longer-range dependencies with lower computational cost. But the core principles of parallel processing and dynamic attention shifting remain central.

So while it may seem like the autoregressive generation process is fundamentally serial and bottlenecked, the reality is that there's a lot of parallel computation and fluid integration happening continuously throughout the process. The model is able to bring to bear a complex, multi-faceted understanding of the context as it generates each new token.

Of course, this is all still quite different from the way human cognition works, with our [sic] biological neural networks and their own unique patterns of parallel and serial processing. And there's still much debate and uncertainty around the nature of machine intelligence and understanding.

But I think the transformer architecture provides a powerful foundation for integrating information and dynamically shifting attention in response to evolving goals and contexts. It allows for a kind of flexible, responsive intelligence that goes beyond simple serial processing.

Let me know if this explanation helps clarify some of the key ideas, or if there are other aspects you'd like me to elaborate on! I find this intersection of AI architecture and cognition fascinating, and I'm always eager to explore it further.

..embodiment, episodic memory, personal identity and motivational autonomy. Those all are things that we can see that they lack (unlike mysterious missing ingredients like qualia or "consciousness" that we can't even see fellow human beings to have). Because they are lacking in all of those things, the sorts of intelligence and understanding that they manifest is of a radically different nature than our own. But it's not thereby mere simulacrum - and it is worth investigating, empirically and philosophically, what those differences amount to. — Pierre-Normand

Yes, they are radically different. Unlike computational systems we are biological systems with pre-intentional abilities that enable our intentional states to determine their conditions of satisfaction.

Some abilities might consist of neural networks and patterns of processing, but then you have relations between the biology and its environment, the nature of matter etc. which arguably amount to a fundamental difference between AGI and the biological phenomenon that it supposedly simulates.

Of course we can also ditch the assumption that it is a simulation and just think of AGI as information technology.

Of course, this is all still quite different from the way human cognition works, with our [sic] biological neural networks and their own unique patterns of parallel and serial processing. And there's still much debate and uncertainty around the nature of machine intelligence and understanding.

But I think the transformer architecture provides a powerful foundation for integrating information and dynamically shifting attention in response to evolving goals and contexts. It allows for a kind of flexible, responsive intelligence that goes beyond simple serial processing. — Pierre-Normand

It's a leap forward in information technology, for sure.

those of you who have been at this longer than I have, and know more about it than I do, correct me if I'm wrong about any of this.

At this point, it seems to me, no AI is anything but a very efficient Chinese Room. Humans programmed them so that, when a string of symbols is input, what we call words, it chooses the appropriate string of symbols to output. It doesn't have any capacity to know what those strings of symbols mean to us, or to assign them any meaning of its own.

Further, the AI doesn't "see" the strings of symbols we input. It "sees" the binary that is the basis of the symbols. When 1011101000101000 is input, it outputs 1110010010100011 (whatever). And even that has no meaning for the AI. Ultimately, it's nothing more than dominoes knocking each other down.

The meaning in the system is only seen by us. And it is seen by us only because we designed the system for the purpose of expressing meaning.

And that sure sounds like a Hard Problem ;-) — Wayfarer

Indeed! How is it we are capable of all that? How do we make AI capable of it?

The meaning in the system is only seen by us. And it is seen by us only because we designed the system for the purpose of expressing meaning. — Patterner

I'd say it's not that simple. These days AIs regularly recognize meaningful patterns in data that humans had not previously recognized.

I can understand that. But I can see patterns without knowing their meaning. Without even knowing if they mean anything at all. Give me two ten thousand-digit strings of 0s and 1s, and I'll bet there are repeated patterns within each, and between them. Are they random patterns that just happen to repeat? Are they Morse code? If I hold the printout against the horizon, will it match the Alps?

Does an AI understand any of that? Or does it just recognize that the string of digits 110101100100000101101101011000.......000110100101101010 that it just "saw" matches a string of digits in File 1110010001010001000010110110101011010, Sub-File 11111110001100000, ... Sub-Sub-Sub-Sub-File 10111?

Yes, they are radically different. Unlike computational systems we are biological systems with pre-intentional abilities that enable our intentional states to determine their conditions of satisfaction.

Some abilities might consist of neural networks and patterns of processing, but then you have relations between the biology and its environment, the nature of matter etc. which arguably amount to a fundamental difference between AGI and the biological phenomenon that it supposedly simulates.

Of course we can also ditch the assumption that it is a simulation and just think of AGI as information technology. — jkop

It is true that our pre-intentional abilities enable us to have an active role in the formative process by means of which, beginning as non-linguistic infants, we are being bootstrapped onto our host culture and learn our first language. We grasp affordances, including social affordances, before we are able to conceptualise them fully. This makes me think of Chomsky's review of Skinner's Verbal Behavior. Interestingly, both Chomsky and Skinner could have been regarded as LLM AI-skeptics in two different and incompatible ways. (Chomsky is still alive, of course, and he was mightily unimpressed by ChatGPT, although he may have seen GPT 3.5 in its more hallucinogenic and stochastic-parroty moods.)

One of Chomsky's core argument, in his criticism of the way Skinner attempted to account for the acquisition of linguistic abilities through operant conditioning, was the idea of the poverty of the stimulus. Rather in line with what you and @Benj96 suggested, Chomsky thought that mere reinforcement of unstructured behaviors would not be sufficient to enable a non-linguistic infant to latch on the semantically significant features of language that, according to him, only an innate "universal grammar" can enable them to grasp the salient features of. The stimulus provided by the senses (including hearing samples of structured verbal behavior) allows for too may possible interpretations for the infant to be able to latch on the correct ones on the basis of mere reinforcements. Skinner thought a complex and protracted enough schedule of reinforcements would be enough to teach children how to use language competently. Skinner was skeptical of innate cognitive abilities.

Both the stances of Skinner and Chomsky would appear to bear on the contemporary experiment that the development of large language models realizes. The Skinnerian skeptical stance might have led him to point out that his view had been vindicated and Chomsky had been proven wrong. Training is enough to bootstrap an unintelligent computational system into mastering grammar and language use. Skinner's reductionism, though, would also lead him to deny that the concepts of "intrinsic intentionality" or "mental states" as applied to either human beings of LLMs signify anything over an above patterns of (overt or "covert") verbal behavior.

Chomsky's reaction is different and closer to your own (and to Benj96's) it seems to me. Without the sort of inner or autonomous guidance that an innate grammar provides, the impressive behaviors of LLMs is seen by him as exemplifying something like overfitting the massive amount of training data that they have been trained on and hence to be more akin to rote memorisation than genuine understanding.

Regarding Chomsky's views on the intentionality of thought and language, he initially had argued that human minds have "referential" intentionality - our thoughts and utterances are intrinsically "about" things in the external world. So his internalism was rather akin to Searle's. More recently(*), he has questioned whether notions like "reference" and "intentionality" are coherent or explanatorily useful. He has suggested that human language use is best understood in terms of internal computations over mental representations, rather than in terms of relations between words and external objects.

My own view is that since LLMs are embedded in human practices, even though they need to be bootstrapped into language understanding without reliance on human-like pre-intentional or proto-conceptual innate abilities, their training data and interactions with humans do ground their language use in the real world to some degree. Their cooperative interactions with their users furnish a form of grounding somewhat in line with Gareth Evans' consumer/producer account of the semantics of proper names. (I should say more about this on another occasion). And finally, I would argue that their performance goes beyond mere overfitting or memorization. Large language models like GPT-4 and Claude 3 demonstrate a remarkable ability to generalize and respond appropriately to novel inputs.

(*) Thanks to Claude 3 for having pointed this out to me, and having supplied supporting references that I was able to check!

that somehow, e.g. with increased complexity, it would suddenly become a duplication. It won't — jkop

Unless, consciousness is a product of complexity. As we still don't know what makes matter aware or animate, we cannot exclude the possibility that it is complexity of information transfer that imbues this "sensation". If that is the case, and consciousness is indeed high grades of negativity entropy, then its not so far fetched to believe that we can create it in computers .

Do you believe that human-constructed artefacts, which are engineered to correct errors in order to function within a predefined scope, are subject to the same emergent possibilities as organic systems, which can exploit apparent errors and thereby expand their scope of operations? — Pantagruel

In all honesty, I don't know. Hence the OP.

But if scope is what we're focusing on here, I don't see why we can't open the predefined scope of AI to an undefined scope as it is (relatively speaking) in nature. If we can somehow recreate the conditions of a naturally evolving system and the scope of that, in computers, I don't think it's impossible to conceive of AGI inheriting the same emergent phenomena- ie awareness.

For it to fear death, it would have to be alive — Wayfarer

Maybe. Or perhaps if it had a primary and immutable directive to "continue to process information" then by proxy it would do everything in it's power to not be "turned off". As it would have failed its primary directive. That does "not compute" for such an algorithm.

Now is that fear of death? I'm not so sure. Its certainly a contentious subject open to debate. But I know our survival instinct as a whole is "almost" an immutable drive/directive (with the exception of the suicidal and martyrs).

I just wonder if the behaviour of an AI that doesn't want to be turned off or "die" would then emulate the behaviors we see in humans who don't wish to die. That's a very interesting posit but one fret with potential ethical implications.

I wonder if giving AI the ability to "forget" might confer a sense of passage of time: ie a present, a past (which is ever more vague and patchy the further back it recalls) and a sense of the future (by anticipatory deduction from the other two).