For anyone that thinks computers are (or someday will be) conscious, what do you say to Bernardo Kastrup's argument here:
"You see, everything a computer does can, in principle, be done with pipes, pressure valves and water. The pipes play the role of electrical conduits, or traces; the pressure valves play the role of switches, or transistors; and the water plays the role of electricity. Ohm’s Law—the fundamental rule for determining the behavior of electric circuits—maps one-on-one to water pressure and flow relations. Indeed, the reason why we build computers with silicon and electricity, instead of PVC pipes and water, is that the former are much, much smaller and cheaper to make. Present-day computer chips have tens of billions of transistors, and an even greater number of individual traces. Can you imagine the size and cost of a water-based computer comprising tens of billions of pipes and pressure valves? Can you imagine the amount of energy required to pump water through it? You wouldn't be able to afford it or carry it in your pocket. That’s the sole reason why we compute with electricity, instead of water (it also helps that silicon is one of the most abundant elements on Earth, found in the form of sand). There is nothing fundamentally different between a pipe-valve-water computer and an electronic one, from the perspective of computation. Electricity is not a magical or unique substrate for computation, but merely a convenient one. A wooden tool called an 'abacus' also computes.
With this in mind, ask yourself: do we have good reasons to believe that a system made of pipes, valves and water correlates with private conscious inner life the way your brain does? Is there something it is like to be the pipes, valves and water put together? If you answer ‘yes’ to this question, then logic forces you to start wondering if your house’s sanitation system—with its pipes, valves and water—is conscious, and whether it is murder to turn off the mains valve when you go on vacation. For the only difference between your house’s sanitation system and my imaginary water-based computer is one of number—namely, how many pipes, how many valves, how many liters of water—not of kind or essence. As a matter of fact, the typical home sanitation system implements the functionality of about 5 to 10 transistors.
You can, of course, choose to believe that the numbers actually matter. In other words, you may entertain the hypothesis that although a simple, small home sanitation system is unconscious, if you keep on adding pipes, valves and water to it, at some point the system will suddenly make the jump to being conscious. But this is magical thinking. You'd have to ask yourself the question: how, precisely, does the mere addition of more of the same pipes, valves and water, lead to the magical jump to conscious inner life? Unless you have an explicit and coherent answer to this question, you are merely engaging in hand waving, self-deception, and hiding behind vague complexity."
https://www.bernardokastrup.com/2023/01/ai-wont-be-conscious-and-here-is-why.html