Can the proposed danger be said to be a real concern to the science community if knowledge development continues seemingly full speed ahead in every field?

That said, I am agreeable to relating to scientists as one would a highly skilled car mechanic. If the mechanic does the job you're paying him for, we applaud, and don't expect them to be responsible for air pollution. — Hippyhead

I’m not sure the analogy is accurate. We pay a mechanic to fix our car, but in all honesty most of us have no clue and no way of determining whether he’s solved the problem or just making it seem like the problem is solved. The same goes for doctors. We are entrusting aspects in our relation with the world to their knowledge and expertise. It’s risky. But if a mechanic, in fixing one problem with our car, causes another, then we’re not going to applaud him for doing the job we paid him for, are we? We don’t expect him to take responsibility for the whole car, but we do expect him to be responsible for his actions in relation to the whole car.

Ethics is an important aspect of scientific development, but it’s also an important aspect of interpreting and communicating scientific knowledge - it’s here that no-one seems to be taking responsibility. The scientific method includes interpreting and making conclusions from the data, but too often this bit is left to science journalists, whose short-term goal is consumption of information. So they will qualitatively structure that information in a way that increases consumption.

The problem is that science is content with that, because they can’t agree on the qualitative structure of the information they have in relation to our experience of reality. So they put it out there as purely quantitative information or data, inviting everyone to interpret it in their own way and for their own purposes. It’s like a mechanic telling you what’s wrong with your car, and then giving you the parts that need replacing.

Ethics is an important aspect of scientific development, but it’s also an important aspect of interpreting and communicating scientific knowledge - it’s here that no-one seems to be taking responsibility. The scientific method includes interpreting and making conclusions from the data, but too often this bit is left to science journalists, whose short-term goal is consumption of information. So they will qualitatively structure that information in a way that increases consumption. — Possibility

Great post, thanks!

Would it be accurate to say that everyone involved is acting intelligently and professionally within their narrow lane, and some talk about the big picture, but nobody is responsible for the big picture? So this huge very intelligent knowledge machine keeps grinding on blindly towards the cliff.

Perhaps nobody is responsible for the big picture because we assume that no one could be, which seems a reasonable theory. If true, then we aren't really in charge, right? Should we reconfigure our view of this from "we are developing knowledge" to "knowledge is developing us"?

Sometimes I think of knowledge as another element of nature, like water, air, space, atoms etc. We've wandered in to a knowledge hurricane and don't know how to find our way out. Or more precisely, we typically don't realize that hurricanes are dangerous?

You might be assuming that but I'm not — Srap Tasmaner

Ok, continue.

I thought your idea of their being two different rates of change was spot on, and very close to what others have said. Given that, we can demand that the new tech demonstrate first, or as it reaches development milestones, whatever, demonstrate that it can be controlled. — Srap Tasmaner

Seems a reasonable step in the right direction. So, as example, before genetic engineering can proceed we'd need some kind of binding global authority which controls funding, or can implement punishment for violators? Something like that? If yes, then this is essentially a political problem, a failure to work together?

This is exactly what did not happen with nuclear weapons — Srap Tasmaner

It can be argued that nuclear weapons have been successfully controlled by MAD, but wow, the potential price tag for such a system is huge, almost beyond comprehension.

If we block their use entirely, we miss out on a good. Are we to have a Ministry of Technology that would approve uses, and police and strictly control their distribution? I'm not in love with the idea, but maybe it's necessary. — Srap Tasmaner

But it has to be a global Ministry of Technology, right? This may raise some interesting questions. What could science contribute, if anything, to the development and effectiveness of global governing systems? If we aimed half of the scientists at that, we might be getting somewhere. Very vague idea so far for sure, but better than nothing?

A key challenge I see is that science culture will very reasonably argue that the current "free for all" status quo has brought enormous benefits. Which is true! And so they will dazzle the public by dangling even more benefits under their eyes. "We are on the verge of curing cancer" etc. I predict it will be enormously challenging to get science culture to embrace any kind of meaningful limits.

Another possible angle....

The problem is not knowledge, or even power, but rather the gap between power and our maturity, or rather relative lack thereof. If science could close this gap by somehow accelerating our maturity to match the demands that will increasingly be placed upon it, in theory that could be a solution.

I don't have the slightest idea how that might actually work, but at least the theory may demonstrate that it is not knowledge which is the enemy, but the gap.

2) Knowledge often delivers power to edit our environment, which is typically why we seek it. — Hippyhead

Another possible angle....

The problem is not knowledge, or even power, but rather the gap between power and our maturity, or rather relative lack thereof. If science could close this gap by somehow accelerating our maturity to match the demands that will increasingly be placed upon it, in theory that could be a solution. — Hippyhead

First of all, science tells how the World is, not how it should be. It doesn't tell us which things are right and which things are wrong. Science isn't normative. We tell that to ourselves. Those normative questions are truly important, yet totally separate from using the scientific method to picture the World around us.

Unfortunately too many people think that "Science" or the "scientific" way of thought will tell us what is good and what is bad. It doesn't go so. Or then anti-scientism portrays science and the scientific community as a perpetrator of bad things: that because physicists created nuclear weapons, science is nearly evil. Again, science and use of technology are two different things.

First of all, science tells how the World is, not how it should be — ssu

Right. We say how the world should be. And then we hire scientists to help work on making it so.

Then they aren't actually making science. No really, science and implementation of a technology should be two separate things. Science and technology aren't synonyms, even if we do us quite loosely the term "scientific".

. No really, science and implementation of a technology should be two separate things. — ssu

Tell it to Oppenheimer?

Would it be accurate to say that everyone involved is acting intelligently and professionally within their narrow lane, and some talk about the big picture, but nobody is responsible for the big picture? So this huge very intelligent knowledge machine keeps grinding on blindly towards the cliff.

Perhaps nobody is responsible for the big picture because we assume that no one could be, which seems a reasonable theory. If true, then we aren't really in charge, right? Should we reconfigure our view of this from "we are developing knowledge" to "knowledge is developing us"?

Sometimes I think of knowledge as another element of nature, like water, air, space, atoms etc. We've wandered in to a knowledge hurricane and don't know how to find our way out. Or more precisely, we typically don't realize that hurricanes are dangerous? — Hippyhead

Who said we were in charge? We are developing knowledge, but this knowledge consists of our relationship to information as part of information’s relationship to each other - of which we are also a part. In a way, we are developing our relationship to information, as information. As complicated as it sounds, this makes more sense to me.

But the ‘big picture’ seems too big now for an individual mind - I think this has been our main issue for some time. Big picture thinking is rare, and seems to be valued far less than specialised knowledge - this is evident in education, employment and modern leadership structures. Quantitatively, we know more and more, but qualitatively, so much of that knowledge is ignorant, isolated and excluded from each other.

It’s a repeat of the Tower of Babel: so much knowledge, but everyone speaking different ‘languages’, unable to understand each other, scattered and dis-organised. It was never about how high we could build this tower of information - a one-dimensional relation to ‘truth’ value - but about a complex relational structure inclusive of humanity that maximises awareness, connection and collaboration with all possible existence.

It is out of fear that we ignore, isolate and exclude information that requires us to dismantle and re-structure our consolidated satellites of knowledge. We draw arbitrary boundaries where physics blends into chemistry and chemistry into biology, and we say ‘stay within your lane’ with clear funding and industry segmentation. It’s no wonder the big picture is out of our grasp.

Our view of scientific knowledge as a commodity is pushing the quantitative slant. Scientists seek to understand quantum-level relational structure, but funding and media attention focuses on the consolidation of a ‘god-particle’; they seek to understand the relational structure of disease, but funding and media attention push for a consolidated cure. Preferential treatment is given to the objects, products and numbers, not the relational structures that create them - the particle, not the wave.

The hurricane is only dangerous because we didn’t predict it - we’ve wandered into it because we weren’t paying attention to the relational structures that have been forming it around us. We’ve looked up from focusing on our individual, specialised tasks and thought, “Crap - how did this happen?”

Who said we were in charge? — Possibility

Do we want to be in charge? Or are we content to ride on this train where ever it is going, even though the most logical outcome is likely some form of disaster?

But the ‘big picture’ seems too big now for an individual mind - I think this has been our main issue for some time. — Possibility

Yea, that seems reasonable. And it's true, big picture thinking is not well rewarded.

We draw arbitrary boundaries where physics blends into chemistry and chemistry into biology, and we say ‘stay within your lane’ with clear funding and industry segmentation. It’s no wonder the big picture is out of our grasp — Possibility

Ok, yes, specialization is both key to the knowledge explosion, and a primary form of blindness.

We’ve looked up from focusing on our individual, specialised tasks and thought, “Crap - how did this happen?” — Possibility

I must agree this is very understandable. After all, a "more is better" relationship with knowledge was perfectly sensible for our entire history, until quite recently. Given enough time, and enough pain, I'm sure we could adjust to the new reality. But we may not have much time, and the pain can now be fatal.

I think a key issue is the scale of the emerging powers.

With small powers one makes small mistakes and so can clean up the mess and correct the course.

With large powers one mistake can be game over, removing the ability to learn and adapt. With every day that passes the room for error is quietly shrinking, shrinking, shrinking, arguably at an ever quickening pace.

Do we want to be in charge? Or are we content to ride on this train where ever it is going, even though the most logical outcome is likely some form of disaster? — Hippyhead

Are they the only two choices? Or do you think we can work instead towards maximising awareness, connection and collaboration? Of course, on the surface it would appear remarkably similar to being in charge. One noticeable difference is humility.

I must agree this is very understandable. After all, a "more is better" relationship with knowledge was perfectly sensible for our entire history, until quite recently. Given enough time, and enough pain, I'm sure we could adjust to the new reality. But we may not have much time, and the pain can now be fatal.

I think a key issue is the scale of the emerging powers.

With small powers one makes small mistakes and so can clean up the mess and correct the course.

With large powers one mistake can be game over, removing the ability to learn and adapt. With every day that passes the room for error is quietly shrinking, shrinking, shrinking, arguably at an ever quickening pace. — Hippyhead

You’re approaching this with a lot of fear - in my view, it is fear that removes the ability to learn and adapt much more than mistakes.

But I personally don’t see the scale of emerging powers as such a key issue. Those ‘large’ powers are not as consolidated as you might think. They consist of relations between relational structures - even if they appear to be individual, autonomous and ‘in charge’. For all the posturing of nuclear weaponry, there’s a reason why ‘that button’ has never been pushed: no man is an island. The ‘larger’ that power seems, the more relational structures hold it in place. And when those relational structures perceive their own potential destruction, they will consolidate against it, and that ‘power’ will shrink.

In this way, we have remained just this side of non-existence, in a dynamic non-equilibrium that always looks more frightening than it is. Many of us are content to ride this train to wherever, while others are fighting to be ‘in charge’.

Tell it to Oppenheimer? — Hippyhead

Ooh, those evil diabolical physicists! I mentioned them earlier already. Still, we use the term "technology" rather than "science" when talking about nuclear weapons.

But seriously,

Where your problem arises is in a science which has a normative agenda and obviously does want to engineer reality and build a better World: medicine. Medicine, the establishing of a diagnosis, a prognosis, a treatment, and the prevention of disease is viewed as a science, not a technology. To prevent people from dying to diseases is obviously such an universal morally correct objective that here science does have an agenda. And the "Death Trap", if one dares one to call it, is of course that our understanding is limited and our actions might be counterproductive and even lethal, even if the agenda we have is to save lives.

A small anecdote how easily this can happen. When in the 1950's American doctors and researchers came to Papua New Guinea, they noticed that the indigenous tribes didn't wash themselves much and hence had the great idea of introducing soap and bathing to them. The tribal people began to use soap with the consequence that some of them died. The simple reason was that the layer of filth on the skin protected from the various critters and protozoans etc. that are found in the jungle. This actually didn't harm the relations much, as the people assumed that the foreign doctors had made the people as a ritual offering as other medicine did cure them. (This story was told to my father, a professor of virology, by a redeemed medical researcher that had been there in the 1950's and 1960's.)

We naturally have a multitude of examples starting from Medieval medicine how our ignorance and limited knowledge has put us on the wrong path and in my view modern psychiatry is still hopelessly lost as we don't understand the brain and sentience and what we term a "mental disease" is quite arbitrary and defined by societal norms.

4) An ever accelerating rate of knowledge development results in an ever accelerating development of new powers.

5) Thus, science gives human beings new powers at an ever accelerating pace. — Hippyhead

These assumptions are wrong actually. First of all, the endless exponential grows doesn't exist in nature, but it's quite widespread delusion. People have a limited mental capacity to grow and to acquire knowledge and we are reached the limit in many areas already.

Secondly, science doesn't exists separate from people. If people can't fully understand science then powers will decay. So here we have quite different picture but I can't say that it less dangerous than technological singularity. Best regards to Asimov

First of all, the endless exponential grows doesn't exist in nature — Skeptic

Once we invent computers, then research in most other fields can then proceed at a substantially faster pace. Once we invent CRISPR, genetic research and engineering can then proceed at a substantially faster pace. Once we invented the printing press, the Enlightenment could proceed at a substantially faster pace. Once we invented writing... Once we invented language... Etc.

What we've learned just in my lifetime likely dwarfs what was learned in a couple of centuries previous.

All that said, you're right that exponential growth is not limitless in nature. Sooner or later it hits the wall and crashes. Just like I started this post to explain.

The only hope is that wisdom will keep pace with scientific knowledge. Wisdom before knowledge and wisdom before power, otherwise there is great danger.

The only hope is that wisdom will keep pace with scientific knowledge — EnPassant

Agreed. Either wisdom has to be dramatically accelerated somehow, or knowledge had to be slowed down dramatically, or some combination of the two.

There is a process which can accomplish this. It's called pain. Disaster. Calamity. That's how we typically learn big stuff.

Sooner or later it hits the wall and crashes. — Hippyhead

I tried to say that there are more possible scenarios and actually it isn't a big deal to cover all of them. More importantly, some of them contains quite bold hints for your question.

The only hope is that wisdom will keep pace with scientific knowledge — EnPassant

It is a pity that wisdom is not inherited

Either wisdom has to be dramatically accelerated somehow, or knowledge had to be slowed down dramatically, or some combination of the two. — Hippyhead

Science by itself won't provide wisdom. Something else is needed.

It is a pity that wisdom is not inherited — Skeptic

So it needs to be found somehow...

So it needs to be found somehow — EnPassant

It's at least conceivable, in theory, that by genetic engineering we could redesign ourselves in to a more intelligent species. But it would be semi-suicidal near cave man idiots doing the redesigning.

Ughh.... Hit head with big stick. See what happens!

by genetic engineering we could redesign ourselves in to a more intelligent species — Hippyhead

wisdom is based on experience, not only on genetics. It means that in any given time in human society will be a significant number of people without wisdom and there is not way to avoid that (at least, without losing human nature). That was the first hint...

It means that in any given time in human society will be a significant number of people without wisdom and there is not way to avoid that (at least, without losing human nature) — Skeptic

Yea, that's what I was referring to, losing human nature. Crafting a new species. I agree there is likely nothing at all realistic about such an idea.

Crafting a new species. I agree there is likely nothing at all realistic about such an idea. — Hippyhead

I'm talking about a bit different thing. Crafting a new species isn't a big deal, but even in such case you will still have the same issue. Personal experience will still be there. The only way to avoid the trap of wisdom absence is to remove personal experience at least partially. Partial case will look like a programmable personalities, complete case will look like a collective mind. Both are too far from human nature and ability to choose own behavior.

So the only way for humans is to find a way to live with it. There is no way to evolve and forget this problem.

So the only way for humans is to find a way to live with it — Skeptic

And die with it. You are probably right.

And die with it — Hippyhead

Probably, but there are other scenarios too. Previously I tried to show the set of scenarios without death.

All that said, you're right that exponential growth is not limitless in nature. Sooner or later it hits the wall and crashes. Just like I started this post to explain. — Hippyhead

Or simply the advances just slows down and in the end state the technology doesn't change.

A wide variety of technology has simply found an efficient state that there isn't much need to change. Just take one of the most loved technical gadgets Americans love and nearly worship: hand held firearms. Handguns have the same mechanism as in start of the 20th Century and even military rifles are in general the same as after the 1950' or 1960's. Even older technology are books. They surely have not changed in technology for a very long time and still are being used even if phone books and dictionaries aren't popular anymore (for obvious reasons).

Many things we have now might be in use in the distant future. Likely a computer (of some sort) can read .doc -files even a hundred years from now. Hence technology doesn't crash, it just keeps the same. Sometimes for Centuries.

It's not just that we have nuclear weapons, we are bored by them. What can that be called other than madness?

After experiencing the Cuban Missile Crisis I would call it "Progress".

I think you need to have greater faith in the "Silent Majority", the Philosophy of science and technological progress. In todays world these are the social mechanisms that promote the common good and limit the power of a "childish minority".

Or simply the advances just slows down — ssu

But the advances have already reached the point where a single bad day can equal game over. We aren't really talking about a speculative future so much as we are the current reality.

Sorry to be brief! Engaging on too many fronts, my bad.

I think you need to have greater faith in the "Silent Majority", the Philosophy of science and technological progress. — Chris1952Engineer

I will admit to having reservations about attempts to pop the delusional dream bubble we are living in. If nothing can be done, perhaps better to let folks enjoy their sleep. Seriously.

I can understand your reservations and your reluctance to effect change in a larger Reality we are all subject to. But I cannot understand your fatalism. We are all "engineers" in a very real sense, driving change on a daily basis that affects others and the world around us just by surviving and following our own dreams. We are inherently doing our part as a member of the "Silent Majority".

Have faith in us.
I do.

But I cannot understand your fatalism — Chris1952Engineer

Would it be fatalism to recall that the Roman Empire thrived and dominated for centuries and then collapsed in to a thousand years of chaos and ignorance? Hasn't every civilization ever constructed eventually collapsed?

Aren't the means for a possible almost instant collapse of our civilization standing by ready for a single human being to press a button? Isn't that a fact? And isn't it true that generally speaking we are bored by that fact? Isn't it true that on a number of occasions we've come within a whisker of Biblical scale calamity?

Meaning no offense, and claiming no perfect knowledge of the future, I don't understand what your faith is based upon. Surely not logic or facts?

And the other question. Assuming for just a moment that my concern is valid, and assuming that neither of us are in a position to do anything about this, should those assumptions be true, should I be pulling back this curtain?

During the Carter administration somebody mistakenly placed a training tape in to the NORAD main computer, and for precious minutes the entire chain of command all the way up to the National Security Advisor were convinced a Soviet first strike was incoming. The National Security Advisor decided not to wake his wife, but to let her die peacefully in her sleep. Is that what I should be doing? A question, not a point, I honestly don't know...