5) Thus, science gives human beings new powers at an ever accelerating pace.

6) Human maturity and judgment advances at an incremental pace at best, if at all.

7) To illustrate the above, imagine a car racing down the highway at ever accelerating speeds, while the driver's skill increases maybe a little bit now and then. — Hippyhead

I don't think it's so clear that there is a widening gap between human maturity and new powers afforded by science. For one thing, advances in science seem to require a maturation of the understanding of reality and of the ability to analyze and synthesize, and thus science cultivates careful reasoning, a universal outlook, honesty and cooperation. At the same time, society builds schools in which the scientists are educated not only about natural sciences but also about humanities, moral values and various aspects of the functioning of the society. Education with less focus on science is also provided for non-scientists, so the society generally is more or less educated and able to participate in beneficial application of science, including by building and respecting regulatory governments and laws.

Pressing the nuclear button may destroy the society but the fact that it hasn't happened yet shows that the modern society is a not just a collection of savages but a sophisticated system with psychological, social and technological safeguards.

In short, many of these journals realize that if people who do not understand the particular science involved this information (if misunderstood and misapplied) could prove to be dangerous; thus is it not open access. — Mayor of Simpleton

This is absolutely untrue. Scientific journals cost money to produce. While referees give up their time for free, the editors are highly skilled people and there are huge production costs.

The typical approach to funding has been to put the costs on the reader because otherwise finances become a barrier to scientific progress. We need good science to be published whether the scientist can afford it or not. Most publications are affiliated to universities which also pay subscriptions to the journals. As such, while authors are asked to make a contribution to the publication of their material, it is almost always waived.

Because this model puts the burden of the cost onto the reader, it has been standard procedure to put preprints on arxiv for many years now, which means that, while you have to pay to see the final article, the content is usually accessible for free. This is not objected to by the journals who are more than aware that universities will continue their subscriptions whether the material is available unofficially or not.

There is movement atm toward a completely open access model. I think all of the Physical Review journals are now freely available, with funding from various sources, including university subscribers still covering the bulk of the bill.

In brief, the effort has always been for ethical journals to allow scientific progress to be as freely available to all as possible.

I think this is a good question. We're currently at the point where knowledge as to how to effectively exploit our environment has led to unforeseen (or often foreseen but obfuscated) negative impacts on that environment.

This puts us in the position of using technology to now fix the problems that our use of technology caused in the first place, which introduces a feedback loop.

Whether this ends up being good or bad depends on other unforeseen or obfuscated environmental impacts of these second-generation technological applications.

It might be that our increasing knowledge allows the intended reduction of environmental impacts to outweigh any unforeseen or obfuscated negative impacts, in which case we should listen to the science and ensure that bad actors do not mislead us.

Alternatively the unforeseen impacts might outweigh our intended efforts, in which case, yes, we should focus on reducing the use of our harmful technologies rather than making new ones to fix the problems.

On the unforeseen/obfuscated distinction, sometimes the knowledge is there that we should *not* use certain technologies, an example being diesel. The UK government sold diesel to the public as a cleaner, safer alternative to petroleum. The science at the time said the opposite, but energy companies had already invested in this technology and governments tend to represent the interests of corporations and hang the consequences.

[EDIT: Just today, research was announced that diesel fumes are a cause of increasing mental health problems, particularly depression.]

So it's not just a matter of unforeseen consequences. We also have to account for the casual evil of people, especially the sorts of people that for whatever reason we tend to elect to office.

I hear ya. To be clear, I have no interest in demonizing scientists. We hire them to develop knowledge, and they do a good job of performing the service we asked them to perform. — Hippyhead

Yes, I believe you. I met many of those professional scientists already. They, unlike I, had to work for some others, not for themselves, in order to gain not only their daily bread but some extra bonuses as well. After all, they deserve these extra bonuses as a humble reward for their submission :)

Yes, that's all it is. The gap between power and wisdom is widening at an ever accelerating rate. Power races ahead while wisdom inches along at best. — Hippyhead

You say it better than me! I'm jealous! :smile:

A key problem that is until we hit the chaos wall the knowledge explosion delivers a wonderful array of amazing goodies. — Hippyhead

I would caution you against speaking too soon. The "amazing goodies" you speak of may come at a heavy price somewhere down the line and it maybe too late by then to, you know, reverse/stall the process that has "catastrophe" written all over it. Isn't this your main worry? :chin:

the knowledge explosion steadily shrinks the room for error. — Hippyhead

You make a valid point. Knowledge brings power and in todays world that power is immense.

As a "Philosopher" however I would offer the following personal observations:

1) Scientific advance has also brought humanity as a whole the tools to monitor progress and provide feedback.

2) To believe that we would as a species willingly follow a path to Mutually Assuered Destruction would be to accept we are doomed by an inherent madness.

3) In any system negative feedback creates greater stability and wider bandwidth (diversity).

Comments?

The problem, as I see it, is the scale of the destructive powers being generated by the knowledge explosion, nuclear weapons being the easiest example. Genetic engineering and AI etc may pose similar existential risks, though that is far harder to calculate. — Hippyhead

Oppenheimer.

For AI: Yudkowsky and Bostrom spring to mind.

For genetic engineering: like, everyone, it's been part of the discussion all along.

This is the point I tried to make in my first response. You're absolutely right that not all nuclear physicists thought through what they were doing, but it is possible to do so and we have an heroic example of doing so.

People worried about the AI alignment problem have been ringing exactly this alarm bell for a while. Bostrom even managed to get a think tank for studying existential risks to humanity created at Oxford.

And noting that we have all too many examples of scientists not recognizing the risks their work gives rise to, we could consider mechanisms for formalizing the feedback, so that we put effort into increasing our ability to use new technologies responsibly or we deliberately dampen the pace of development until we can do so, perhaps forever.

I just recently watched Thomas Schelling's Nobel acceptance speech, which is really curious because he doesn't talk at all about the theory for which he was receiving the prize but about the history of not using nuclear weapons, in part to show how far we've come. Back when he was trying to prevent such use, in the fifties and sixties, everyone thought nuclear war was a near certainty.

Humans don't need science to destroy their environment, and in fact, they'd likely do more damage without it.

Perhaps I need to clarify...

By open access I mean 'open source', as in open to the public.

To my knowledge the vast majority of journals I access, being in Orthopedics, Sports Medicine, Dietary Science and Rheumatology, are not open source, as in open to the public.

Indeed the economics involved with the publishing process play a role, but no journal I know of is there as a profit making enterprise.

One of the main reasons as to this not being open source is the tendency for patients to self-medicate or misunderstand a potential treatment or medication and subsequently petitioning the medical professionals without end to treat them with this misunderstood option. (in the case of Rheumatology, we have patients calling the main office of the Professional Organization at least 2 to 4 times a week claiming to be 'professionals in the field' and demanding access to journals and other sensitive information)

I know this sort of discretion with such information in Austria is the norm, but in the USA or the UK I'm not up to date. Things such as Pharmaceutical Companies giving out 'goodies' at medical conferences have been eliminated for quite some time now and to have them create advertisements prescription medication to the public is forbidden. (in short none of those commercials where you are told to 'ask if this is right for you' you have in the USA)

As Pharmaceutical Companies fund the majority of the research a control needs to be put into place and one of these controls is in open public access that can potentially turn into a means for commercial gain at the cost of quality medical care.

For that matter the people who attend a medical conference have to have a proper accreditation. These are not open source to the public and in the event patient are needed (as is the case in medical workshops dealing with various illnesses) these patients are carefully escorted as to not grant them access to any other part of the conference where they do not have an accreditation other than the workshops where they play a role in hands on techniques involved in those sessions.

By open access I mean 'open source', as in open to the public. — Mayor of Simpleton

Yes, so do I. And the movement is toward open source.

Indeed the economics involved with the publishing process play a role, but no journal I know of is there as a profit making enterprise. — Mayor of Simpleton

There are lots (Elsevier, for instance) but that isn't the point. Publishing a journal costs money, which requires income, even for non-profit publishers like the APS. They are not putting up a paywall to deter lay people: they do it to cover their costs. The fact that pretty much any journal will publish a manuscript that is available freely on arxiv demonstrates that they're not trying deter people from reading science.

One of the main reasons as to this not being open source is the tendency for patients to self-medicate or misunderstand a potential treatment or medication and subsequently petitioning the medical professionals without end to treat them with this misunderstood option. — Mayor of Simpleton

That may be the case for specific journals, but it isn't general, even to pharmaceuticals. In fact, the journal Pharmaceuticals is open access. Do you have an example of a journal that has this policy? e.g. do any journals with paywalls forbid preprints on arxiv?

[EDIT: Pharmaceuticals, Perspectives on Medicinal Chemistry, Drug Delivery, International Journal of Nanomedicine, Journal of Pharmaceutical Analysis, and Pharmaceutical Biology are all open access. It seems harder to find pharmaceutical journals that aren't.)

Either way, it is not generally true that this is the reason for paywalls, as evidenced by the fact that the major non-profit publishers are moving toward open access and have allowed open access preprints for decades. All of my papers, for instance, are available in preprint form on arxiv for free, and now are available as finished articles from the APS for free.

Funny... I could not find a single article published by any one my colleagues on arXiv, medRxiv or bioRxiv.

As these platforms are not journals in themselves, are there standards for peer review?

Is there any standard in place to prevent the dissemination of articles from predatory journals?

Funny... I could not find a single article published by any one my colleagues on arXiv, medRxiv or bioRxiv.

As these platforms are not journals in themselves, are there standards for peer review?

Is there any standard in place to prevent the dissemination of articles from predatory journals? — Mayor of Simpleton

They are not journals, just preprint databases. Point being that even journals that charge for access allow the contents of their publications to be freely available on such a database. The fact that your colleagues don't use it isn't especially relevant afaics: it is up to the author whether they do it or not. On the other hand, the fact that these databases exist and contain preprints of manuscripts published in journals with paywalls tells us that said journals are not trying to limit access to their contents.

I'm not aware of any journal that won't publish an article on these databases, but there might be some. Which journals did you think had this policy of deterring lay persons?

To be honest, I've never really heard of these reprint databases. Probably because they simply aren't necessary for me to gain access to the journal articles.

I'm happy nothing I've been a part of publishing is to be found here, as it would have been re-printed without any consent.

As I was not aware of these databases, I can't say if any of the journals we publish in have restrictions or not.

Then again, most of what we publish is in German, so maybe it fails the language criteria, if they have one.

For genetic engineering: like, everyone, it's been part of the discussion all along. — Srap Tasmaner

Yes, everyone will claim they already know it, they're already doing it, no need for feedback from the public, yada, yada and more yada, but...

The march for more and more and more knowledge (and thus power) continues full speed ahead.

Why are scientists responsible for someone else's relationship with knowledge? — Mayor of Simpleton

How about responsible for their own relationship?

To believe that we would as a species willingly follow a path to Mutually Assuered Destruction would be to accept we are doomed by an inherent madness — Chris1952Engineer

Whether we are doomed is unknown, but we are certainly mad. It's not just that we have nuclear weapons, we are bored by them. What can that be called other than madness? And it's not just those stupid people over there, it's almost the entire culture.

Well, as long as they are not promoting pseudoscience of preference and non-peer reviewed findings I'd say they are doing their job as they should and being responsible to their relationship with knowledge.

At the moment I can't seem to make heads or tails of what relationship you are referring to.

It seems you are still sort of holding scientists responsible for the misguided actions of others, as if those are somehow predictable and certain outcomes.

I wonder if the manufactures of the baseball bats I own to play baseball should face criminal charges for the actions of football hooligans?

If the science doesn't have intentions of malice in the results, how exactly are they supposed to be held responsible for the actions of malice others do by misusing their findings?

Yes, everyone will claim they already know it, they're already doing it, no need for feedback from the public, yada, yada and more yada, but...

The march for more and more and more knowledge (and thus power) continues full speed ahead. — Hippyhead

WTF?

Is that anything like what I said in the post as a whole?

I thought you brought up a really interesting issue; I pointed out that others have noticed related issues and even devoted much of their careers to it; and I suggested other steps that might be necessary if we don't get our shit together.

But it turns out what you really want to say is just, fuck science and fuck scientists, they're ruining everything.

Or maybe my paraphrase is unfair. Gee, what's that like?

People worried about the AI alignment problem have been ringing exactly this alarm bell for a while — Srap Tasmaner

What both of us said is true. You're right, the alarm bell has been rung. I'm right, it makes no difference, the relentless march for more knowledge continues.

What could be interesting is a scientist who argues the relentless march be paused. I'm guessing such a person would find their career soon at an end, but I could be wrong of course.

I'm supportive of science. The problem is that the biggest threat to science is science.

You're right, the alarm bell has been rung. — Hippyhead

But I am very far from dismissing your worries. Oppenheimer is heroic precisely because he was rare. But after Oppenheimer, it's pretty hard to hide behind "pure research" as a shield.

I don't happen to know what the debates look like around gene editing and the like, but I've been around long enough to know that the whole field has been steeped in ethical debate from the beginning, so I just have to hope it has had an effect. It's hard to imagine being in that particular field and not feeling you're under scrutiny.

For AI, things are a little mixed, but there are loud voices, Nick Bostrom and Eliezer Yudkowsky being only two that I happen to know a bit about, trying to get a hearing. I'm not sure that's going all that well honestly because my impression is that before some recent strides with new ML techniques the field had started developing a general defeatism that I think has left a hangover -- a sort of, ah, we'll never manage to do it anyway, so why worry. But the two I mentioned are among those who are very worried, and also worried about accidental "success", if that's what it turns out to be, and not averse to halting research programs until we're clearer on their impact. At least that's my impression, been a while since I read any of that stuff.

I agree with you, we are not okay. And I think there are areas where it just makes sense to have some sort of formal impact analysis required up front and some periodic review. With AI, we do have to be careful not to get too lucky.

One further point about AI in particular comes up in The Social Dilemma, that we don't have to have a war with the machines for things to have gone wrong: some of the algorithms running at Facebook and Google are no longer anything a human being can understand, not well enough to assess their impacts in a meaningful way. We were still on the edge of that when the problems with YouTube's recommendation engine came out and Google was able to take some reasonable steps to address that. It's no longer perfectly clear that the sort of action they could take then is possible now. In other words, we may already have reached the "shut it off, pull the plug" point for some of the stuff going on at say Facebook.

I'm still really glad you brought this up and it's a good reminder that we should all learn a lot more about what's going on.

But I am very far from dismissing your worries — Srap Tasmaner

Understood, no problem. But if you'll review my post again you might see it's actually not about you.

I don't happen to know what the debates look like around gene editing and the like, but I've been around long enough to know that the whole field has been steeped in ethical debate from the beginning, — Srap Tasmaner

I hear you, true enough. But genetic engineering is still racing ahead at full speed, isn't it? Or am I missing some restrictions which have been agreed upon?

I'm still really glad you brought this up and it's a good reminder that we should all learn a lot more about what's going on. — Srap Tasmaner

Thank you, and thanks for your patience when I type too quickly. I'm probably engaging in too many places right now. I don't even know why I type about this actually, as I suspect the bus is just going to have to run off the road before we're willing to take it seriously. Human beings learn by pain.

What's the relational structure of knowledge? — Hippyhead

The complexity of relationships between all information - this is the source of wisdom. We’re currently isolating and micro-managing edits to the environment so much that we’ve lost sight of how it’s all arranged to collaborate so efficiently with minimal effort from us.

The quantitative power of knowledge is only part of this capacity to edit the environment. We also need to understand the most efficient and collaborative ways to focus and channel power sustainably, otherwise we’re just creating chaos. And we need to understand our biases: why are we editing the environment? Because we can? Because it brings us pleasure? Or because we’ve ignored, isolated or destroyed the relational structure that enables the environment to edit itself?

Science is an effective tool for attributing knowledge to a human observer. But we’ve been ignoring where we fit in the whole structure, and how that limits our perspective. Shut Up And Calculate and other small thinking, exclusive or isolated approaches to scientific knowledge focus on quantitative aspects without much concern for how our power to edit one area of the environment relates to processes in another. To say this isn’t a concern for science is irresponsible.

Well, watching public hearings where elected officials clearly have no idea what the tech people they're talking to are really up to -- not encouraging.

Perhaps the central problem here is that the only people genuinely capable of understanding the issues involved are scientists and technologists.

Option 1: we get them to effectively police themselves. I think in practice that means some of them policing others, and that means providing institutional support to those scientists and technologists who have been forthcoming about the issues and take them seriously. The institutional support Bostrom has received is just a tiny start. We could actually institute review boards or something: you don't convince our representatives within your community that this is cool, you don't get funding and other resources. That sounds a little sketchy, and leaves the problem of how we could have the competence to select such representatives. And it could just fail -- the US used to garner Nobel after Nobel in high energy physics, then we dropped the ball and suddenly all of our postdocs and young researchers were headed to Europe and elsewhere. People can always pack up their science and go somewhere else.

Option 2: we raise our own competence and we make Mexico pay for it get scientists to help. Scientists themselves are aware of the problems of not communicating with the wider public about their work -- I think the fights over creationism left a mark and then the failure (what I was talking about above) to get the superconducting super-collider built left more like a wound. I think those two things, within living memory for a lot of folks, made it clear that communicating with the wider public is not something science as a whole can shrug off or leave to journalists.

I think we could really step up and support Option 2. Provide serious funding and support for communication efforts and in turn raise our expectations of what we laypeople will get in return. It could amount to "you don't get our money until we understand what you're doing, so explain it" in the best possible way. We all know academia has suffered from the current climate in which only research matters because it brings in the money (and oh yeah we also teach a little on the side). Think of science as being in a similar position: not much in the way of funding resources or prestige attached to public outreach, and specifically in communication that would make us competent to regulate this stuff. We can fix that anytime we want.

To say this isn’t a concern for science is irresponsible — Possibility

Can the proposed danger be said to be a real concern to the science community if knowledge development continues seemingly full speed ahead in every field?

That said, I am agreeable to relating to scientists as one would a highly skilled car mechanic. If the mechanic does the job you're paying him for, we applaud, and don't expect them to be responsible for air pollution.

Perhaps the central problem here is that the only people genuinely capable of understanding the issues involved are scientists and technologists — Srap Tasmaner

Your point is taken, but I'm not sure the core issue involved is really a technical one. It seems to be more a matter of our relationship with knowledge, and the power that flows from it.

The knowledge explosion has been good to us, and so naturally we want more of it. More and more and more, seemingly without limit. Such a "more is better" philosophy was sensible for most of human history when knowledge was scarce. It seems far less sensible in the midst of a historic ever accelerating knowledge explosion.

As I see it, we are attempting to apply a 19th century philosophy to 21st century problems. Technology races ahead at an ever accelerating rate, while our relationship with technology remains stuck in the past. The gap between the two seems to steadily widen.

I'm happy nothing I've been a part of publishing is to be found here, as it would have been re-printed without any consent. — Mayor of Simpleton

It would only be there if you gave consent for it to be there.

As I was not aware of these databases, I can't say if any of the journals we publish in have restrictions or not. — Mayor of Simpleton

So... the obvious follow-up question... how can you justify your claim that journals are attempting to hide content from lay readers and not know if they're okay with authors making that content open access? It seems to me that any policy of hiding content behind a paywall would demand overt and strict exclusivity.

Science bad, let’s get rid of it.

Scientists themselves are aware of the problems of not communicating with the wider public about their work — Srap Tasmaner

Agreed. But when they learn to communicate better, won't they use that ability to sell us on more of their work?

It could amount to "you don't get our money until we understand what you're doing, so explain it" in the best possible way. — Srap Tasmaner

So they will explain it. And then keep doing more of it.

How about this? "You don't get our money until you demonstrate an understanding that science is a death trap."

Arguing against the death trap theory would require one to demonstrate that human beings are capable of successfully managing any amount of power delivered at any rate. If they would like to rise to that challenge I would surely be interested to hear the case.

Do you have any contacts in the science community? Know anybody who does? If you can arrange a friendly debate in neutral territory that would be a good next step for this conversation.

Example. Human culture has survived for thousands of years without genetic engineering. So why not pull the plug on that field and engage a century long conversation on that subject before continuing?

A reasonable argument can be made that this would be impossible because if we don't do it somebody else will. That is, we aren't in control of the knowledge explosion, it's in control of us. That could very well be true. If it is, we are fucked. So let's face that squarely, and stop pretending we are going to somehow cleverly avoid the logical outcome of ever more power delivered at any ever faster rate.

An individual has to face the fact that they are inevitably going to die. Time for our civilization to do the same?

"You don't get our money until you demonstrate an understanding that science is a death trap." — Hippyhead

You might be assuming that but I'm not. I thought your idea of their being two different rates of change was spot on, and very close to what others have said. Given that, we can demand that the new tech demonstrate first, or as it reaches development milestones, whatever, demonstrate that it can be controlled. This is exactly what did not happen with nuclear weapons, exactly what did not happen with the deployment of AI by social networking platforms.

The latter is tricky. The same sorts of techniques (way down at the bottom) put to reprehensible use by Facebook can be put to excellent use in reading X-rays, where machines I believe are capable of outperforming radiologists now. So what do we do there? If we block their use entirely, we miss out on a good. Are we to have a Ministry of Technology that would approve uses, and police and strictly control their distribution? I'm not in love with the idea, but maybe it's necessary.

I'd say that there's another option of bringing around scientists and technologists so that we don't have to police them, but even if almost everyone bought in and acted responsibly, thousands and thousands of Oppenheimers, there would always be somebody who'd break the rules. Maybe this is the place for some sort of governmental action.

By and large this is how regulation of any industry works; there's usually a certain amount of trust between the watchers and the watched and that makes it cheaper and more efficient for everyone, but only with the understanding that violating these norms can bring the full force of the law down on you. It's a solid model, but in practice we all too often end up with a mess as bad as what we're trying to get a handle on.

Still, we could aim for fixing our whole approach to regulating the private sector. Maybe science and technology are not particularly special cases in this regard.