They were looking for a desired outcome, chances are you would not have been included if they didn't get it.
Wait, was this a dating site?

Also see the replication crisis, in which it turns out that nobody can replicate most of the so-called studies that come out. Or as the longshoreman philosopher Eric Hoffer noted, "Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket." Science, especially social science, has hit the racket stage.

https://en.wikipedia.org/wiki/Replication_crisis

Exactly - and that's part of what of is so flagrant about this. The replication crisis is The Big Topic in psychological circles -& has been for a pretty long while at this point! It's not just that the approach is bad for the reasons in the OP - it's that those doing it can't plead ignorance. Everyone (in these circles) knows there is a crisis of replication - yet they're, many of them, still hacking the system for flashy results.

I don't want to impugn the character of the researchers, because I don't think the problem comes down to character. It feels like this is when you really know its bad - everyone knows something is wrong - but the incentive structures push people to keep doing this stuff nevertheless. It's this weird zombie thing. I wonder what would really crash it, and force a meaningful restructuring?

(I occasionally, if in a particularly 'Jesus, c'mon!' mood, leave a comment in the study to these effects, but as good as it feels, I suspect that probably won't heal the sickness)

I don't want to impugn the character of the researchers, because I don't think the problem comes down to character. — csalisbury

I don't impugn characters. They're rational people responding to incentives. You're a young postdoc. You've busted your butt for years to get your Ph.D. then busted it again to find an academic job. You know that you have to "publish or perish." So you publish. You or I would do the same. And if you're not in a hard science, any research you do will be fuzzy and subject to all kinds of subtle biases. Your deadline's due, you need to show results to get your grant renewed. So you publish what you can. And everyone else is too busy to notice because they're playing the same game.

That's my understanding, anyway. The softer the science, the more it happens. You can't get away with irreproducible results in physics, but you can in sociology.

You can't get away with irreproducible results in physics, but you can in sociology. — fishfry

And medicine and biology and ecology and genetics and...

And medicine and biology and ecology and genetics and... — fdrake

Yes true. And actually a lot of physics is irreproducible these days, being entirely mathematical and not subject to any experimental verification at all. Industrial-scale rationality is failing entirely, as is industrial-scale everything.

So you publish what you can. And everyone else is too busy to notice because they're playing the same game. — fishfry

Old saying: "Everybody lies, but it doesn't matter, because nobody listens."

And actually a lot of physics is irreproducible these days, being entirely mathematical and not subject to any experimental verification at all — fishfry

From the same publication as the remark above (from the 1970s):

"If it's green or if it wriggles, it's biology. If it stinks, it's chemistry. If it does not work, it's phyics."

So true. I used sweat blood sweat and tears to show the preservation of energy in action with a simple, very very simple, heat-transfer experiment in first year. And I still had to fudge the data to get the result the TA wanted, by needing to say "some heat was transferred to the external environment." He bought it, that TA did.

And actually a lot of physics is irreproducible these days, being entirely mathematical — fishfry

I am sorry, but if someone can't copy numbers from one sheet to another, he should't have earned his post-doc standing in quantitative analysis of nanomolecules suspended in highly viscous medium flowing through carbon-based hex molecules.

I don't want to impugn the character of the researchers, [snip] but the incentive structures push people to keep doing this stuff. — csalisbury

If you can be ordered about by incentive structures, you have no character. Character is that which resists manipulation.

I suppose there are very good reasons why the distinction the soft sciences (psychology, sociology, etc.) and the hard sciences (physics, chemistry, etc.) exists in the first place. The former are recent entrants into the sciences - expect as many slip-ups from them as possible - and the latter are highly experienced veterans - expect of them a sterling performance in a manner of speaking.

I suggest we cut the soft sciences some slack if only because its harder to put them on a firm foundation of mathematical precision. Its one thing to quantify mass, speed, etc. and another to put emotions, thoughts, etc. on numerical scale. I suspect it all boils down to the fact that the soft sciences deal with matters that have a more subjective component to them, making measurement, if that's possible in the first place, harder and imprecise. Reminds me of the hard problem of consciousness somehow.

If you can be ordered about by incentive structures, you have no character. Character is that which resists manipulation — unenlightened

If character's the ability to resist manipulation (this is a reallyslippery slope though...) then character's on a spectrum. It may be true (i'm not sure) that saints with extreme self-determination + capacity for suffering can remove themselves entirely from incentive structures, but sainthood would be a high bar to clear in order to do labwork. When I say I don't want to impugn their character, I don't mean 'I think that scientists have better developed character than 99.99% of the population.'

sainthood would be a high bar to clear in order to do labwork. — csalisbury

Yeah, that's the Christian tradition, though. The religion of love brooks no compromise. If you blather on about the sanctity of knowledge, you have to uphold your own values. Bish bash bosh. If you have no love you are no Christian, and if the truth doesn't come first, you are no scientist.

I think that's a narrow idea of what Christianity is (though maybe close to extreme puritans like Jonathan Edwards...but even he would be less strict). Grace, atonement, forgiveness, redemption (among others) are important concepts in Christianity, but those concepts don't make sense if 'sinning' is grounds for immediate excommunication.

↪fishfry Exactly - and that's part of what of is so flagrant about this. The replication crisis is The Big Topic in psychological circles -& has been for a pretty long while at this point! It's not just that the approach is bad for the reasons in the OP - it's that those doing it can't plead ignorance. Everyone (in these circles) knows there is a crisis of replication - yet they're, many of them, still hacking the system for flashy results. — csalisbury

In my field, the phrase people openly use when talking about their data is “how can we spin this”? I am not making this up, I’ve had a conversation a few weeks ago at a conference with a poster presenter who used the words, “if we want to, we can spin the result like this”. It’s part of the discourse and it is understood that there is a storytelling(“spin”) aspect to it and nobody objects to this. Editors have more than once asked us to rewrite our paper so that our post-hoc findings can be re-cast as an a priori prediction; there is no sense that there’s anything wrong with that.

People who tell this kind of storytelling dominate the field, they dominate the funding and the job scene. If as a young student you are trying to do the right thing, you will not publish in top journals because your “story” is too ambiguous and tentative. People in these fields have no hesitation in making the strongest possible claim and then going beyond that. They publish in top journals, get the jobs and the research money. The honest student can’t compete. Once you have hundreds of articles to your name, it’s a winner take all situation. Fiske, in her interview yesterday, mentioned that she has some 300+ published articles; that’s the kind of number that gives you money when you want it, and where you want it, for whatever comedic project you come up with. So it is imperative that people are shocked into stopping this.

Disgruntled prof linked

I think that's a narrow idea of what Christianity is — csalisbury

1 Corinthians 13:2

In the matter of what Christianity is, if in no other case, The New Testament is definitive.

:up: Solid link. I found going through the other comments on that thread is fascinating too. Like how one person objects to the author's point (as made in the OP ,above the one you quoted) on the grounds that ridiculing bad studies is objectionable because it won't help make things better. Some people agree. Then some others say that mockery is necessary, because other means won't work, you have to sort of punch through the over-politeness. And then back and forth. It's cool to see in real time why its such a struggle to actually change things. The emotional tone is high, and you can see the defenses against change play out

The emotional tone is high, and you can see the defenses against change play out — csalisbury

:up:

That blog is a goldmine, it gives me the impression that internal squabbles in academia play out very much like they do on the forum. Posturing, misdirection, playing out the emotions rather than the facts, motivated reasoning. Easy to be reasonable when you're used to defining yourself as reason's voice etc.

I think it's better not to go down this kind of path. If you're interested in what that verse may mean in the context of a lived faith - e.g. why its translated sometimes as 'charity', sometimes as 'love' - or the relationship between scripture, tradition and interpretation in general- I would recommend diving into the literature, there's a lot of interesting stuff.

To go back to the main point, I think its easy to call-out failure, and much harder and more interesting to figure out why failure happens. I'd invite you to think about the implications of the relationship between character and manipulation you describe when applied generally. It's a slippery slope.

Point of all of this? I guess: be very skeptical of studies that involve surveys, and 'markets solve everything' approaches to academia. — csalisbury

I'd be more wary of this:

Russell Group university accused of Soviet-style censorship
Camilla Turner 7 hrs ago

A Russell Group university has been accused of Soviet-style censorship after requiring new humanities courses to “move away” from a “white, Eurocentric” curriculum.

Academics at Exeter University’s department of Social Sciences and International Studies (SSIS) have been told that they should “integrate” these changes when updating existing modules or creating new ones.

One lecturer said he is “shocked” at the stipulation and claimed the faculty - which oversees a number of disciplines including law, politics, sociology, philosophy, and anthropology - is undermining academic freedom “in the most profound sense”.

"It is like there is a Maoist cultural revolution taking place in our centres of learning,” one academic told The Telegraph.

"It is just ridiculous - we are supposed to be a leading Russell Group university. This affects thousands of students and hundreds of academics.”

The academic said the movement to “decolonise” the curriculum has swiftly progressed from a “faddish fringe theory” to being “adopted as the new orthodoxy” in universities.

He likened the approach to the Soviet Union where academics might be asked to prove how their courses would advance radical socialism in the face of reactionary capitalist imperialism from the West.

“What’s the difference here in the UK, where we are supposed to be a free liberal democracy?” he said.

https://www.msn.com/en-gb/news/world/russell-group-university-accused-of-soviet-style-censorship/ar-BB1gL6aH?ocid=msedgntp

I think it's better not to go down this kind of path. — csalisbury

Your privilege. I won't drag you.

I'd invite you to think about the implications of the relationship between character and manipulation you describe when applied generally. It's a slippery slope. — csalisbury

Ha! That is rich! A moral hard line is a slippery slope! You'll have to spell out those implications if you want me to think about them.

Ha! That is rich! A moral hard line is a slippery slope! You'll have to spell out those implications if you want me to think about them. — unenlightened

Here's a few ways to look at how moral hard line falls on its face:

1. Fudge -- by a socially well-accepted way among your peers -- data or else don't, but at the price of letting your children starve, or not allowing them to reach their potential because you can't afford to support them in that endeavour.
2. Be a maverick, and fail to publish, due to a character that gives breaking social customs more importance than the importance it gives to complete economic, academic and social failure.
3. Be praised for character, for not publishing anything, since you can't obtain repeatable data. This stops your academic advancement in its tracks. But since nobody will notice you, your character won't be praised after all. You sacrifice yourself for a cause for action that nobody will notice.
4. Your character will make you not publish; but your character won't create a way to publish the truth. Finding out and publishing the truth will remain evasive. Unrepeatable social experiments, no matter whether you create them or not, have no alternative to them.

Russell Group university accused of Soviet-style censorship
Camilla Turner 7 hrs ago

A Russell Group university has been accused of Soviet-style censorship after requiring new humanities courses to “move away” from a “white, Eurocentric” curriculum. — counterpunch

Exactly! The carbon copy why I was not allowed to study philosophy at the University of Western Ontario. I objected to the forceful, totalitarian, and basically unnatural feminization of the School of Philosophy. They were frothing at their mouths, and they called me in to sit with rabid feminists working the top echelon of administration. If looks could kill they probably would have. All I said (in my usual and customary provocative style) was that it was self-contradictory to exclude white males for the benefit of males of visible and other minorities and for the benefit of women in the department, because males are males, whether they are white, purple, black or a very intelligent shade of blue.

I spit at the dean and the faculty at Western. They are a bunch of frightened little piggies and tapeworms, who fear for their careers and their pension, so they let the rabid feminists run their lives like Stalin run the entire Soviet empire.

You sacrifice yourself for a cause for action that nobody will notice. — god must be atheist

Indeed, no good deed ever goes unpunished, usually by crucifixion. Principles are an expensive luxury, and not recommended for comfort lovers.

Indeed, no good deed ever goes unpunished, usually by crucifixion. Principles are an expensive luxury, and not recommended for comfort lovers. — unenlightened

Sacrificing anything of value for nothing, however, is recommended not to comfort lovers, but to suicidal people, the insane, and the extremely stupid.

Sacrificing anything of value for nothing, — god must be atheist

And there we arrive at the inescapable conclusion - that principles are nothing, that values have no value. Now that's what I call a slippery slope!

It's the deliberate biasing of academic research; and it reminds me of Superman II - where Lex Luthor, played by Gene Hackman says one of the most profoundly evil things I've ever heard. He says: "It's not enough that I succeed - everyone else must fail." That's the left wing academic agenda in a nutshell. They cannot make their arguments stick where others are free to point out the obvious, and systematic inadequacies of their ideas - which simply fall flat if not shoved down people's throats under threat of excommunication. That is, if you can penetrate the unnecessary jargon, hiding a post modern philosophical basis that encourages deceit insofar as it downgrades objective truth to socially constructed power relations, constructed, they claim, solely to oppress women, homosexuals and people of colour. Who thinks like that? ...other than Lex Luthor?

I've been on both sides--taker of surveys and producer of surveys; people are inherently untrustworthy and unreliable when it comes to taking surveys. Our responses are inflected by the mood of the moment; we want our responses to reflect well on us; we want to be "good survey takers" the same way we want to be "good drivers", "good employees", or "good mates"; wording of a question can throw our responses off. All of this is known by survey producers and administrators.

Our health education group used to administer a survey on gay male sex behavior and condom use at the annual gay pride festival (back in the late 1980s). Our cohort of survey takers was dominated by men who were eager to report their sexual behaviors and who, apparently, like taking surveys alfresco. As a result, our surveys (which were quite long and detailed) showed that the guys were performing all of the expected behaviors and that too many of them reported using condoms consistently. IF all the men taking the survey were both honest and representative of the much larger gay population, then why did we have so many new cases of AIDS in our community?

So, the results were probably not valid or reliable. It was a useful exercise because we got "data" we could use in reports. It reflected well on us, but we fully understood that it was a bit of a self-selecting farce. Still, we wanted to know what exactly gay men were doing in real-life sexual situations.

What was our alternative? Focus groups? 1 on 1 interviews? Hidden cameras and microphones? Participant observation? I was willing to use hidden cameras and mics, but my employer was decidedly not willing (cue the dithering over privacy rights, etc).

My bailiwick was outreach in high risk settings. I decided I would try a behavioral test in a high risk setting (an adult book store's basement cruising and video area). The idea was that I would propose oral sex first, and then see if they were willing to use a condom. Whether they were or not willing, was beside the point, because I didn't plan on giving a blow job in either case. As it happened, the first guy I tried this out on didn't appreciate the bait and switch, and forced me to carry through. He was bigger than me, so... In other settings--like the gay bathhouse--the participant observer approached worked better. The upshot was pretty much what we expected. A significant number of men were not willing to use condoms consistently.

OK, let's go back to invalid surveys.

If a survey for pay was actually studying something other than the stated topic under the guise of asking questions about canned food preferences, like how do people respond to certain words in the various questions, or what word order leads to more or less inconsistencies in responses, the survey could theoretically produce useful information. Usually surveyors want subjects to move right along, and not second guess their answers. So your speedy approach was probably not a problem.

The replication problem falls as much on the nature of the subjects as it does on the experimenters or surveyors. We are a shifty lot.

I regret taking the tone that I did above. I was discomfited by the introduction of a moral absolutism lens, and concerned that we were beginning to get into a point-scoring dynamic, one which I admittedly was contributing to.

There's a lot to say about the bible verse you quoted, 1 Corinthians 13:2. My understanding of that verse is that love is central to christianity. All of the other virtues are empty if they're not enlivened by love. Love is a translation of 'agape' which is theologically complex. As I noted above, it is also sometimes translated as 'charity.' Agape/love/charity has a lot to do with the way we treat those we live with. We should love our neighbor as ourselves, & a big part of that is judging not, lest we be judged & casting the beam out of our own eye before casting the mote out of our brother's. We should be charitable in understanding others. It would be a bad application of Agape to react to bad goings-on in town by showing that others aren't saintlike.

In general I think it's good practice on the one hand to minimize any blame of a situation when focusing on our own mistakes (because this takes power away from us, and reinforces learned helplessness) but, on the other hand, to maximize the role situational factors play when approaching the mistakes of others. Doing this first - another word for this is compassion, or empathy - will better allow that person to then, hopefully, cultivate their own ascension out of learned helplessness - that is, empower them. This is not hard-and-fast, its a loose rule that itself changes dependent on the situation. I think this is what Christianity at its best helps with - its very concerned with the understanding that we and others will fail, and developing ways to heal those ruptures.

In other parts of 1 Corinthians 13 we are told that Love is not proud, boastful, self-seeking, easily angered, seeking to dishonor others and so forth. But any close - or even unclose - reading of Paul's letters shows that Paul is very much all of these things. That could mean, as in certain atheist readings, that the bible is contradictory and should be tossed out. We might also say that Paul is a brilliant, but flawed interpreter of the gospels, and the Jesus event, and let that inflect our understanding of the new testament, so that we are less likely to see it as perfect whole . We could also apply both the gospels and Paul's own insights to our relationship with Paul's letters, and scripture in general. There are a lot of different approaches.

For these reasons, and others, I was frustrated with your post responding with that single verse, because while you meant it as a proof that my claim that you were taking a narrow of christianity was wrong, the post itself seemed to be another example of that narrow view of Christianity. And it also felt like that verse, particularly, suggested a very different approach to the problem in the OP than the one I took you to be taking.

I think that character, like any virtue, is not something one either has or doesn't, but is something cultivated with many zigs and zags. It can create problems, ifwe say that someone who allows themselves to be manipulated lacks character full stop. For instance, is a woman in an unpleasant situation who gives in to sexual requests in order to escape that situation, allowing herself to be manipulated, demonstrating a lack of character? Is this the same situation as the one in the OP? Of course not, but that's the point - applications of moral judgments require a lot of finesse.

Another figure who comes to mind when I think of stark, absolute approaches to Christianity is Kierkegaard. A key fact about Kierkegaard is that he was independently wealthy, he was removed from the thorniest, existential aspect of the more fundamental incentive-structures.

Now, look, certainly character does play some role in this whole thing. When I say I'm not looking to impugn character, I'm signalling what I think is the bigger factor at play. Certainly, a scientist may not play ball, and not get published - and that is good for him! But then the rest of us are still getting the stuff that is published. So while we might esteem him for his resistance, it's not changing the system that determines what sort of stuff gets published - and the OP was about that problem, not about the souls of individual scientists. I think soul-stuff is very important, don't get me wrong, but this thread isn't focused on that.

I hope that clarifies what I'm trying to say a bit.

My bailiwick was outreach in high risk settings. I decided I would try a behavioral test in a high risk setting (an adult book store's basement cruising and video area). The idea was that I would propose oral sex first, and then see if they were willing to use a condom. Whether they were or not willing, was beside the point, because I didn't plan on giving a blow job in either case. As it happened, the first guy I tried this out on didn't appreciate the bait and switch, and forced me to carry through. He was bigger than me, so... In other settings--like the gay bathhouse--the participant observer approached worked better. The upshot was pretty much what we expected. A significant number of men were not willing to use condoms consistently. — Bitter Crank

Ha, love it. Well the idea of the gonzo approach anyway (that experience sounds brutal, man)

And I do take your point that all surveys have substantial 'baked-in' problems that are probably unavoidable. Your examples are good. I think more than anything, my uneasiness with the phenomenon in the OP is the factory-approach, and the fact that many people are taking hundreds, if not thousands surveys a year. I think that once you have a population of professional survey takers, churning them out, you're messing with something fundamental, but its harder to nail what that is precisely.

I agree - having a small group of monetarily motivated survey takers dominating the results is undesirable in terms of obtaining "reliable" and "valid" results. I think there are ways of avoiding the problem. First of all, lots of people will take surveys for free. (I have nothing against you paying the rent by survey servitude.) I do surveys once in a while; they are too boring to do very often.

The problem of unreproducible results occurs in structured situations, too, I gather. Subjects come into a lab; they are identified; they complete some sort of experimental task, and leave. Maybe they return for several sessions. The conditions are controlled. The experiment is approved by institutional review boards and faculty advisors. It's all on the up and up--and the results still unreproducible,

Some kinds of labs do produce good results: tests of color perception, hearing, visual processing, response time, taste and olfactory sensitivity, skin-response, learning, memory, and so forth. Those sorts of experiments should produce reliable, valid, and reproducible results. It's basically bio-measurement.

It's much dicier when researchers are out to find the motivational factors in product purchases, for instance. Maybe an fMRI would be a better research method than surveying 1000 car owners as to why they bought a Ford instead of a Toyota, or pink-23 instead of red-45 lipstick.

Take a look at the art market if you think the social sciences are something of a racket. Art has aspects of major league racketeering about it. I'm not talking about the Louvre, or the Guggenheim. It's the up-and-coming go-getters in the art-biz who are the racketeers.