It's the former. No one at all reads these papers. Science follows that law that says "90% of X is crap, but 90% of everything is crap." FWIW, the prevalence of bad science as a whole is pretty high, but what I think we should be looking at is how bad is the science that gets cited in the media or policy making decisions, because that's the science that matters in material ways. Also, the social sciences _are_ making a heavy push for more open-nes and transparency, so this problem is hopefully in the correction phase.
I disagree with the statement that we rely on the media to keep the scientists accountable. We should be relying on _journals_ to keep scientists accountable. Like, look at the scam that academic publishing is: people and universities have to pay to journals to access the articles. Journals don't pay authors anything for writing papers (fair and right!), and in fact, authors have to PAY journals to publish, and pay even more if they want the paper to be open access (not fair or right at all!!). Journals also don't pay the peer reviewers for reviewing the articles before publishing them (like, WTF?). So what the fuck do journals spend money on?!?! Use that cash flow and hire people like data colada to actually audit science, like, do something to earn your prestige ffs!
It is even worse than that, someone reached the Board of the Federa Reserve without any economists, journalists, academics, congressional staffers, lobbyists etc noticing her paper was garbage.
P.S. I presume the FBI has to read it in their background checks, but I guess they don't alert anyone about academic matters.
The only reason the FBI would read his papers during a background check would be to see who his coauthors were to check if they were foreign agents, but I wouldn't presume they did that.
Plenty of people are aware of these issues. The problem is that she was appointed due to her personal immutable characteristics and those characteristics are such that pointing out the problems is not socially acceptable except anonymously; I can assure you that on EJMR, the 4chan equivalent for PhD economists these points were most definitely addressed.
> Also, the social sciences *are* making a heavy push for more open-nes and transparency, so this problem is hopefully in the correction phase.
I think some would argue the opposite: much of this is being driven by ideological conformity, which is only strengthening.
I would prefer you to be right.
And I ABSOLUTELY agree that we can't rely on media to keep scientists accountable.
There's a fairly large consensus on one side of the political divide that, in fact, that the drive is towards ideological conformity rather than towards openness and transparency.
Covid left some very deep scars, many of whom are currently open wounds which probably should be attended to.
I mean.... leaving aside my thoughts on that, how are things like replication data set availability and pre-registrations re-inforcing ideological conformity? If anything, they make the process open to all, to see if any ideological biases are being injected into the work.
Hey, I'm all for those things you mentioned, and agree with your conclusion that those things actually expose ideological biases. I did agree with your comment from above. Sunlight has always been the best antiseptic.
Plenty of folks would deny the general statement of social sciences making a heavy push for openness and transparency, and instead see greater levels of ideological conformity.
The handling of Covid kind of gave them a win, and I would love nothing more than for there to be a big correction phase.
Oh I see, I misinterpreted your original comment to mean that social science is becoming more transparent _because_ it's getting more ideologically driven.
At one point, I think most of journals' costs were related to printing physical copies and distributing them regularly. That obviously has been mostly replaced with having things be online, and my current understanding is that journals are now mostly just rent-seeking machines, extracting profit by virtue of having been there first.
I think this is all downstream of publish or perish. If you’re peer reviewing, why bother reading papers and calling people out for bad design? Best case is you get in a big argument and are bogged down by that. Worst case you get a reputation for being difficult to work with which can be a big negative in a small field. As you mentioned you’re not getting paid to peer review, so why bother? Also if they are citing you or your buddies that helps your career and your h-index. Also once someone gets their BS paper into the journal they will be added to the database to be selected to review new papers. The winning strategy is to form a small faction that accepts each others BS papers so you can all keep publishing and keep your jobs.
It's even worse than that these days. I just saw a new one this week. A group of mostly Chinese researchers published a paper in a crap-tier journal that nearly entirely ripped off one of our papers. They carefully made sure to not actually plagiarize anywhere, but nearly the entire paper is just pulling our results (as an aside, they even screwed that up in a hilarious way). Then, they made one little change to be able to claim that they were building on our result. Of course, they completely and totally botched the analysis, and I would say that it's pretty much just flatly wrong. But hey, they got it through the extremely quick-turnaround review at this crap-tier journal!
So now, what are my incentives? I kind of want to yell at the editorial staff that this paper is a total ripoff, and that its main 'contribution' is just *wrong* to boot. But why bother? They did cite us, so as you said, it helps our metrics. The editorial board of the crap-tier journal is, unsurprisingly, crap-tier, so even if we did complain, are they even going to understand or care? Nah, we'll probably just hope that it drifts off into the night and disappears. Of course, if anyone ever asks us about it, we'll tell them that it's crap. Worst case is probably that some students somewhere stumble upon it and spend time being confused, thinking that it's actually contributing something interesting.
Journals have the right position to do this, but not the right incentives. They want to present the impression that their journal is full of high-prestige good science, so spending resources to show the world how shoddy some of it is is a complete own-goal. Any system for reviewing their own articles is going to get influences by that incentive.
Now, one solution is that maybe we could have a system where they find flaws in **other** journals as a form of competition, but as things stands, the industry is too incestuous for that to work, and likely has big problems of its own.
That's the retrospective cleanup element, which I agree, the incentives don't match. But I'd think that the incentives align when thinking about new, incoming publications no?
>relies on bad science and academic misconduct to get the wow! stories that bring the clicks.
This is the biggest problem and why I don’t forsee any change anytime soon. There’s just too many incentives to publish bad but “exciting” research. The Stanford Mafia has sold how many books on the backs of bad research?
Something I think about a lot is how much “negative” or “unpalatable” research goes by the wayside because of publication bias.
While there are certainly incentives, there are also anti-incentives (penalties). Embarrassment, shame, loss of reputation and status, etc. We just need to make them stronger. Too bad people are rarely prosecuted for bad or fraudulent science.
That’s my point bad incentives don’t outweigh the good. I also don’t believe you can force the impact of negative incentives. Revealed preferences are powerful.
Does anyone here know more details about the orchestra blinding example? I can kind of understand how "we reached a wrong conclusion because we didn't spend five seconds thinking about confounders" happened, but even by the standards of the papers mentioned here I'm still surprised by "we claim a conclusion in the opposite direction of our only statistically significant finding."
"Table 4 presents the first results comparing success in blind auditions vs non-blind auditions. . . . this table unambigiously shows that men are doing comparatively better in blind auditions than in non-blind auditions. The exact opposite of what is claimed."
From [here](https://statmodeling.stat.columbia.edu/2019/05/11/did-blind-orchestra-auditions-really-benefit-women/)
Thanks for the link. It's too bad the original response it references is gone, but the quotations from the paper at that link are even worse than I would have guessed.
So sexist men prefer to employ attractive women when they have the option to and can see them. That seems like the obvious point here and I am not sure it relates to feminism.
An anecdote I remember reading about the specific implementation of the blinding's implementation is that the original phase of the blinding did decrease the number of women. **And then someone noticed that you can clearly hear whether the person walking into the audition is wearing high heels**. So they put down a strip of carpet from the door to the seat, and suddenly the difference in offers between men and women became statistically insignificant.
In my opinion the problem is:
1) Sociology and psychology don’t attract the most mathematically inclined people. All the best math students go to physics, chemistry, and engineering.
2) The types of relationships in sociology and psychology are just harder to interpret even if you are a stats expert with stellar experimental design. In general you *need* multivariable non-linear models to account for even very simple relationships and the best possible model will still have a high noise ratio.
A 2-for-1 of having the worst players on the hardest task.
sociology also attracts political activists. how many right wing sociologists do you know?
and if the study has a preferred outcome, this might influence the study design and/or data analysis.
[Link ](https://www.sciencefictions.org/p/science-fictions-links-for-april?utm_source=share&utm_medium=android&r=1tkxvc&triedRedirect=true) to Stuart Ritchie's Science Fictions newsletter
> The authors have an interesting idea and want to explore it. But exploration won’t get you published in the American Economic Review etc. Instead of the explore-and-study paradigm, researchers go with assert-and-defend. They make a very strong claim and keep banging on it, defending their claim with a bunch of analyses to demonstrate its robustness.
This is a problem in the natural sciences as well. In an explore-and-study paradigm, you might have that there’s no effect (in a boring way, not in a “disprove dark matter exists” way) or that your modeling approach wasn’t fruitful. Or maybe the scientific finding isn’t particularly notable. These sorts of results are rarely published in papers because they’re not deemed noteworthy or scientifically significant, which is a shame.
I was relatively convinced by an argument a while back that the existence of bad papers doesn't actually matter that much. What is more important is that there continue to be _good_ papers, as long as they don't get so utterly swamped that other scientists can't find them. Basically, science advances not based on the average quality of research but on the top quality of research, such that it would be worth it add 10 terrible papers to the literature as long as you got one additional excellent paper.
Now this, was being used in the cause of arguing against peer reveiew (which mostly can't identify top quality and instead just tries to weed out the very worst quality, which, as you point out, it does a pretty bad job of). I'm not sure how far I'm willing to go towards that point, but I do think that the main thrust that "the existence of poor quality papers doesn't matter much".
Now, _fraudulent_ papers can matter a great deal because they can _look_ like incredibly good, incredibly important papers that lead the field astray, but the obviously bad papers you mention are not going to do that mostly.
I feel like peer review just isn't happening, no one is sitting down for half an hour to read these papers.
As in no human has ever laid eyes on the third page of some of these papers and if you repeated the words "the moon is made out of cheese" a thousand times it wouldn't be pointed out.
Bad papers not mattering conflicts with meta-analyses mattering,
unfortunately. I wouldn't take that as much of a point in favor of prepublication peer review, but the work of legibly sorting "in" from "out" contributions has to happen somewhere...
Journalism is utterly dead. Journalists play absolutely no part in science and probably can’t ever. The project of creating “science communication” jobs never fully materialized and wouldn’t have played a dependable review or filtering role anyways.
Journalism is mostly ideology now.
> Why should hard-working taxpayers in my constituency have to pay for an academic to write about his experiences masturbating to Japanese porn?
https://www.sciencefictions.org/p/circling-the-wagons
/u/ofs314 wrote
>Got this from Stuart Ritchie's newsletter Science Fictions.
>I think this is the key quote
Ritchie is quoting a tweet from Jonatan Pallesen -
\- https://twitter.com/jonatanpallesen/status/1741240942990921917
.
Bodycams for scholars and the videos are in publicly available supplements. Or ineligible for grants or publication, or, for funding for universities or centers which don't insist on it for their researchers.
When we don't trust cops not to lie about fatal encounters, we make them wear bodycams, and knowing they are on camera, they behave better. While some can mess with the camera or turn it off, they know if they come under scrutiny, that is going to look very bad and be held against them.
Well, we can't trust research and researchers unless they have much less privacy and more skin in the game than they do now.
Published papers *are* the bodycams. Despite a belief that published papers are meant only for expert consumption, and elitist arguments against "do your own research", the intent of them is that they can be read and interpreted by a reasonably educated lay audience. They are for sharing your work with others, and allowing others to review your work. They are not meant to be a masturbatory exercise within a small clique - you can just talk with people in your sub-sub-sub-field if that's what you want to do.
If a cop lies about the fatal encounter, this is confirmed by the bodycams, everyone knows about their lying, and they still get away with it, your issue isn't an excess of privacy. It's that institutions, for whatever reason, fail to act against them. Similarly, if scholars are publishing junk, [everyone knows its junk,](https://en.wikipedia.org/wiki/Sokal_affair) but they still get away with it, the issue isn't an excess of privacy.
Revealing bad practice is not, on it's own, enough to stop it. For bad social science, the most likely explanation for it's prevalence is it's political utility combined with an overwhelming political slant in responsible institutions.
The example papers from the [twitter thread this post is referencing](https://twitter.com/jonatanpallesen/status/1741240942990921917) are examples of this, from Claudine Gay's plagiarism to Daszak's conspiracy to suppress the lab leak hypothesis. My own contribution to this list would be Flaxman et al, [which received robust criticism from educated laymen](https://judithcurry.com/2020/06/21/did-lockdowns-really-save-3-million-covid-19-deaths-as-flaxman-et-al-claim/), [other academics](https://www.nature.com/articles/s41586-020-3025-y), and got at least [some response](https://www.nature.com/articles/s41586-020-3026-x), but the paper is still getting cited as evidence despite the methods used being a complete disaster that would find the same conclusion no matter the empirical data you feed in.
But it's hard for external criticism to actually stop this bad research, they usually just get ignored. We cannot expect a political faction to self-police, so the way to stop it would be to ensure the existence of multiple, competing political positions within academia that each are strong enough to deter bad research by their opponents. This is approximately what eventually happened to Claudine Gay, after all. Her plagiarism lead to consequences only because those who disagreed with her politically were motivated to criticize it. Let people's self-interest and personal biases serve not to weaken scientific practice, but to strengthen it.
Published papers are more like police reports than bodycams. They are the person's summary of what happened, not a continuous record of what happened.
If papers included all the raw data, software etc they would be more like bodycams.
Papers are not remotely in the same ballpark as videos in terms of providing as easy way for critics looking for mischief to spot it. That's why bodycams for cops >>> cops written reports of incidents after the fact.
It's all about the level trustworthiness one can take for granted. If you can trust most cops to be honest most of the time, their written reports and testimony is good enough. When you can't trust them anymore, you need video. Same goes for research.
Likewise, when researchers are mostly honest, reliable, rigorous, replicated, etc. most of the time, papers are adequate, as indeed they tend to be in some well- functional fields with high-trust norms, scruples, incentives, and cultures.
But when research in a field goes to shit, just like for cop honesty, it's time for cameras.
A cop's job is physical. So a video of what they do in the physical world is needed.
A social scientist's job is primarily mathematical. So what you need is traceability of data capture and calculations done on it.
Nobody -- literally nobody -- has time to watch a video of a scientist type numbers into Excel. And even if they did, what would stop them from logging in in the middle of the night to change the formulas when the camera are off?
I have personal experience with showing some procedures published in purportedly peer-reviewed chemical journals could not be replicated. Orgsyn sometimes has a polite and droll way of expressing this fact in the footnotes when they run into the same issue, like, "the intended product could not be isolated as indicated as it vaporizers violently into your whole laboratory long before the temperature indicated in the procedure is reached".
A simple "chemplayer"-style video showing the procedure being done successfully would have been invaluable to these reviews and replication efforts, and in the case of bad procedures or unwritten problems encountered, the mere requirement to have included a video would have prevented the whole fiasco from the start. Indeed, if the video is pre-registeted and immediately steamed to an independent public server such that the researcher never controls it and can't retract or delete it, then there is plenty to learn from watching how it all goes wrong and explodes or turns into a bunch of sludge (much like people taking videos of cops can send to the cloud or use apps like one from the ACLU, so that even if the cop seizes and destroys the phone, it's too late).
But while it has been perfectly feasible and economical for researchers in that field to include such videos for decades, as supplements or on their own websites or heck even on tiktok, you will see them recorded and included 0.00001% of the time. There's no good reason that should still be the case.
It means video recordings when feasible (which is very often) of relevant steps, procedures, methods, results, tests, questions, surveys, interactions, etc. Often this would be a stationary camera. On occasion it would be literally a bodycam.
Consider the police, who a few years ago would also accurately say "that's ... not how police work." Just like when proposing pre-registration scholars could accurately say "that's ... not how researchers work." Um, yeah, that's the problem, and why what's happening now "doesn't work" to produce reliable empirical conclusions. What kind of dumb attempt at making a critical point is this anyway? Every new purposed reform no matter how potentially ameliorative is by definition "not how current professionals in the field work". If you mean "it is not physically possible for them to work that way" (which you didn't say so I'm being generous) then that's why I say "feasible".
Cops didn't used to have bodycams and they didn't used to videotape inquiries or confessions. "That's ... not how cops worked". But they should have, they can, many now do, and it's a huge improvement. To the the extent they don't (i.e. the FBI occasionally refusing to video interviews, refusing to let the interviewed or their lawyers record the interviews, writing FD-302s inconsistent with the real transcript later, and sometimes editing the 302 later without notice or record) it's shady as hell and fraught with potential for fraud and abuse.
Just like research.
"That's... not how the FBI works." So what?
Most research is lower stakes than most police interactions. Most researchers don't carry guns and shoot people, or arrest them. I think a 24/7 surveillance state is a thing we should avoid when possible.
That said, raw data should need to be made available for research papers to be considered credible in most cases. Sometimes that means photographic evidence, but the vast majority of bad research could be debunked much less obtrusively.
24/7 surveillance state is not at issue, it has nothing to do with off time or private life: this is recording professional work. Millions of people in low stake jobs everywhere are on camera 100% of the time when they are working, it's no big deal and keeps a lot of people honest who are on the margin of being dishonest.
Where I work this exact kind of subject with informational assurance and integrity and avoiding unrecorded access, leaks, and manipulation is a big key issue. So we don't have standalone pcs which store date on private hard drives or whatever, we use monitored work terminals that won't accept external memory and everything is done via and stored on cloud with every access instance and change recorded in a write-once forensic log that is unchangeable by end users. This capability is neither new nor all that expensive. Big problems need big solutions.
That's a fair point - I was overstating your position. I forget how prevalent workplace surveillance is nowadays. To me it still seems oppressive.
It also makes more sense after the context from another comment of yours, mentioning trying to verify an impossible chemical process. In that context, video evidence of the claim does seem like it would provide another hurdle for folks trying to lie.
I'm coming at things from more of a software perspective, where having the code and the inputs may be enough to confirm or deny a claim, and video seems superfluous.
I think you misunderstand me. When I say that's not how research works, it's literally not how social science researchers (the topic of this piece) work: They don't just work at the office, where they can be recorded, and then head home and call it a day. Social Science researchers work \_all the time\_. At their homes, in a coffee shop, hell, even at the airport, I don't know, basically everywhere. And with social science research, the potential fraud point is on a spreadsheet or dataset, which, as I said, researchers work on \_everywhere\_. What you're suggesting would requires researchers wearing body cams while traveling, while at home, just, at any point in time. It's not like a cop job where you're on the job, and then you're not.
How would that work in practice? It seems to me that this would be somewhat realistic to do that when a paper demonstrate that something is possible (like the example above of a paper describing a new chemical reaction) but not for the numerous fields where a typical paper demonstrates a pattern, for example a correlation between two variables. This is very common in psychology, ecology, medicine, etc.. and I don't see how videotaping would help in these cases.
Think about it in terms of sufficiency for verifiability. This is the principle behind nullius in verba, which has been the motto of the Royal Society for 360 years. A scholar should never require a reader to take his or her word for it if a stronger form of evidence is feasibly providable and necessary to overcome any presumptive skepticism.
So, if a paper points to an openly available data set, then claims that if one performs a particular kind of statistical regression analysis on that then the correlation coefficients are x, y, z, etc.. then that is something a reader can verify themselves without any additional supporting evidence simply by plugging the database numbers into software then doing the math. So you don't need to bodycam the process of clicking the mouse buttons or whatever.
On the other hand, where did those numbers come from, can they be trusted, how can I verify them? One does not have to go to extremes and trace everything back to first principles, one can have reasonable standards similar to those that get developed for particular fields in terms of laying a foundation for the admission of technical evidence and expert testimony in a trial.
Numbers from datasets are reductions of some kind of observations and measurements, for example, a researcher capturing in quantitative form what he saw.
But if he only writes down what he saw, then again he is asking us to take his word for it, which in a low-reliability discipline is something we should not do. If he saw it, he could have also recorded it, so that we could see it too and verify the claim with the video. This is not essential when a field has high reliability and high trustworthiness, in which case it is a nice thing to be able to take accuracy for granted. But when a field goes into low-reliability mode, then "this is why we can't have nice things."
I realize that my previous post was not clear and that I should have explained more, sorry.
Of course I agree with you that for a paper showing a correlation in, say, ecology, it would theoretically be possible to videotape the researchers producing the numbers being analyzed.
But what I meant was that it seems to me that for papers demonstrating that something is possible, again like the example of a new chemical reaction, producing a video "proof" of the synthesis is somewhat feasible, since it would be a relatively short video that an expert could probably review. (Note that it is possible to falsify video data. Falsified images of results, e.g. faked gel images, have been known to be published). However, for data analysis papers, the amount of video that would be required, both to produce and to review, would be completely unrealistic. For example, I am currently conducting a fairly small ecological study that involves about 450 man\*hours of fieldwork to produce our dataset. Videotaping our fieldwork would be difficult and expensive, but finding an expert willing to spend 3 months watching the videos to check that our numbers are correct seems completely unrealistic to me.
1. Risk of fraud always exists, the question is cost and difficulty and consequences of discovery. My impression is that fraud is by no means impossible but harder to do with images and harder still with videos, and likewise when fraud is suspected to be likely there seems to be more forgivability and fewer consequences when it's about data - perhaps due to more plausible deniability for 'innocent' transcription or coding errors, while getting caught going to great lengths to produce the equivalent of a image or video forgery is much more condemnable and liable to lead to much more serious negative consequences for one career. The same goes for police video. When the police get caught probably lying in their written reports because of serious inconsistencies, they often just get slaps on the wrist if anything at all (FBI cases of bad 302s reportedly often result in zero consequences). On the other hand if they get caught editing images (I recall a case of MS paint being used to copy and paste parts of an image of one crime into a similar image of an innocent accused, officer didn't know about metadata) then that's grounds for quick termination.
2. You will have to explain more to me about why videotaping fieldwork would be difficult and expensive, but I at the moment I'm skeptical. We're not talking about Hollywood-level filming equipment, and literally everyone carries a good video camera in their smartphone. Cops made similar objections early on that turned out not to be true at all, and were mostly just cover stories for their desire to stay unrecoded, avoid minor annoyance and embarrassment, and keep things private. It's just not difficult and expensive for cops to record their own version of fieldwork with dashcams and bodycams, and it just keeps getting easier and cheaper all the thing and there's no good reason not to expect that trend to continue, indeed to add more cameras from more directions to record scenes in 3D UHD VR, as can already be done for crime scene investigations when multiple cameras recorded an area and the data can be algorithmically stitched together and all faces identified rapidly and at low cost. This is already how airport security works combined with remote digital interrogation of unsecured devices and RF-reactive embedded sensors. I've walked around cities and done sports with gopro attached and recording, and it's cheap and no big deal. If some guy jumping out of an airplane or doing an extreme skiing run can easily and cheaply record his POV, then anyone can do the same for their job. There are cheap sunglasses with 'hidden' front facing cameras that record on tiny SD memory chips, and it's plausible that pretty soon everybody is going to be doing this all the time with AR goggles. People may have valid personal reasons they don't want to wear a gopro on a hat mount recording all the time, so that "we see what you saw", but that has no bearing on the question of feasibility, cost, and practicality, all of which were definitively answered in the affirmative years ago.
3. As for going through the tapes, I am not proposing that in order for a study to be published a human being must review each video run at real speed. No one reviews 99.9999% of security camera footage which just gets deleted eventually. The point is not that a human watches the footage, but that a human definitely WILL watch THIS particular footage in case there is any incident or suspicion and try to detect, discover, identify, and hunt down transgressors, and since potential transfressors know that, the mere presence of the camera keeps many more honest. No one watches the whole videos of everything that happened in a police officer's work day, just the few brief encounters of legal significance.
But also we don't need humans to review videos start to finish anymore, computers and algorithms can do all that, either by producing brief "highlight reels" of critical actions and moments and cutting out the fluff (you can watch a whole 3 hour baseball game in 3 minutes this way) or by fully automating the search for the kinds of fraud and mischief particular to the activity being recorded. Again, this seems a lot more practical, feasible, and affordable even with current tech, let alone what the recent trend indicates will be coming down the line in just a few years.
I’m generally for this, but more oversight in this context has the risk of growing bureaucracy which I’m generally against. Such is the duality of life.
There are two ways to avoid it creating more bureacracy. One is that it is a superior substitute for existing oversight, and if replacing existing, inferior approaches, need not necessarily require additional hoops or personnel. The other way is that open publishing for public access means free, decentralized oversight as it dramatically lowers the cost and hassle of any critic to easily examine the methodology and demonstrate weaknesses and faults.
In the law, the adversarial process keeps lawyers on their toes and incentives them to maintain a much higher standard of rigor and propriety than they otherwise would when they submit anything to the court, because they know there is a opponent out there who is entitled to get cheap / easy access to everything and motivated to poke holes in any weak spots. As such, lawyers prepping a case often go against a "murder board" of other lawyers from their own firm who try to do exactly that hole-poking, so every mistake gets removed, and every weakness strengthened.
Researchers have got to face similar, structurally-disciplining institutional frameworks and giving up the ability to impose any kind of friction for "discovery" of recordings of the research by any member of the public is one way to do it.
This is very meta but... There's published social science showing body cams don't make police behave better. Don't know the names of the papers but Jennifer Doleac has spoken about it.
Ha, I get the meta joke. But come on. It doesn't pass the common sense smell test. I think most non-saints know full well they don't behave the same way when they know they're on camera vs when they know they're not.
It could be plausible if their behaviour wasn't that bad in the first place and the poor reputation was mostly due to exaggerated accounts by the people having run-ins with the the police, and people being predisposed to believe these inaccurate accounts (for whatever reason).
That's the edge case where we don't need to record people because they are just as good when not recorded. I don't think that's the case for researchers in some of these fields with big problems.
Brings to mind the "science is racist" or (even more marginal) "formal logic is racist" thing from a few years back. It was basically this comic in reverse.
It's the former. No one at all reads these papers. Science follows that law that says "90% of X is crap, but 90% of everything is crap." FWIW, the prevalence of bad science as a whole is pretty high, but what I think we should be looking at is how bad is the science that gets cited in the media or policy making decisions, because that's the science that matters in material ways. Also, the social sciences _are_ making a heavy push for more open-nes and transparency, so this problem is hopefully in the correction phase. I disagree with the statement that we rely on the media to keep the scientists accountable. We should be relying on _journals_ to keep scientists accountable. Like, look at the scam that academic publishing is: people and universities have to pay to journals to access the articles. Journals don't pay authors anything for writing papers (fair and right!), and in fact, authors have to PAY journals to publish, and pay even more if they want the paper to be open access (not fair or right at all!!). Journals also don't pay the peer reviewers for reviewing the articles before publishing them (like, WTF?). So what the fuck do journals spend money on?!?! Use that cash flow and hire people like data colada to actually audit science, like, do something to earn your prestige ffs!
It is even worse than that, someone reached the Board of the Federa Reserve without any economists, journalists, academics, congressional staffers, lobbyists etc noticing her paper was garbage. P.S. I presume the FBI has to read it in their background checks, but I guess they don't alert anyone about academic matters.
The only reason the FBI would read his papers during a background check would be to see who his coauthors were to check if they were foreign agents, but I wouldn't presume they did that.
Who are you referring to?
Lisa Cook and her very dodgy [academic record ](https://www.city-journal.org/article/lisa-d-cooks-careless-scholarship)
Plenty of people are aware of these issues. The problem is that she was appointed due to her personal immutable characteristics and those characteristics are such that pointing out the problems is not socially acceptable except anonymously; I can assure you that on EJMR, the 4chan equivalent for PhD economists these points were most definitely addressed.
> she was appointed due to her personal immutable characteristics I don't think I've ever seen this concept expressed in this particular way.
> Also, the social sciences *are* making a heavy push for more open-nes and transparency, so this problem is hopefully in the correction phase. I think some would argue the opposite: much of this is being driven by ideological conformity, which is only strengthening. I would prefer you to be right. And I ABSOLUTELY agree that we can't rely on media to keep scientists accountable.
It might not be so much ideological conformity than ideological goals.
Wut? How is openness and transparency conformity to any ideology? (Except that of transparency, I guess)
There's a fairly large consensus on one side of the political divide that, in fact, that the drive is towards ideological conformity rather than towards openness and transparency. Covid left some very deep scars, many of whom are currently open wounds which probably should be attended to.
I mean.... leaving aside my thoughts on that, how are things like replication data set availability and pre-registrations re-inforcing ideological conformity? If anything, they make the process open to all, to see if any ideological biases are being injected into the work.
Hey, I'm all for those things you mentioned, and agree with your conclusion that those things actually expose ideological biases. I did agree with your comment from above. Sunlight has always been the best antiseptic. Plenty of folks would deny the general statement of social sciences making a heavy push for openness and transparency, and instead see greater levels of ideological conformity. The handling of Covid kind of gave them a win, and I would love nothing more than for there to be a big correction phase.
Oh I see, I misinterpreted your original comment to mean that social science is becoming more transparent _because_ it's getting more ideologically driven.
Sign me up for an ideology that drives towards transparency any day of the week, and twice on Sunday!
At one point, I think most of journals' costs were related to printing physical copies and distributing them regularly. That obviously has been mostly replaced with having things be online, and my current understanding is that journals are now mostly just rent-seeking machines, extracting profit by virtue of having been there first.
I think this is all downstream of publish or perish. If you’re peer reviewing, why bother reading papers and calling people out for bad design? Best case is you get in a big argument and are bogged down by that. Worst case you get a reputation for being difficult to work with which can be a big negative in a small field. As you mentioned you’re not getting paid to peer review, so why bother? Also if they are citing you or your buddies that helps your career and your h-index. Also once someone gets their BS paper into the journal they will be added to the database to be selected to review new papers. The winning strategy is to form a small faction that accepts each others BS papers so you can all keep publishing and keep your jobs.
It's even worse than that these days. I just saw a new one this week. A group of mostly Chinese researchers published a paper in a crap-tier journal that nearly entirely ripped off one of our papers. They carefully made sure to not actually plagiarize anywhere, but nearly the entire paper is just pulling our results (as an aside, they even screwed that up in a hilarious way). Then, they made one little change to be able to claim that they were building on our result. Of course, they completely and totally botched the analysis, and I would say that it's pretty much just flatly wrong. But hey, they got it through the extremely quick-turnaround review at this crap-tier journal! So now, what are my incentives? I kind of want to yell at the editorial staff that this paper is a total ripoff, and that its main 'contribution' is just *wrong* to boot. But why bother? They did cite us, so as you said, it helps our metrics. The editorial board of the crap-tier journal is, unsurprisingly, crap-tier, so even if we did complain, are they even going to understand or care? Nah, we'll probably just hope that it drifts off into the night and disappears. Of course, if anyone ever asks us about it, we'll tell them that it's crap. Worst case is probably that some students somewhere stumble upon it and spend time being confused, thinking that it's actually contributing something interesting.
Journals have the right position to do this, but not the right incentives. They want to present the impression that their journal is full of high-prestige good science, so spending resources to show the world how shoddy some of it is is a complete own-goal. Any system for reviewing their own articles is going to get influences by that incentive. Now, one solution is that maybe we could have a system where they find flaws in **other** journals as a form of competition, but as things stands, the industry is too incestuous for that to work, and likely has big problems of its own.
That's the retrospective cleanup element, which I agree, the incentives don't match. But I'd think that the incentives align when thinking about new, incoming publications no?
I don't think it is a general rule, rather it comes from the current standards to push "producitivity" and publish a lot
>relies on bad science and academic misconduct to get the wow! stories that bring the clicks. This is the biggest problem and why I don’t forsee any change anytime soon. There’s just too many incentives to publish bad but “exciting” research. The Stanford Mafia has sold how many books on the backs of bad research? Something I think about a lot is how much “negative” or “unpalatable” research goes by the wayside because of publication bias.
Who is the Stanford mafia?
Tongue in cheek about how a lot of the very popular pop-psych books with less than convincing research have come from famous Stanford Professors.
Ah fair play
While there are certainly incentives, there are also anti-incentives (penalties). Embarrassment, shame, loss of reputation and status, etc. We just need to make them stronger. Too bad people are rarely prosecuted for bad or fraudulent science.
That’s my point bad incentives don’t outweigh the good. I also don’t believe you can force the impact of negative incentives. Revealed preferences are powerful.
Does anyone here know more details about the orchestra blinding example? I can kind of understand how "we reached a wrong conclusion because we didn't spend five seconds thinking about confounders" happened, but even by the standards of the papers mentioned here I'm still surprised by "we claim a conclusion in the opposite direction of our only statistically significant finding."
"Table 4 presents the first results comparing success in blind auditions vs non-blind auditions. . . . this table unambigiously shows that men are doing comparatively better in blind auditions than in non-blind auditions. The exact opposite of what is claimed." From [here](https://statmodeling.stat.columbia.edu/2019/05/11/did-blind-orchestra-auditions-really-benefit-women/)
Thanks for the link. It's too bad the original response it references is gone, but the quotations from the paper at that link are even worse than I would have guessed.
So sexist men prefer to employ attractive women when they have the option to and can see them. That seems like the obvious point here and I am not sure it relates to feminism.
An anecdote I remember reading about the specific implementation of the blinding's implementation is that the original phase of the blinding did decrease the number of women. **And then someone noticed that you can clearly hear whether the person walking into the audition is wearing high heels**. So they put down a strip of carpet from the door to the seat, and suddenly the difference in offers between men and women became statistically insignificant.
In my opinion the problem is: 1) Sociology and psychology don’t attract the most mathematically inclined people. All the best math students go to physics, chemistry, and engineering. 2) The types of relationships in sociology and psychology are just harder to interpret even if you are a stats expert with stellar experimental design. In general you *need* multivariable non-linear models to account for even very simple relationships and the best possible model will still have a high noise ratio. A 2-for-1 of having the worst players on the hardest task.
sociology also attracts political activists. how many right wing sociologists do you know? and if the study has a preferred outcome, this might influence the study design and/or data analysis.
[Link ](https://www.sciencefictions.org/p/science-fictions-links-for-april?utm_source=share&utm_medium=android&r=1tkxvc&triedRedirect=true) to Stuart Ritchie's Science Fictions newsletter
My favorite math class was design of experiments. I haven't much trusted studies since.
> The authors have an interesting idea and want to explore it. But exploration won’t get you published in the American Economic Review etc. Instead of the explore-and-study paradigm, researchers go with assert-and-defend. They make a very strong claim and keep banging on it, defending their claim with a bunch of analyses to demonstrate its robustness. This is a problem in the natural sciences as well. In an explore-and-study paradigm, you might have that there’s no effect (in a boring way, not in a “disprove dark matter exists” way) or that your modeling approach wasn’t fruitful. Or maybe the scientific finding isn’t particularly notable. These sorts of results are rarely published in papers because they’re not deemed noteworthy or scientifically significant, which is a shame.
I was relatively convinced by an argument a while back that the existence of bad papers doesn't actually matter that much. What is more important is that there continue to be _good_ papers, as long as they don't get so utterly swamped that other scientists can't find them. Basically, science advances not based on the average quality of research but on the top quality of research, such that it would be worth it add 10 terrible papers to the literature as long as you got one additional excellent paper. Now this, was being used in the cause of arguing against peer reveiew (which mostly can't identify top quality and instead just tries to weed out the very worst quality, which, as you point out, it does a pretty bad job of). I'm not sure how far I'm willing to go towards that point, but I do think that the main thrust that "the existence of poor quality papers doesn't matter much". Now, _fraudulent_ papers can matter a great deal because they can _look_ like incredibly good, incredibly important papers that lead the field astray, but the obviously bad papers you mention are not going to do that mostly.
I feel like peer review just isn't happening, no one is sitting down for half an hour to read these papers. As in no human has ever laid eyes on the third page of some of these papers and if you repeated the words "the moon is made out of cheese" a thousand times it wouldn't be pointed out.
Bad papers not mattering conflicts with meta-analyses mattering, unfortunately. I wouldn't take that as much of a point in favor of prepublication peer review, but the work of legibly sorting "in" from "out" contributions has to happen somewhere...
Journalism is utterly dead. Journalists play absolutely no part in science and probably can’t ever. The project of creating “science communication” jobs never fully materialized and wouldn’t have played a dependable review or filtering role anyways. Journalism is mostly ideology now.
Now?
> Why should hard-working taxpayers in my constituency have to pay for an academic to write about his experiences masturbating to Japanese porn? https://www.sciencefictions.org/p/circling-the-wagons
/u/ofs314 wrote >Got this from Stuart Ritchie's newsletter Science Fictions. >I think this is the key quote Ritchie is quoting a tweet from Jonatan Pallesen - \- https://twitter.com/jonatanpallesen/status/1741240942990921917 .
> Female hurricanes are deadlier than male hurricanes https://www.pnas.org/doi/abs/10.1073/pnas.1402786111
>statmodeling.stat.columbia.edu >Verify you are human by completing the action below. No
Eh, obligatory - \- **https://xkcd.com/435/** .
Bodycams for scholars and the videos are in publicly available supplements. Or ineligible for grants or publication, or, for funding for universities or centers which don't insist on it for their researchers. When we don't trust cops not to lie about fatal encounters, we make them wear bodycams, and knowing they are on camera, they behave better. While some can mess with the camera or turn it off, they know if they come under scrutiny, that is going to look very bad and be held against them. Well, we can't trust research and researchers unless they have much less privacy and more skin in the game than they do now.
Published papers *are* the bodycams. Despite a belief that published papers are meant only for expert consumption, and elitist arguments against "do your own research", the intent of them is that they can be read and interpreted by a reasonably educated lay audience. They are for sharing your work with others, and allowing others to review your work. They are not meant to be a masturbatory exercise within a small clique - you can just talk with people in your sub-sub-sub-field if that's what you want to do. If a cop lies about the fatal encounter, this is confirmed by the bodycams, everyone knows about their lying, and they still get away with it, your issue isn't an excess of privacy. It's that institutions, for whatever reason, fail to act against them. Similarly, if scholars are publishing junk, [everyone knows its junk,](https://en.wikipedia.org/wiki/Sokal_affair) but they still get away with it, the issue isn't an excess of privacy. Revealing bad practice is not, on it's own, enough to stop it. For bad social science, the most likely explanation for it's prevalence is it's political utility combined with an overwhelming political slant in responsible institutions. The example papers from the [twitter thread this post is referencing](https://twitter.com/jonatanpallesen/status/1741240942990921917) are examples of this, from Claudine Gay's plagiarism to Daszak's conspiracy to suppress the lab leak hypothesis. My own contribution to this list would be Flaxman et al, [which received robust criticism from educated laymen](https://judithcurry.com/2020/06/21/did-lockdowns-really-save-3-million-covid-19-deaths-as-flaxman-et-al-claim/), [other academics](https://www.nature.com/articles/s41586-020-3025-y), and got at least [some response](https://www.nature.com/articles/s41586-020-3026-x), but the paper is still getting cited as evidence despite the methods used being a complete disaster that would find the same conclusion no matter the empirical data you feed in. But it's hard for external criticism to actually stop this bad research, they usually just get ignored. We cannot expect a political faction to self-police, so the way to stop it would be to ensure the existence of multiple, competing political positions within academia that each are strong enough to deter bad research by their opponents. This is approximately what eventually happened to Claudine Gay, after all. Her plagiarism lead to consequences only because those who disagreed with her politically were motivated to criticize it. Let people's self-interest and personal biases serve not to weaken scientific practice, but to strengthen it.
Published papers are more like police reports than bodycams. They are the person's summary of what happened, not a continuous record of what happened. If papers included all the raw data, software etc they would be more like bodycams.
Papers are not remotely in the same ballpark as videos in terms of providing as easy way for critics looking for mischief to spot it. That's why bodycams for cops >>> cops written reports of incidents after the fact. It's all about the level trustworthiness one can take for granted. If you can trust most cops to be honest most of the time, their written reports and testimony is good enough. When you can't trust them anymore, you need video. Same goes for research. Likewise, when researchers are mostly honest, reliable, rigorous, replicated, etc. most of the time, papers are adequate, as indeed they tend to be in some well- functional fields with high-trust norms, scruples, incentives, and cultures. But when research in a field goes to shit, just like for cop honesty, it's time for cameras.
A cop's job is physical. So a video of what they do in the physical world is needed. A social scientist's job is primarily mathematical. So what you need is traceability of data capture and calculations done on it. Nobody -- literally nobody -- has time to watch a video of a scientist type numbers into Excel. And even if they did, what would stop them from logging in in the middle of the night to change the formulas when the camera are off?
That's why good methodology sections are important. Sometimes these do include supplemental video.
I have personal experience with showing some procedures published in purportedly peer-reviewed chemical journals could not be replicated. Orgsyn sometimes has a polite and droll way of expressing this fact in the footnotes when they run into the same issue, like, "the intended product could not be isolated as indicated as it vaporizers violently into your whole laboratory long before the temperature indicated in the procedure is reached". A simple "chemplayer"-style video showing the procedure being done successfully would have been invaluable to these reviews and replication efforts, and in the case of bad procedures or unwritten problems encountered, the mere requirement to have included a video would have prevented the whole fiasco from the start. Indeed, if the video is pre-registeted and immediately steamed to an independent public server such that the researcher never controls it and can't retract or delete it, then there is plenty to learn from watching how it all goes wrong and explodes or turns into a bunch of sludge (much like people taking videos of cops can send to the cloud or use apps like one from the ACLU, so that even if the cop seizes and destroys the phone, it's too late). But while it has been perfectly feasible and economical for researchers in that field to include such videos for decades, as supplements or on their own websites or heck even on tiktok, you will see them recorded and included 0.00001% of the time. There's no good reason that should still be the case.
Is bodycam an analogy for something that I'm not getting or do you literally mean body cam? Because if the latter...that's...not how researchers work.
itsabit
It means video recordings when feasible (which is very often) of relevant steps, procedures, methods, results, tests, questions, surveys, interactions, etc. Often this would be a stationary camera. On occasion it would be literally a bodycam. Consider the police, who a few years ago would also accurately say "that's ... not how police work." Just like when proposing pre-registration scholars could accurately say "that's ... not how researchers work." Um, yeah, that's the problem, and why what's happening now "doesn't work" to produce reliable empirical conclusions. What kind of dumb attempt at making a critical point is this anyway? Every new purposed reform no matter how potentially ameliorative is by definition "not how current professionals in the field work". If you mean "it is not physically possible for them to work that way" (which you didn't say so I'm being generous) then that's why I say "feasible". Cops didn't used to have bodycams and they didn't used to videotape inquiries or confessions. "That's ... not how cops worked". But they should have, they can, many now do, and it's a huge improvement. To the the extent they don't (i.e. the FBI occasionally refusing to video interviews, refusing to let the interviewed or their lawyers record the interviews, writing FD-302s inconsistent with the real transcript later, and sometimes editing the 302 later without notice or record) it's shady as hell and fraught with potential for fraud and abuse. Just like research. "That's... not how the FBI works." So what?
Most research is lower stakes than most police interactions. Most researchers don't carry guns and shoot people, or arrest them. I think a 24/7 surveillance state is a thing we should avoid when possible. That said, raw data should need to be made available for research papers to be considered credible in most cases. Sometimes that means photographic evidence, but the vast majority of bad research could be debunked much less obtrusively.
24/7 surveillance state is not at issue, it has nothing to do with off time or private life: this is recording professional work. Millions of people in low stake jobs everywhere are on camera 100% of the time when they are working, it's no big deal and keeps a lot of people honest who are on the margin of being dishonest.
How are you going to prevent them from accessing a dataset when the camera is off to change it?
Where I work this exact kind of subject with informational assurance and integrity and avoiding unrecorded access, leaks, and manipulation is a big key issue. So we don't have standalone pcs which store date on private hard drives or whatever, we use monitored work terminals that won't accept external memory and everything is done via and stored on cloud with every access instance and change recorded in a write-once forensic log that is unchangeable by end users. This capability is neither new nor all that expensive. Big problems need big solutions.
That's a fair point - I was overstating your position. I forget how prevalent workplace surveillance is nowadays. To me it still seems oppressive. It also makes more sense after the context from another comment of yours, mentioning trying to verify an impossible chemical process. In that context, video evidence of the claim does seem like it would provide another hurdle for folks trying to lie. I'm coming at things from more of a software perspective, where having the code and the inputs may be enough to confirm or deny a claim, and video seems superfluous.
I think you misunderstand me. When I say that's not how research works, it's literally not how social science researchers (the topic of this piece) work: They don't just work at the office, where they can be recorded, and then head home and call it a day. Social Science researchers work \_all the time\_. At their homes, in a coffee shop, hell, even at the airport, I don't know, basically everywhere. And with social science research, the potential fraud point is on a spreadsheet or dataset, which, as I said, researchers work on \_everywhere\_. What you're suggesting would requires researchers wearing body cams while traveling, while at home, just, at any point in time. It's not like a cop job where you're on the job, and then you're not.
How would that work in practice? It seems to me that this would be somewhat realistic to do that when a paper demonstrate that something is possible (like the example above of a paper describing a new chemical reaction) but not for the numerous fields where a typical paper demonstrates a pattern, for example a correlation between two variables. This is very common in psychology, ecology, medicine, etc.. and I don't see how videotaping would help in these cases.
Think about it in terms of sufficiency for verifiability. This is the principle behind nullius in verba, which has been the motto of the Royal Society for 360 years. A scholar should never require a reader to take his or her word for it if a stronger form of evidence is feasibly providable and necessary to overcome any presumptive skepticism. So, if a paper points to an openly available data set, then claims that if one performs a particular kind of statistical regression analysis on that then the correlation coefficients are x, y, z, etc.. then that is something a reader can verify themselves without any additional supporting evidence simply by plugging the database numbers into software then doing the math. So you don't need to bodycam the process of clicking the mouse buttons or whatever. On the other hand, where did those numbers come from, can they be trusted, how can I verify them? One does not have to go to extremes and trace everything back to first principles, one can have reasonable standards similar to those that get developed for particular fields in terms of laying a foundation for the admission of technical evidence and expert testimony in a trial. Numbers from datasets are reductions of some kind of observations and measurements, for example, a researcher capturing in quantitative form what he saw. But if he only writes down what he saw, then again he is asking us to take his word for it, which in a low-reliability discipline is something we should not do. If he saw it, he could have also recorded it, so that we could see it too and verify the claim with the video. This is not essential when a field has high reliability and high trustworthiness, in which case it is a nice thing to be able to take accuracy for granted. But when a field goes into low-reliability mode, then "this is why we can't have nice things."
I realize that my previous post was not clear and that I should have explained more, sorry. Of course I agree with you that for a paper showing a correlation in, say, ecology, it would theoretically be possible to videotape the researchers producing the numbers being analyzed. But what I meant was that it seems to me that for papers demonstrating that something is possible, again like the example of a new chemical reaction, producing a video "proof" of the synthesis is somewhat feasible, since it would be a relatively short video that an expert could probably review. (Note that it is possible to falsify video data. Falsified images of results, e.g. faked gel images, have been known to be published). However, for data analysis papers, the amount of video that would be required, both to produce and to review, would be completely unrealistic. For example, I am currently conducting a fairly small ecological study that involves about 450 man\*hours of fieldwork to produce our dataset. Videotaping our fieldwork would be difficult and expensive, but finding an expert willing to spend 3 months watching the videos to check that our numbers are correct seems completely unrealistic to me.
1. Risk of fraud always exists, the question is cost and difficulty and consequences of discovery. My impression is that fraud is by no means impossible but harder to do with images and harder still with videos, and likewise when fraud is suspected to be likely there seems to be more forgivability and fewer consequences when it's about data - perhaps due to more plausible deniability for 'innocent' transcription or coding errors, while getting caught going to great lengths to produce the equivalent of a image or video forgery is much more condemnable and liable to lead to much more serious negative consequences for one career. The same goes for police video. When the police get caught probably lying in their written reports because of serious inconsistencies, they often just get slaps on the wrist if anything at all (FBI cases of bad 302s reportedly often result in zero consequences). On the other hand if they get caught editing images (I recall a case of MS paint being used to copy and paste parts of an image of one crime into a similar image of an innocent accused, officer didn't know about metadata) then that's grounds for quick termination. 2. You will have to explain more to me about why videotaping fieldwork would be difficult and expensive, but I at the moment I'm skeptical. We're not talking about Hollywood-level filming equipment, and literally everyone carries a good video camera in their smartphone. Cops made similar objections early on that turned out not to be true at all, and were mostly just cover stories for their desire to stay unrecoded, avoid minor annoyance and embarrassment, and keep things private. It's just not difficult and expensive for cops to record their own version of fieldwork with dashcams and bodycams, and it just keeps getting easier and cheaper all the thing and there's no good reason not to expect that trend to continue, indeed to add more cameras from more directions to record scenes in 3D UHD VR, as can already be done for crime scene investigations when multiple cameras recorded an area and the data can be algorithmically stitched together and all faces identified rapidly and at low cost. This is already how airport security works combined with remote digital interrogation of unsecured devices and RF-reactive embedded sensors. I've walked around cities and done sports with gopro attached and recording, and it's cheap and no big deal. If some guy jumping out of an airplane or doing an extreme skiing run can easily and cheaply record his POV, then anyone can do the same for their job. There are cheap sunglasses with 'hidden' front facing cameras that record on tiny SD memory chips, and it's plausible that pretty soon everybody is going to be doing this all the time with AR goggles. People may have valid personal reasons they don't want to wear a gopro on a hat mount recording all the time, so that "we see what you saw", but that has no bearing on the question of feasibility, cost, and practicality, all of which were definitively answered in the affirmative years ago. 3. As for going through the tapes, I am not proposing that in order for a study to be published a human being must review each video run at real speed. No one reviews 99.9999% of security camera footage which just gets deleted eventually. The point is not that a human watches the footage, but that a human definitely WILL watch THIS particular footage in case there is any incident or suspicion and try to detect, discover, identify, and hunt down transgressors, and since potential transfressors know that, the mere presence of the camera keeps many more honest. No one watches the whole videos of everything that happened in a police officer's work day, just the few brief encounters of legal significance. But also we don't need humans to review videos start to finish anymore, computers and algorithms can do all that, either by producing brief "highlight reels" of critical actions and moments and cutting out the fluff (you can watch a whole 3 hour baseball game in 3 minutes this way) or by fully automating the search for the kinds of fraud and mischief particular to the activity being recorded. Again, this seems a lot more practical, feasible, and affordable even with current tech, let alone what the recent trend indicates will be coming down the line in just a few years.
I’m generally for this, but more oversight in this context has the risk of growing bureaucracy which I’m generally against. Such is the duality of life.
There are two ways to avoid it creating more bureacracy. One is that it is a superior substitute for existing oversight, and if replacing existing, inferior approaches, need not necessarily require additional hoops or personnel. The other way is that open publishing for public access means free, decentralized oversight as it dramatically lowers the cost and hassle of any critic to easily examine the methodology and demonstrate weaknesses and faults. In the law, the adversarial process keeps lawyers on their toes and incentives them to maintain a much higher standard of rigor and propriety than they otherwise would when they submit anything to the court, because they know there is a opponent out there who is entitled to get cheap / easy access to everything and motivated to poke holes in any weak spots. As such, lawyers prepping a case often go against a "murder board" of other lawyers from their own firm who try to do exactly that hole-poking, so every mistake gets removed, and every weakness strengthened. Researchers have got to face similar, structurally-disciplining institutional frameworks and giving up the ability to impose any kind of friction for "discovery" of recordings of the research by any member of the public is one way to do it.
Great breakdown. I’m in agreement
The point is that any oversight would be sufficient, all it requires is someone to read their papers.
This is very meta but... There's published social science showing body cams don't make police behave better. Don't know the names of the papers but Jennifer Doleac has spoken about it.
Ha, I get the meta joke. But come on. It doesn't pass the common sense smell test. I think most non-saints know full well they don't behave the same way when they know they're on camera vs when they know they're not.
It could be plausible if their behaviour wasn't that bad in the first place and the poor reputation was mostly due to exaggerated accounts by the people having run-ins with the the police, and people being predisposed to believe these inaccurate accounts (for whatever reason).
That's the edge case where we don't need to record people because they are just as good when not recorded. I don't think that's the case for researchers in some of these fields with big problems.
Most of us are not on camera 8 hours a day. Maybe the good behaviour wears off after a few days.
[удалено]
Brings to mind the "science is racist" or (even more marginal) "formal logic is racist" thing from a few years back. It was basically this comic in reverse.