This is the same behaviour I've seen time and time again in biology labs.
People there are re-doing the same experiment over and over until it gives them the result they want, and then they publish that. It's the only field where I've heard people saying "Oh, yeah, my experiment failed, I have to do it again". What does it even mean that an experiment failed? It did exactly what it was supposed to: it gave you data. It didn't fit your expectations? Good, now you have a tool to refine your expectations. But instead, we see PhD students and post-doc working 70h hours week on experiments with seemingly random results until the randomness goes their way.
A lot of them have no clue about statistical treatment of data, making a proper model to try and test assumptions against reality. Since they deal with insanely complicated system, with hidden variables all over the place, a proper statistical analysis would be the minimum expected to be able to extract any information from the data, but no matter, once you have a good looking figure, you're done. In cellular/molecular biology, nobody cares about what a p-value is, so as long as Excel tells you it's <0.05, you're golden.
The scientific process has been forgotten in biology. Right now it's basically what alchemy was to chemistry.
I very happy to see efforts like this one. Sure, they might show that a lot of "key" papers are very wrong, but that's not the crux of it. If there is a reason for biologists to make sure that their results are real, they might try to put a little more effort into checking their work. And when they figure out how much of it is bullshit, they might even try to slow down a little on the publications and go back to the basics for a little while.
I'm sorry about this rant, but I've been driven away from a career in virology by those same issues, despite my love for the discipline, so I'm a bit bitter.
You should know that there are a million of tiny ways for an experiment to "fail", requiring one do repeat it. Reagents could be bad, a machine could have broken mid-cycle, a positive (or negative) control could have been wrong... Basically meaning, "it didn't work". In this case, any "data" that would have gotten would be incredibly suspect and you'd need to repeat the experiment.
In the vast majority of labs, this is nothing nefarious, it's the way science is done. If something fails, you try to figure out why it failed, try to fix the issue, and then repeat the experiment. Only once everything works correctly can you get valid data to then try and interpret. And if you get an interesting result, you still need to repeat the experiment 2-3 times to be sure.
Repeating failed experiments isn't an issue - and has nothing to do with alchemy. It's just basic troubleshooting.
The issue is there is "failed" and then there is "failed". Yes, many times you have to repeat an experiment because of bad reagents, broken machines, little tweaks are needed to some obscure parameter, or someone left the lab door open...
However, if you experiment is well controlled, then the controls will reveal this sort of "failure". When I was still running gels, many times when first running a new setup we'd run only controls. If your experiment fails because your controls failed, then that's just science.
But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice.
I am encouraged, though, that there recently seems to be a movement toward, if not outright publishing such negative results, then at least archiving them and sharing them with others in the field. After all, without Michelson and Morley's "failure" we might not have special relativity.
>But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice.
Why does this happen? Clearly this is what the article insinuates. Is publish or perish that strong? Every honest experiment with honest results benefits society. Not every prediction and result combination results in a prize in your lifetime, but that in no way should influence someone's value as a scientist. That science may be used later for something we had not intended (could i offer you the hope of posthumus recognition?). Finding a way it does not work may save someone else some time. This benefits the scientific community.
Not everyone gets to walk on another planet, some people have to build the ship.
For better or worse, most scientific journals still operate on a business model dependent on physical subscriptions. Since this sets something of a limit on how much can be published, and since scientists tend to prefer paying for positive results vs negative, there has been a strong cultural bias toward favoring positive results.
The good news is that this is gradually changing. As scientists begin to understand that online distribution models don't have the same sorts of limitations, and that search can be a powerful tool, there has been a move toward at least collecting negative results. Of course, they still don't benefit the scientists in the "publish-or-perish" world, but even that may be changing...maybe...
>But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice.
I absolutely agree that sometimes, you need to redo an experiment for good reasons.
In most cases I've seen, people do not know why they redo the experiment, though. They know it hasn't produced the data they expected, so they redo it. Maybe it was because a reagent was bad, or a co-worker left the incubator open overnight, or maybe it was because the model is stupid. Who knows?
That's my point, actually. Biologists are playing with systems they do not understand, changing parameters somewhat randomly without any control over them, and they then try to interpret whatever comes out, but ONLY it fits what they wanted. If it doesn't, then "Oh, the PCR machine is at it again!", and they throw the results away.
It seems like you had a really bad experience in lab. I'm sorry for that. But it's a mistake to paint the entire field in a negative light because of this. Not all labs are bad, and some produce really outstanding work.
Sometimes the issue is the PCR machine. Sometimes it's the water (my favorite troubleshooting experience from grad school). And figuring out where the issues are (is the the protocol? the reagents? or is this real signal?) can be difficult.
Playing with systems we don't understand is kinda the point.
I've been lucky enough to work in outstanding labs, with people published in Nature, and other journals of that quality. I've worked in 2 countries, and for 4 different labs. I've also talked with people from all over the world, who have worked everywhere, from Harvard to the Pasteur Institute to Cambridge University. The stories are all the same. I hoped I would find some place where people were trying to do things the right way, but what I found is that currently, you don't need to to be published in top journals, so why bother?
It's really refreshing to hear you talk about trying to troubleshoot why an experiment didn't work the way you expected, I hear mostly of people retrying blindly until it "succeeds". What did you do with what you learned with the water causing the failure? Did you publish this, so that someone (or you!) could try to figure out why water was a problem, or at least so that no one would have the same issue? This is the other point: when people do bother about finding about why things fail, I've never seen any of them try to follow up on that, and figure out not only what made it fail, but why it made it fail. "Yeah, the annealing temp was not the right one". Ok, but why?
Of course playing with systems we don't understand is the point, but we have to be very careful about them. We should be varying 1 parameter at a time. This is mostly impossible in biology, but right now we're not even trying to do anything about it.
People don't investigate things like that, because, frankly, no one cares. Nor should they. If your computer is acting funny, and you find out a spider has made a nest inside it, and it works fine once you clear out the spider nest, would you then decide to determine exactly why that spider nest, in that place, causes the exact problems you observed? No, of course not. Because that's not an interesting question. Computers are complicated machines, and they can break in lots of different ways, most of which are not very interesting in their details.
Similarly, suppose you study cultured cells (which are notoriously finicky), and you want to compare what the cells do in the presence of drug X vs control. But at first, you find that all of your control cells die. And eventually you find out that if you use brand X of bottled water vs tap water, the control cells thrive. Are you seriously proposing that you should then drop all work on drug X, and get to work determining exactly what is up with the tap water in your town? I mean, maybe that would be a fruitful research avenue, if you're worried that the tap water isn't safe for human consumption, or you think that there's something interesting about exactly how the tap water is killing your control cells. But most of the time, investigating the tap water would be an expensive distraction from the question you actually want to answer. And most scientists, I think, would (reasonably) decide to get on with investigating the effects of drug X on their cells, and not worry too much about precisely why the tap water killed the control cells. And I don't think there's anything wrong with that. Life is short, and you have to choose your questions carefully.
So you just accept for no reason that tap water is bad somehow, and discard the result you've just gotten?
I do understand that you have a limited amount of time, and can't just go after everything, but when something happens in science, it needs to be documented. Yeah, maybe someone else should investigate, but someone should. Maybe that particular phenomena that lead to the water influencing your result will give you knowledge about cell metabolism. Who knows? If it has that much of an effect on cell growth that you need to deal with it, it's already more active than a lot of compounds we try out, anyway...
To go back to the computer analogy, it feels like my program is bugged, and to debug it I'm changing variables names (which as far as I know shouldn't matter), and then the code magically works again. Sure, some days, I'll go "Ok, compiler magic, got it", but most days I'd be pretty intrigued, and I'd look into it, because yeah, I might just have found a GCC bug.
I agree, no one cares, but I did. I don't know what I don't know yet, and I don't want to presume anything. The tap water thing might actually lead us to solid models which would explain why tap water breaks the experiment. That's why I really think we should start a movement of publishing everything, and trying to deal with simpler models/systems we do understand before going up to models with so many unknowns that the results are basically a dice roll.
This makes me think of Feynman's comments on Cargo Cult Science:
"In 1937 a man named Young did a very interesting [experiment]. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train the rats to go in at the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.
The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe the rats were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell.
He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go in the third door. If he relaxed any of his conditions, the rats could tell.
Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers that clues that the rat is really using-- not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running.
I looked up the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running the rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic example of cargo cult science."
"So you just accept for no reason that tap water is bad somehow, and discard the result you've just gotten?"
What is the "result" that you referring to here? That the tap water in town X kills cultured cells of type Y? Yeah, I guess you could try writing that up and publishing it, but that's a good way to waste a lot of time publishing results that are interesting only to a very small audience. Honestly, if it were me, I'd send an e-mail to people in the same town that might be working with cells of the same or similar type, and then move on.
"but when something happens in science, it needs to be documented"
No, it really doesn't. Stuff doesn't document itself. That takes time, which sometimes is better spent doing other things. Like answering more interesting questions.
"Maybe that particular phenomena that lead to the water influencing your result will give you knowledge about cell metabolism. Who knows?"
That's true, but my point is that it doesn't make you a bad scientist if you shrug your shoulders about why the tap water kills your cells, and get on with your original experiment. For every experiment you do, there's a million others you're not doing, and so it makes sense to focus on the one experiment you're most interested in, not chase after a bunch of side-projects that will (probably) not lead to any kind of an interesting result.
And all of this is very different than a case where you compare control to treatment, find no difference, and therefore just start fiddling with other experimental parameters until you do get a difference between control and treatment. That is, I think you'll agree, a bad way to do science.
"To go back to the computer analogy, it feels like my program is bugged, and to debug it I'm changing variables names (which as far as I know shouldn't matter), and then the code magically works again. Sure, some days, I'll go "Ok, compiler magic, got it", but most days I'd be pretty intrigued, and I'd look into it, because yeah, I might just have found a GCC bug."
I think a better analogy would be if compiler x acts in ways you don't understand, so you switch to gcc, which (most of the time) works as you expect. Are you really required to figure out exactly why compiler x acts as it does? Or would you just get on with using a compiler that works the way you think it should?
"I agree, no one cares, but I did. I don't know what I don't know yet, and I don't want to presume anything. The tap water thing might actually lead us to solid models which would explain why tap water breaks the experiment."
They might, but there are a lot of experiments that have a very very small chance of an interesting outcome, and a near-one chance of a pedestrian outcome. You can do those experiments, and you might get lucky and get the interesting result, but probably you will just get the pedestrian result. And there's nothing wrong (and a lot right) with instead focusing on experiments where (for instance), either outcome would be interesting to people in the field.
"That's why I really think we should start a movement of publishing everything, and trying to deal with simpler models/systems we do understand before going up to models with so many unknowns that the results are basically a dice roll."
I think you're conflating a number of things here. I agree that a reductionist approach to science has bourne a lot of fruit, historically. I agree that studying systems with a lot of unknowns has risks. And it may be that "publish everything" would work better than what we have now. But even if scientists all decide to publish everything they do, they'll still have to make strategic choices about what experiment to do on a given day, and in many cases that will mean not doing a deep dive into questions like "why does the tap water kill my cells, even in control"?
I think we'll have to agree to disagree on most of those points then.
I do not think there are trivial/uninteresting questions. You have to prioritise, but you can't just sweep stuff under the rug and call it a day. I'm not even using the "it might be huge!" argument, just that science is about curiosity. Most math won't end up as useful as cryptography, but it doesn't matter.
I do think that it is part of your job, as a scientist, to document what you do, and what you observe. If a software engineer on my team didn't document his code/methodology correctly, he'd be reprimanded, for good reason. Yeah, it takes time, but it's part of the job. This way, we avoid having 4 people independently rediscovering how to set up the build tools.
* tap water---control failed
* bottled water (generic store brand)---control failed
* distilled water---control succeeded
and then when writing up the experiment, mention the use of distilled water? You might not be interested in why only distilled water worked, but someone else reading the paper might think to investigate.
The problem with everything you've said is that statistical significance tests are almost always statistical power tests -- do you have enough statistical power given the magnitude of the effect you've seen. The underlying assumption of something like the p-value test is that you are applying to p-value test to all known data sampled from an unknown distribution.
If it is standard laboratory procedure to discard results that are aberrant and to repeat tests, and then to apply the p-value test ONLY to the results that conform to some prior expectation, then the assumptions underlying the p-value test are not being followed -- you're not giving it ALL the data that you collected, only the data that fits with your expectations. Even if this is benign the vast majority of the time -- if 99.9% of the times you get an aberrant result are the result of not performing the experiment correctly -- using the p-value test in a way that does not conform to its assumptions increases the likelihood of invalid studies being published.
"That's why I really think we should start a movement of publishing everything, and trying to deal with simpler models/systems we do understand before going up to models with so many unknowns that the results are basically a dice roll."
I would love to see this implemented, and encouraged in every lab around the world! It's not like we don't have computer programs that could collate all the data?
I don't think I will ever see this happen; because the truth is Not what companies want. They want a drug/theory they can sell. It's a money game to these pharmaceutical companies in the end. Companies, and I believe many researchers want positive results, and will hide, cherry pick the experiments/studies that prove their hypotheses? I know their must be honest, addenda free researchers out there, but I have a feeling they are not working for organizations with the money to fund serious, large scale projects?
Take for instance, Eli Lilly--whom has a history of keeping tight control over their researchers. The history of Prozac is a good example of just how money produces positive results;
"Eli Lilly, the company behind Prozac, originally saw an entirely different future for its new drug. It was first tested as a treatment for high blood pressure, which worked in some animals but not in humans. Plan B was as an anti-obesity agent, but this didn't hold up either. When tested on psychotic patients and those hospitalised with depression, LY110141 - by now named Fluoxetine - had no obvious benefit, with a number of patients getting worse. Finally, Eli Lilly tested it on mild depressives. Five recruits tried it; all five cheered up. By 1999, it was providing Eli Lilly with more than 25 per cent of its $10bn revenue."
(1) I love how Prozac was tested on mild depressives. Don't current prescribing guidelines only recommend the administration of Prozac for the most seriously ill--the clinically depressed? Actually--no, it's still recommended for for a myriad of disorders? Wasn't Prozac proved to be only slightly better than placebo? If you dig deeper, their are some sources that don't see any benefit over placebo.
(2) Wouldn't patients/society benefit from seeing all of the studies Eli Lilly presented to the FDA? Not just the favorable ones? How many lives would have been saved if this drug was given an honest evaluation--if every study was published, and put through statistical calculations in the 90's? Think about the millions of patients who suffered terrible side effects of this once terrible expensive drug? Think about the birth defects that could have been prevented?
So yes, I would love to see everything published, but I don't think the business cycle/politics will ever allow it? They want results! They want money! They want endowments! It's a sick game, until you are the one needing help. Help that only honest, good science can produce?
Making sure that everyone working in the field knows not to use tap water seems worth doing, though, even if the reason why isn't understood yet. It sounds like replication is a problem because this sort of knowledge isn't shared widely enough?
tl;dr: Sonnenschein and Soto were studying effects of estrogens on in vitro proliferation of breast cancer cells. Their assays stopped working. Eventually they figured out that Corning had added p-nonylphenol, which is estrogenic, to the plastic, to reduce brittleness.
"We need to get a new carbon filter for that MilliQ machine"
"Wait there's a carbon filter in the milliQ machine?"
"Yeah..."
"...oh, that explains all those results I got"
^ Actual conversation I had in a breakroom.
Stuff breaks all the time. The temperature of the lab changes. Connections get loose. 95% of the time, when the experiment doesn't work, it's because you screwed up on something that's either difficult to control or impossible to control or which you didn't even realize probably wasn't the thing you expected it to be.
But then you risk thinking you understand the system, but you actually don't. How can you be sure the equipment failure didn't make you see something that wasn't there?
That's why you repeat the experiment to confirm your results. A result due to a sporadic equipment failure likely won't be reproducible. If you can't replicate your own results, how could you expect anyone else to be able to?
Spot on with the alchemy remark, I've made similar comparisons before. Coming into bioinformatics/computational biology with a strong discrete math background I found a lot of professors excited to work with me until I started telling them how their ideas and models and experiments didn't imply what they wanted them to. Just like the startup world is awash with "it's like Uber for X" the biology world is full of "let's apply X mathematical technique to $MY_NICHE" and somehow this is supposed to always generate novel positive results worthy of publication. Then you tell them that you applied such-and-such mathematical/statistical model to their pet system and that the results contradict their last 10 years of published papers . . . and they ask you to do it again.
I remember one professor studied metabolic reaction networks modeled with differential equations. The networks themselves were oversimplifications and relied on ~5N parameters (N being the number of compounds in the network). The problem was that while all the examples in publication converged on a nice steady state (yay homeostasis is a consequence of the model!) it was trivial to randomize the parameters within their bounds of experimental measurement and create chaotic systems. Did this mean the model wasn't so great? No, it just meant those couldn't be the real-life configuration of those parameters . . . sigh. And now I'm a data engineer and no one asks me to get data from an API that doesn't actually provide it and I'm much happier.
"it was trivial to randomize the parameters within their bounds of experimental measurement and create chaotic systems. Did this mean the model wasn't so great? No, it just meant those couldn't be the real-life configuration of those parameters . . . sigh."
Why is this sigh-worthy? Maybe the model is still ok. Maybe some of the parameters are constrained in nature in ways that you/we just don't know about yet. Of course, it could be that the model is bad, but you act as if your results, as stated, demonstrate this. Which they do not, as far as I can see.
I hoped that it wasn't as bad in computational biology, or ecology, or any other biology field where systems and models are actually defined. It saddens me to read that your experience was as bad as mine...
I'm a CompBio PhD student, and my experience is that folks in that field are much more careful with statistics than in, say, molecular biology labs, but it varies from lab to lab. My PI is exceedingly meticulous about stats -- for instance, we don't report p-values, but rather entire distributions -- but that's because our work is all in silico, so it's easy to run tons of replicate simulations. Wet lab work that's finicky should definitely be held to high statistical standards, but I don't think it's fair to presume everyone in the field guilty until proven innocent.
But wet lab scientists should be even more careful! They have way less control over the system they're trying to study than you do, so stats are the only security net we have to even attempt to do anything with the data we produce.
I also agree on the innocent until proven guilty part, but by now I've seen and talked to hundreds of people with the best intentions, who do not realise how important careful examination of the data is, so I'm growing a bit disillusioned.
I think it is entirely fair to presume biologists incompetent and sloppy until they have proven otherwise.
(My impression from admittedly limited contact with biology students and from browsing through the occasional paper is that most of them barely approach mediocrity from below.)
Many fields (macroeconomics, IO, etc.) write models that end up with some type of calibration which is implemented in thousands of lines of code. Those lines are written by RAs with almost no programming experience, so what happens is this:
While (result!=expectation)
Ask assistant to look for bugs and repeat simulation
The end result is that you don't stop when there are no more bugs, you stop when you've got the coefficient signs you want, and then get published.
Only if your paper has a high impact, has the code and number open, AND was coded in a very easy language, people discover the flaws:
"the paper ... was, and is, surely the most influential economic analysis of recent years [...] First, they omitted some data; second, they used unusual and highly ques ionable statistical procedures; and finally, yes, they made an Excel coding error"
> we see PhD students and post-doc working 70h hours week on experiments with seemingly random results until the randomness goes their way.
There are known and tested protocols that can fail. Not every step can be accurately recorded. It's very common that an experiment will not work well the first time it's performed (even when supervised by someone experienced). Over time, researchers improve their skills and achieve better results following the same exact protocol. Does that mean that the science behind the experiment is bad?
Does that mean that the science behind the experiment is bad?
No, the science might be solid. But if attempts by peers to reproduce the results fail more often than they succeed, the paper describing the science is (by definition?) inadequate. The level of detail required in the paper varies from field to field, and experiment to experiment, but if the techniques aren't described well enough for others to follow them, then the paper needs more detail.
Even commercial products (where they have a financial incentive to provide clear and comprehensive instructions) often take a lot of training before they work properly. How can we expect a small research lab to be better?
You are right to point out that giving clear instructions for a complex task is difficult. And elsewhere in this thread, 'nycticorax' makes some great points. I fear my answer is along the lines of Rutherford's often ridiculed quote about statistics and experimental design: "If you need to use statistics, then you should design a better experiment."
If the experiment that you describe in your paper is too difficult for others to reproduce, perhaps it shouldn't yet be published as a paper? Would the public interest be better served by funding researchers who do simpler but reproducible work, rather than complex work where the results essentially need to be taken on faith? Carried to the extreme, this is a terrible rule, but I feel there is a kernel of truth to it.
I guess the right strategy depends on how much faith you have in the correctness of published results, evaluated solely on plausibility and the reputation of the researcher, and thus how much value there is in a conclusion based on irreproducible results. I think there is a currently a justified crisis of belief in science, and that many fields would do well to get back on to solid ground.
Isn't this actually an attractive ethical hazard† (in a very broad incentive sense) - and as such, to counteract it couldn't we actually encode an ethical obligation to immediately publish the data from any experiment, to counteract this hazard? Just in any old place, not as a full paper.
You could re-run your experiment of course if you thought there was some experimental methodological error, but as you disclose somewhere your first, your second, and your third dataset all showing the same data with more or less the same methodology, you would have to show increasing confidence in your fourth or fifth dataset (the one you would otherwise publish alone), because it has to explain all of the earlier datasets as statistical flukes only: you would no longer have the incentive to run the fourth experiment alone and publish it without reference to earlier trials.
To give an example, suppose anyone is rewarded by publicity if they show that flipping U.S. coins favors heads by more than 2%. This is temptingly easy to do if you don't publish experiments that don't statistically prove that: you just keep re-running the experiment until you get the result you want at the p-value you want, so if the reward for the result is more than the number of experiments you need to do to get it times the cost of each experiment (which can be truly tiny), the experiment presents an attractive ethical hazard.
But if you are ethically forced to publish (or even summarize) your earlier complete experiment and dataset really in any old place, then if there is no actual difference it becomes vanishingly unlikely that you can suddenly prove your theory and explain all of the earlier experiments as statistical outliers. You would stop exploring after your second or third experiment. (Which you would still quickly summarize.) This does however present an added burden to researchers, especially if they quickly test something before really refining the methodology to do so. So these quick disclosures could still be considered quite dirty and not very meaningful. However their disclosure would give a good indication regarding how strong a result really is. (i.e. by glancing at how much dirty data precedes the actual experiment being published.)
-
† I'd like to recall the words "moral hazard" because it does indicate that people are tempted toward the bad behavior. But economically I think that term is too specific (meaning risk-taking where someone else has to clean up in case of failure) -https://en.wikipedia.org/wiki/Moral_hazard
Yeah, I've seen this xkcd. It's on point as usual, but it doesn't show the frightening thing: it's the scientists who are going "Whoa" and believing in the results they've produced...
On the other hand, reproducing badly-designed, not-well thought of, and probably inadequately executed experiments won't be of much help to anything. In biology its not often clear what is known, what is not, and what are the open questions, and the scientists themselves have a lot of the blame for it (for religiously sticking to closed publishing models and a general lack of initiative to share their data).
People there are re-doing the same experiment over and over until it gives them the result they want, and then they publish that. It's the only field where I've heard people saying "Oh, yeah, my experiment failed, I have to do it again". What does it even mean that an experiment failed? It did exactly what it was supposed to: it gave you data. It didn't fit your expectations? Good, now you have a tool to refine your expectations. But instead, we see PhD students and post-doc working 70h hours week on experiments with seemingly random results until the randomness goes their way.
A lot of them have no clue about statistical treatment of data, making a proper model to try and test assumptions against reality. Since they deal with insanely complicated system, with hidden variables all over the place, a proper statistical analysis would be the minimum expected to be able to extract any information from the data, but no matter, once you have a good looking figure, you're done. In cellular/molecular biology, nobody cares about what a p-value is, so as long as Excel tells you it's <0.05, you're golden.
The scientific process has been forgotten in biology. Right now it's basically what alchemy was to chemistry.
I very happy to see efforts like this one. Sure, they might show that a lot of "key" papers are very wrong, but that's not the crux of it. If there is a reason for biologists to make sure that their results are real, they might try to put a little more effort into checking their work. And when they figure out how much of it is bullshit, they might even try to slow down a little on the publications and go back to the basics for a little while.
I'm sorry about this rant, but I've been driven away from a career in virology by those same issues, despite my love for the discipline, so I'm a bit bitter.