Research Conduct and Metrics
From Chapter 25:
Here we come to the essence of the problem with the system. Researchers are frequently doing more politics than science. When people propose a theory, get some result and get funded based on the initial results, are they going to make a 180 degree turn and say: “We just found out that we were wrong, but please continue funding us.” Both things are unlikely to happen. Surprisingly, there is very little “I was wrong” in science. If so, the “funding faucet” would be shut off upon such a claim. What does this mean? It means that the system fosters dishonesty. If one makes a mistake and admits it, the system will punish that person! Most scientists build their career around something; they will persist with that theory and reject the experimental results that are in opposition as “outliers.” Yes, scientists are guilty of sticking to a widely accepted theory even if there are experimental results in opposition to it! It would take an overwhelming amount of evidence to change the theory. Challenging the common pattern of thought has never been looked on favorably. Louis Pasteur had a hard time making the medical establishment accept that infections are caused by microorganisms. The medical community laughed at him… at first. How many of you are aware that the standard medical practice for resuscitating drowning victims was composed of pumping tobacco smoke up the victim’s anus? That was going on just two hundred years ago. Heroin was an over-the-counter painkiller just a hundred years ago. When radium was discovered, it was given to babies! There were radioactive tonics being sold for promoting overall health and well-being! Were they endorsed by universities? Probably not, but it demonstrates that nobody voiced their objection due to lack of data. Who dares to say that we are pass this kind of thinking on and that the accepted practices of today will not be something that future generations will laugh about? Speaking practically, a recent study showed that as much as 75% of science papers are not reproducible. In other words, people make shit up or hide something. If one goes through an experiment multiple times and explains the procedure in a paper, wouldn’t anyone with sufficient amount of training and resources be able to reproduce the results published in the “prestigious” journals? Yes, scientists make up things or “doctor” their results to make them look good and favorable for publications. We frequently hide something in the protocol so the competition that often reviews the manuscript cannot simply read the cook-book and repeat it. There is purposeful dishonesty right here, but who is to blame? Scientists, or the system, or both? People resort to writing protocols in such way so they resemble patent applications. Everything seems well explained, yet some crucial piece of information is missing, making the reader incapable of reproducing
the results. If there was no fear of competition there would be less dishonesty. “Peer reviewed” usually mean “competition stalled.” How many times did you feel that the reviewer was asking you to do unnecessary experiments that would take you months to produce? They are slowing you down! You may have done the same to other people. How is this behavior advancing science? Same thing happens with grant applications. They frequently end up on the desk of the fiercest competition. Grant reviewing is anonymous, giving people the opportunity to say things they would not say if everyone knew who they were. Sometimes we figure out who was on the other side of the grant. I personally witnessed a situation where my former supervisor’s grant was literally destroyed by our lab’s competition. Why? Why not! No funding to the competition means less competition. Is this fair? Looking at the other extreme: is it fair that two buddies review each other’s grant so everyone is happy? If grant reviewing was completely impartial and people did not have the fear of competition so they did not try to stall the other person, why would anyone fake results? Research labs are like mini dictatorships. Conferences are like dictator summits where people seem to agree and hopefully collaborate. The bosses demand results. Sometimes the demands are unreasonable. Students spend long hours endlessly repeating experiments with the hope of making them look the way they are supposed to look according to some theory. The stressed out boss wants certain kind of results as grants depends on them. What would a student who is constantly pushed to produce those results do? It is easy to judge, but one should get into the mindset and walk in the shoes of those people. Some students and postdocs have families to feed. Certain results mean obtaining food and housing for their families and, in many cases, staying in the country. There is so much at stake and all it takes is moving some dots on a graph. The dumb ones would manipulate digital data. That is easy to check. However, the smart ones make sure to add, say, more reagent or something else that would produce favorable raw data. It is possible to check this but not as easy. Since 75% of data reported in well-known journals is irreproducible, it means that the waste of time and resources are enormous! How can this practice benefit science? It is actually doing the opposite by misleading researchers and creating an even greater waste of time and resources. The metrics are equally faulty. Imagine a lab that produces ten papers in prestigious journals per year. If we apply the recent knowledge which tells us that three quarters of published results are not reproducible, statistically speaking, the lab that publishes ten papers a year is producing only 2.5 papers that can be of use to other people. Most labs do not produce ten papers, meaning that labs producing less than 4 papers statistically speaking may contribute to science as much as labs that produce no papers. When regular metrics areapplied, the principle investigator of the lab capable of producing ten papers is a star – or so it seems. The system favors dishonesty and overproduction! If someone wants to spend enough time to produce a quality paper, he will either run out of funding or get scooped by the labs that are not as careful. If someone wants to report a previous mistake, that will be like shooting oneself in the foot. Why do we have a system that prevents people from doing what is right? Why do we have a system that makes us discover on a deadline? Regarding research conduct and managing funds for your lab, how many of you took $5,000 allowance from your grant money to cover travel expenses for a conference but spent only $2,000? What happened to the rest? Did you return it to the lab’s budget? Do you travel a lot? I happen to know professors who travel three weeks out of four in a month. Surprisingly, they have self-financed companies. Why is the system allowing people who are well-funded by public money to siphon those funds into private ventures?It is easy to play an entrepreneur while at the same time being a tenured professor.
Apparently, job security for the rest of one’s life is not enough. This was not the case fifty years ago. The law has changed thanks to corrupt politicians. Nothing financed by public money was patentable until the 80s. This is why I said at the beginning of the letter that many scientists are using public money to eventually cash out through a startup on the side. Graduate students and postdocs serve as general labor that will generate data that is supposed to attract investors. Is this practice going to push science forward? Another thing that is seriously hurting many people is the “caste system” we built around publications. It literally seems that findings published in lower-ranked journals are less relevant. It creates the need to be “trendy”. Researchers know what is trendy so they can change their research direction. Being trendy is not in the best interest of science, just as having pious women is not in the best interest of a nation that wishes to grow in numbers. If someone is conducting valid research, but none of it is trendy, that lab is destined to publish only in the journals that are not considered important, so the overall performance of the lab judging by the current metrics is going to be low. In the long run, the ever- shrinking funding situation will eventually kill that lab. As you can see, some people are forced to be trendier just to stay in the game. The number of journals has exploded in the past few decades. Researchers of the past had only a handful of journals to consider. Now, publishing scientific findings is a great business. The most desired journals are not open access, so researchers who wish to increase the visibility of their work are paying an extra fee to make them truly open access, which makes perfect sense from the business perspective of the journal. However, from the common sense perspective, it means that someone whose research is funded by public money needs to pay extra to make those findings truly publicly available! The alternative is to publish in lesser-known journals that are open access, but that comes with a penalty if the journals are not high-impact. Luckily, the number of open access journals is growing fast, but if someone calculated how much money was spent on open access fees in well-known journals, it may be a surprisingly large number. What could have been done with those funds instead?
No Comments