Until now, discovery was often considered the main goal of medical science. But nowadays discovery is almost too easy. Anyone with a little funding and a few biological specimens in a refrigerator can make thousands of postulated “discoveries.”
Indeed, the number of research questions that we can pose is increasing exponentially. Medical kits the size of a thumbnail can measure a million different biological factors on an individual with an infinitesimal amount of blood. A million research questions can be asked on the spot. But even with proper statistical testing, many tens of thousands of these biological factors may seem to be important due to mere chance. Only a handful of them really will be. The vast majority of these initial research claims would yield only spurious findings.
So the main issue nowadays is to validate “discoveries” by replicating them under different settings. Several different teams of researchers need to see them “work” again and again using common rules. Moreover, all the teams should agree not to select and report only the data that seem most impressive. With selective reporting, we would end up with a long list of all the false discoveries made across all research teams, with only a few true findings buried among this pile of non-replicated waste.
In fact, empirical data suggest the significance of this danger. In a paper in the Journal of the American Medical Association published in July 2005, I showed that refutation is very common, even for the most prestigious research findings. I examined the 45 clinical research findings that had received the greatest recognition in the scientific world, as documented by the number of times other scientists had cited them over the last 15 years.
Even with the most robust types of research – for example, randomized clinical trials – one of four of these results had already been found to be wrong or potentially exaggerated within a few years after publication. For epidemiology (e.g. studies on vitamins, diet, or hormones in terms of their association with health outcomes in the general population), four-fifths of the most prestigious findings were rapidly contradicted. For molecular research, in the absence of extensive replication, the refutation rate may occasionally exceed 99%.
But we should not panic. It is to be expected that the majority of research findings are rapidly contradicted and refuted; indeed, it is part of how progress of science occurs. However, we need to adapt to this situation. Instead of taking scientific evidence as dogma, we should consider it as tentative information that should be ascribed a level of credibility.
It is not too late to watch our AI Action Summit event.
Click the link to watch world leaders, tech experts, and other distinguished speakers – including Justin Trudeau, Petr Pavel, Daron Acemoglu, Reid Hoffman, Marianna Mazzucato, James Manyiga, Audrey Tang, Sylvain Duranton, Celina Lee, Patrick Pouyanné, and others – discuss some of the most important questions raised by the rise of artificial intelligence.
Watch Now
There is nothing wrong with disseminating scientific information that has a credibility of 10%, or even 1%. Sometimes, this will be the best evidence we have. But we should get used to understanding that some research findings have very low credibility, while others may be more likely to stand the test of time. Scientists themselves may be able to ascribe these levels of credibility to their own work in fairness, if they describe in detail what they set out to do, and how they did it.
Science is a noble pursuit, but genuine progress in scientific research is not easy to achieve. It requires a lot of time, continuous effort, uncompromising integrity, appropriate funding and material support, and unwavering commitment. Proposed scientific advances require careful validation and replication by independent scientists. Scientific knowledge is never final, but it evolves continuously. This is part of the great fascination of science, and it fosters liberty of thought.
While these principles are probably well known to serious scientists, they are often forgotten when scientific information is disseminated. Our society is awash with inflated information, which is inherent to efforts in many human activities – entertainment, law courts, stock markets, politics, and sports, to name but a few – to gain greater public attention in the framework of mass civilization.
But it would be a damaging to expect science to “show off” in this way. Exaggeration contradicts the key hallmarks of scientific reasoning: critical thinking and careful appraisal of the evidence.
To have unlimited access to our content including in-depth commentaries, book reviews, exclusive interviews, PS OnPoint and PS The Big Picture, please subscribe
As Americans and the rest of the world adapt to another four years of constant chaos, it will be important to remember that Donald Trump's principal purpose in issuing any pronouncement is to gain attention. Unfortunately, much of the US media is uninterested in helping the public separate signal from noise.
upbraids US journalists for refusing to distinguish bluster from genuine policy initiatives.
Chinese tech startup DeepSeek’s release of a high-performance, low-cost artificial-intelligence platform has roiled financial markets and raised fundamental questions about the business models of American AI giants. Will the technology turn out to be a boon for competition in a critical industry, or should it be viewed as a geopolitical shot across the West’s bow?
Given the pace of AI development, policymakers and civil society must step in now to ensure that the next general-purpose technology serves the public interest. Otherwise, already dominant monopolists will supercharge the socially harmful digital business models they perfected over the past decade.
calls for urgent action to shape the development of AI before it becomes another means of rent extraction.
Log in/Register
Please log in or register to continue. Registration is free.
Until now, discovery was often considered the main goal of medical science. But nowadays discovery is almost too easy. Anyone with a little funding and a few biological specimens in a refrigerator can make thousands of postulated “discoveries.”
Indeed, the number of research questions that we can pose is increasing exponentially. Medical kits the size of a thumbnail can measure a million different biological factors on an individual with an infinitesimal amount of blood. A million research questions can be asked on the spot. But even with proper statistical testing, many tens of thousands of these biological factors may seem to be important due to mere chance. Only a handful of them really will be. The vast majority of these initial research claims would yield only spurious findings.
So the main issue nowadays is to validate “discoveries” by replicating them under different settings. Several different teams of researchers need to see them “work” again and again using common rules. Moreover, all the teams should agree not to select and report only the data that seem most impressive. With selective reporting, we would end up with a long list of all the false discoveries made across all research teams, with only a few true findings buried among this pile of non-replicated waste.
In fact, empirical data suggest the significance of this danger. In a paper in the Journal of the American Medical Association published in July 2005, I showed that refutation is very common, even for the most prestigious research findings. I examined the 45 clinical research findings that had received the greatest recognition in the scientific world, as documented by the number of times other scientists had cited them over the last 15 years.
Even with the most robust types of research – for example, randomized clinical trials – one of four of these results had already been found to be wrong or potentially exaggerated within a few years after publication. For epidemiology (e.g. studies on vitamins, diet, or hormones in terms of their association with health outcomes in the general population), four-fifths of the most prestigious findings were rapidly contradicted. For molecular research, in the absence of extensive replication, the refutation rate may occasionally exceed 99%.
But we should not panic. It is to be expected that the majority of research findings are rapidly contradicted and refuted; indeed, it is part of how progress of science occurs. However, we need to adapt to this situation. Instead of taking scientific evidence as dogma, we should consider it as tentative information that should be ascribed a level of credibility.
PS Events: AI Action Summit 2025
It is not too late to watch our AI Action Summit event.
Click the link to watch world leaders, tech experts, and other distinguished speakers – including Justin Trudeau, Petr Pavel, Daron Acemoglu, Reid Hoffman, Marianna Mazzucato, James Manyiga, Audrey Tang, Sylvain Duranton, Celina Lee, Patrick Pouyanné, and others – discuss some of the most important questions raised by the rise of artificial intelligence.
Watch Now
There is nothing wrong with disseminating scientific information that has a credibility of 10%, or even 1%. Sometimes, this will be the best evidence we have. But we should get used to understanding that some research findings have very low credibility, while others may be more likely to stand the test of time. Scientists themselves may be able to ascribe these levels of credibility to their own work in fairness, if they describe in detail what they set out to do, and how they did it.
Science is a noble pursuit, but genuine progress in scientific research is not easy to achieve. It requires a lot of time, continuous effort, uncompromising integrity, appropriate funding and material support, and unwavering commitment. Proposed scientific advances require careful validation and replication by independent scientists. Scientific knowledge is never final, but it evolves continuously. This is part of the great fascination of science, and it fosters liberty of thought.
While these principles are probably well known to serious scientists, they are often forgotten when scientific information is disseminated. Our society is awash with inflated information, which is inherent to efforts in many human activities – entertainment, law courts, stock markets, politics, and sports, to name but a few – to gain greater public attention in the framework of mass civilization.
But it would be a damaging to expect science to “show off” in this way. Exaggeration contradicts the key hallmarks of scientific reasoning: critical thinking and careful appraisal of the evidence.