The researchers, led by the University of Cambridge, analysed more than 12,000 research papers on breast cancer cell biology. After narrowing the set down to 74 papers of high scientific interest, less than one-third – 22 papers – were found to be reproducible. In two cases, Eve was able to make serendipitous discoveries.
The results, reported in the journal Royal Society Interface, demonstrate that it is possible to use robotics and artificial intelligence to help address the reproducibility crisis.
A breast cancer cell up close. Image credit: NCI
A successful experiment is one where another scientist can achieve the same result in a different laboratory under similar conditions. But more than 70% of researchers have tried and failed to reproduce another scientist’s experiments. More than half have been unable to produce some of their experiments: this is the reproducibility crisis.
“Good science relies on reproducible results; otherwise, the results are meaningless,” said Professor Ross King from Cambridge’s Department of Chemical Engineering and Biotechnology, who led the research. “This is particularly critical in biomedicine: if I’m a patient and I read about a promising new potential treatment, but the results aren’t reproducible, how am I supposed to know what to believe? The result could be people losing trust in science.”
Several years ago, King developed the robot scientist Eve, a computer/robotic system that uses artificial intelligence (AI) techniques to carry out scientific experiments.
“One of the big advantages of using machines to do science is they’re more precise and record details more exactly than a human can,” said King. “This makes them well-suited to attempting to reproduce scientific results.”
As part of a project funded by DARPA, King and his colleagues from the UK, US and Sweden designed an experiment that uses a combination of AI and robotics to help address the reproducibility crisis by getting computers to read scientific papers and understand them and getting Eve to attempt to reproduce the experiments.
For the current paper, the team focused on cancer research. “The cancer literature is enormous, but no one ever does the same thing twice, making reproducibility a huge issue,” said King. He also holds a position at the Chalmers University of Technology in Sweden. “Given the vast sums of money spent on cancer research and the sheer number of people affected by cancer worldwide, it’s an area where we urgently need to improve reproducibility.”
From an initial set of more than 12,000 published scientific papers, the researchers used automated text mining techniques to extract statements related to gene expression changes in response to drug treatment in breast cancer. From this set, 74 papers were selected.
Two different human teams used Eve and two breast cancer cell lines and attempted to reproduce the 74 results. Statistically significant evidence for repeatability was found in 43 papers. The results were replicable under identical conditions, and considerable evidence for reproducibility or robustness was found in 22 articles, meaning the results were replicable by different scientists under similar conditions. In two cases, automation made serendipitous discoveries.
While only 22 out of 74 papers were found to be reproducible in this experiment, the researchers say that this does not mean that the remaining papers are not scientifically reproducible or robust. “There are many reasons why a particular result may not be reproducible in another lab,” said King. “Cell lines can sometimes change their behaviour in different labs under different conditions. We found the most important difference was that it matters who experiments because every person is different.”
King says that this work shows that automated and semi-automated techniques could be an essential tool to help address the reproducibility crisis and that reproducibility should become a standard part of the scientific process.
“It’s quite shocking how big of an issue reproducibility is in science, and it’s going to need a complete overhaul in the way that a lot of science is done,” said King. “We think that machines have a key role to play in helping to fix it.”
Source: Cambridge University