R&D and science at large has been grappling with a “Replication Crisis,” where positive results of experiments can’t be reproduced. Failure to replicate experiments results in over $20B in losses across science and industry each year. We explored this topic in our recent webinar: The Replication Challenge: Charting a Course for Streamlined Science. Read on for a summarized preview of the webinar, or watch the full video, below.
The discussion includes industry experts from academia and R&D and is moderated by Jenny Hu, STARLIMS Head of Customer Success and Growth.
The Webinar Agenda
- Fireside chat with Youyou Wu of University College London and Uraisha Magan of Newform Foods
- What is the Replication Crisis? How does it impact science in a time of explosive innovation?
- The implications of the Replication Crisis on commercial R&D
- Actionable takeaways and practical strategies for solving the Replication Crisis
What is the Replication Crisis?
Youyou Wu: The Replication Crisis refers to a significant challenge in science and innovation where empirical studies fail to replicate. Meaning if you repeat the same study, you will not get the same results, which is known as a “replication failure.” This raises the question of how much we can still trust published research findings. This problem is present in many different disciplines, including economics, biology, chemistry, and medicine. Being able to replicate or reproduce results from scientific experiments is a foundational requirement of Research & Development, enabling us to bring new, innovative products to the market.
Replicability is the bedrock of scientific progress because of two things: its practical importance and the cumulative nature of knowledge production. In terms of practical importance, we trust the medicines we take because science demonstrates that they consistently work. But even when a piece of knowledge is not immediately applied in real life, it is still important that it is replicable because other scientists or practitioners will build upon past discoveries to do further work and interventions. For example, there might be drugs developed later based on the principles discovered among animals. Science can build upon itself and lead to more discovery when it is replicable.
YouYou Wu, Assistant Professor in Psychology at University College London (UCL). She studies metascience and methodology, particularly reproducibility, and is interested in how one can best predict replication outcomes before an actual replication takes place. She is a co-author of “How AI Can Help Researchers Navigate the “Replication Crisis” published by Kellogg Insight.
How big of a problem is the Replication Crisis?
Youyou Wu: The negative effects can be tremendous. In life sciences alone, more than 20 billion USD a year is wasted on preclinical research that is not reproducible, which can very much delay drug development. The numbers beyond that are likely far more, as this is a challenge that impacts every sector of science.
What is the relationship between academia and industry, and how does the Replication Crisis play out there?
Uraisha Magan: R&D often relies on the scientific work published in academic papers, and many products have been developed using information found within scientific publications. The Replication Crisis affects both academia and industry differently but have similar consequences.
Trust in and credibility of science requires it be reproducible and replicable. For industry specifically, the primary risk is cost from resource allocation. For start-ups, wasted costs on R&D that is not replicable can impact the company significantly.
If a biotech product is sold to the public that is based on research that is not sound, it can have devastating effects on the company, such as loss of customers, damage to its reputation, and investors losing interest.
Uraisha Magan, PhD, R&D Lead at Newform Foods, boasts over 10 years of experience in applied and academic research with a PhD in Microbiology from the University of Pretoria and a Master’s degree in Biochemistry. Her expertise in the cultivated meat industry is extensive. Previously, as a Senior Scientific Officer at a cultivated seafood start-up, she managed projects on fish cell lines. Magan’s background also includes developing a Bacillus-based chicken feed probiotic, now a licensed product, and working on a cattle tick vaccine as a Postdoctoral Research Fellow. Her diverse roles encompass bioprocess development, product formulation, and quality control in the industry.
Can you give real-life examples of how the Replication Crisis manifests in scientific research?
Uraisha Magan: While in academia, I was attempting to grow MDCK II cells to form a monolayer to act as a “passive” intestinal barrier. I attempted for 11 months and was not successful in growing this. I had read several papers and tried to replicate the process as it was published. Eventually after losing almost a year, we had to pivot to something else. I could write my failed attempts in my thesis, however I could not publish my work, as I did not get a result.
At CSIR which is a R&D institute, I attempted to grow a new probiotic strain, with countless issues. Eventually through modifications of our own, the team and I were successful. In industries that produce biological materials based on biotech, it’s imperative to achieve consistent production processes, but this often happens over time, due to the nature of biological materials. In these cases, when you get a recipe/experiment to work, there can be reluctance to share it publicly, because it could lead you to losing your competitive edge.
In my current field of cultivated meat, the reliance on publications and scientific reports is common. The biggest challenge is the ability to scale up production of cultured meat, of which the information is not readily available. Other experiments such as isolation of the primary cells are published frequently and in abundance, however with certain species it’s not always as straightforward as the publications make it out to be, and it takes months to get it right often with significant changes to the protocol.
What measures should R&D teams implement to address these challenges?
Uraisha Magan: The most important is clear, concise documentation and records of all experimental work. This should be shared with everyone on the team. Making research outputs openly available allows for greater transparency, scrutiny, and replication by other researchers. Then of course there are the more obvious things to ensure good research, such as requiring replication of independent experiments.
For industry, the most important things are standard operating procedures and sufficient training that allows different members of the team to perform the same experiments and get the same results. Also it’s important to promote a culture of transparency in experimental procedures, which fosters a supportive, inclusive environment.
Beyond documentation, what else should R&D teams consider?
Uraisha Magan: In very intricate scientific experiments, an important factor that is not always documented is the technique of how the experiment was performed. Another aspect that is often not considered is the environment. Labs don’t always have the consistency of environment needed to induce replication, especially in the biological field. These could include equipment and instrumentation or the conditions of the lab—environmental factors such as temperature, humidity, and lighting. All of these can impact experimental outcomes. Inconsistent or poorly controlled laboratory conditions can introduce variability and confounding factors, making it difficult to replicate results across different settings.
Crucially, failed experiments and learnings are not always documented or published. It’s always the polished version that could have taken the researcher months to perfect. I am a firm believer in documenting negative or failed experiments as there is a huge amount of learnings in these reports.