How can flaws in clinical trial design, conduct and reporting impact clinical decision-making?

Clinical trials are considered an important source of information for healthcare professionals to keep up to date with the latest research in their field¹. As part of evidence-based practice, the results of clinical trials can also influence prescribing decisions and patient management^2,3.

When appropriately designed, conducted and reported, randomised controlled trials (RCTs) are considered the ‘gold standard’ in evaluating the safety and effectiveness of healthcare interventions^4,5.

However, many factors can complicate interpretation of a study’s results^6,7. Although no clinical trial is perfect in design and execution, limitations and flaws can impact a study’s validity, trustworthiness and credibility^6,7. Poorly designed, conducted and reported clinical trial research can overestimate benefits and underestimate the harms of a treatment approach⁸, and may lead to inaccurate conclusions^9,10.

In this first article of a 3-part series, explore key issues reported with quality and accuracy in clinical trial publications, and their potential impacts on clinical decision-making.

Fraudulent data, zombie trials and a growing list of retractions

Although some may assume that biased, inaccurate and false reporting are minimised through the peer review processes, this is not necessarily the case¹¹. Among research and clinical trial findings published over the past decade, there has been a continuous stream of retractions, reports of poor quality, inaccurate or fabricated data, and fraudulent studies^7,9,11-15.

A new term has even been minted – ‘zombie trials’ – to describe trials with flawed or untrustworthy data that would be considered sufficient for retraction¹¹.

In a sample of 153 RCTs submitted to Anaesthesia during 2017–2020, 44% had false data and 26% were categorised as zombie trials – that is, the extent of fabricated data was sufficient to invalidate their scientific value¹⁵

Other key issues: Bias, misreporting, spin and limited applicability to real-world patients

A range of other issues can impact the accuracy, quality and applicability of clinical trial data. These include:

• Bias in the design, conduct and reporting of published clinical trials^7,16 – the validity of a trial’s findings may be questioned if it is considered to have a high risk of bias¹⁷
• Use of patient populations, treatment regimens and/or outcome measures that have limited applicability to real-world patients and settings^7,18
• Small sample size⁷ – a critical consideration in trial design, sample size can affect interpretation of a study’s findings¹⁹
• Misreporting of outcomes, or changing outcomes from those that were pre-specified in the study protocol (‘outcome switching’)^16,20-22
• Selective or incomplete reporting of an intervention’s impact, which can lead to misinterpretation or overstatement of impact¹⁰, or an under-reporting of adverse events¹⁸
• Use of ‘spin’ when reporting results, which can overemphasise the benefits and safety profile of an intervention^8,23
• Extrapolating findings beyond the clinical trial population²³

Issues with clinical trial publication quality are not just restricted to specific or niche journals^20,24. In 2019, Goldacre and colleagues found misreported outcomes in 58 of 67 RCTs published across 5 high-impact journals during a 6-week period in 2015²⁰. This included trials with missing primary outcomes, incorrectly reported primary and secondary outcomes, and reporting of additional outcomes without declaring they were novel²⁰. Since then, studies have continued to report issues with the quality and accuracy of randomised trials across various high-impact journals^5,25.

Can flawed clinical trial data adversely affect clinical decision-making?

Findings from clinical trials are used to inform guideline recommendations⁷, clinical decision-making in practice²⁶, and are also a basis for regulatory approval of medicines^27-29 and decisions for government reimbursement^30,31.  

There is potential that flawed or fabricated clinical trial results could lead to use of ineffective interventions in practice, or even patient harm^14,32

The impact of poor-quality research on clinical decision-making was highlighted in the context of fast-moving research during the COVID-19 pandemic^9,14. In a tidal wave of research into potential COVID-19 treatments, the availability of pre-print publications, low quality evidence, and studies with serious methodological limitations led to confusion about the effectiveness of medicines for treatment or prophylaxis of COVID-19^9,14. This caused confusion amongst not only the general population, but also healthcare professionals⁹. Some of these medicines were used to treat patients with COVID-19 in clinical practice, despite a lack of sufficient evidence to assess their efficacy or safety for this indication^9,33.

Numerous publications on COVID-19 treatments were later retracted due to issues such as^34-36:

• Study methodology, including lack of appropriate randomisation
• Inclusion of claims within the conclusion that were not supported by study methods
• Flaws in source data, and concerns about data integrity and fraud

Concerningly, some of these publications continued to be cited in literature after being retracted^34,37.

What about the impact of flawed clinical research on clinical decision-making outside the context of the COVID-19 pandemic?

While this is more challenging to define, it is clear that issues with the quality and accuracy of clinical research publications can have a flow-on effect, especially related to aggregation of data from multiple trials.

Flaws in the design, conduct and reporting of individual clinical trials may be carried over into systematic reviews – which can magnify the impact of inadequate data⁷

Systematic reviews may also be susceptible to the effects of fraudulent data⁷. One study reported that exclusion of fabricated trials would likely have changed the conclusions of at least one out of 25 systematic reviews in the field of analgesia and anaesthesiology³⁸.

Flawed clinical research can also impact guideline recommendations. It has been suggested that probable fabricated data may have contaminated previous WHO guideline recommendations relating to use of high-concentration oxygen and surgical site infection^7,39. These guidelines were originally based upon a meta-analysis of 15 RCTs, of which two were authored by a research group implicated in data integrity issues and plagiarism³⁹. Although the guideline was later updated to exclude the studies in question³⁹, this demonstrates the potential impact poor quality data can have on the conclusions of meta-analyses, as well as guideline recommendations that can directly inform clinical practice.

Why do these quality issues continue to occur in research?

In the context of the COVID-19 pandemic, the rapid pursuit of effective options to prevent and treat COVID-19 led to a focus on research quantity over quality, and may have contributed to numerous retractions^14,40.

Other ongoing issues that likely contribute to poor quality and flawed published research include pressure on researchers to publish at high rates^32,41, obtain research funds³² and deliver results for funding sources³². The quality and accuracy of published research may also be influenced by insufficient training on good research practice and ethics¹⁴, lack of adequate quality control checks^7,21, and ineffective management of conflicts of interest^32,42.

Improving transparency and detection: How far have we come?

Although steps have been taken towards improved transparency in publishing clinical trials over the past decade, there is still a long way to go.

Some journals require study protocols, their amendments and statistical analysis protocols to be submitted with manuscripts during the submission process⁴³ (e.g., Annals of Internal Medicine⁴⁴, The New England Journal of Medicine⁴⁵). However, journal publishing requirements are variable¹¹ and standards for reporting are not always followed in practice²⁰. Overburdened peer reviewers may not have sufficient time – or be asked – to cross-check source data and other information²¹.

There is still a lot of trust placed into authors of clinical trial publications¹¹ – we are trusting that they have followed the methods as stated, and have accurately reported complete data from the trial.

Where ‘problematic’ RCTs are published, the INSPECT-SR project aims to prevent them contaminating the results of meta-analyses. The aim of the project is to develop and prospectively test a tool to assess the trustworthiness of RCTs, helping researchers to identify and exclude those with a high probability of containing false data⁴⁶.

Where to next?

RCTs remain an important source of evidence for evaluating the safety and effectiveness of healthcare interventions^4,5. However, the quality and accuracy of data reported in clinical trial publications remains a cause for concern¹⁴.

Ongoing issues in clinical trial design, conduct and reporting support a cautious approach to interpreting results of clinical trial publications – particularly if results have not been replicated across multiple studies, or if the trial data have not been subjected to appropriate quality assessments or critical appraisal

So, how can healthcare professionals navigate these issues as they endeavour to implement evidence-based practice? Read the next article in this three-part series to explore this predicament and some potential solutions in more detail.

Read article 2 in this series - Interpreting clinical trial results through a frame of reference: Why take the time?

Would you prefer to skip to key practical considerations that may assist healthcare professionals in using a more critical approach when reading clinical trial publications? Click the link below.

Read article 3 in this series - Looking under the surface of clinical trials: Are the findings valid and relevant to your patients?

Explore medical education and publication summaries on Medthority

Explore a range of independent and sponsored medical education across therapeutic areas including cardiology, immunology, oncology, and more. Upskill on key developments via continuing medical education in various formats including Learning Zones, with expert insights from key opinion leaders in videos and podcasts, as well as webinars, quizzes and more. Medthority Learning Zones also feature publication digests, which provide concise summaries of key publications across different fields.

Explore medical education content by specialty on Medthority