Looking under the surface of clinical trials: Are the findings valid and relevant to your patients?

Over the past decade there has been a seemingly endless list of concerns raised about the quality, accuracy and integrity of clinical trial publications^1-10. These problems are not rare, and continue to be reported even in high-impact journals^2-5,11-14. As clinical trial findings can influence real-world treatment decisions, these issues could result in the use of ineffective treatments, or may even lead to patient harm^2,12,15,16.

In this final article of a 3-part series, we describe some approaches that may assist healthcare professionals to use a more critical approach to reading clinical trial publications. These approaches may enable a more informed interpretation of a study’s findings. They may also assist healthcare professionals with determining whether findings are of a sufficient quality and standard to inform their clinical decision-making.

Consider the foundation of the study: Are the design and methods valid?

Before looking at the results of the study, consider the design and methods, as they provide important context and can indicate if results are valid^17,18. This may involve consideration of^17-19:

• The trial design and whether it is appropriate to address the clinical question
• Whether randomisation was valid, and if potential confounders were balanced between groups in baseline characteristics (including prognostic factors and co-administered interventions)
• Whether participants were analysed in the groups to which they were allocated (intention-to-treat, ITT); or if a per-protocol or modified ITT analysis was used, was this adequately justified and appropriate?
• The length of follow-up, and extent of loss to follow-up (particularly between groups)
• Whether the study had sufficient statistical power to detect a minimum important difference in the primary outcome of interest

A sample size calculation is critical to determine the target number of participants in an RCT. This target should be met during recruitment as well as during each time point in follow-up¹⁷. A high number of dropouts, crossover or loss to follow-up can cause a study to lose adequate power¹⁹.

Underpowered studies can be difficult to interpret (particularly when not statistically significant); whereas over-powered studies may yield results that are not clinically important¹⁹

It is also important to note that a published article is not the entire study – it is not possible for the publication to convey full details on study design, execution and all raw data collected. Further details can be found in the study protocol and statistical analysis plan²⁰.

The heart of an RCT: Study results and treatment effect

When interpreting the results of a study, consider aspects such as the magnitude of the treatment effect, its precision, the clinical importance of the estimate, and whether adherence to the intervention could have affected the outcome^18,19.

Magnitude of effect: Absolute and relative measures

The magnitude of treatment effect is often assessed via measures of association, including relative and absolute risk measures^18,21.

• Relative risk measures include risk ratio, relative risk, odds ratio and prevalence ratio²¹
• Absolute risk measures include risk difference, prevalence difference and differences in proportions, or number needed to treat²¹

When reporting the magnitude of treatment effect for binary outcomes, both relative and absolute outcomes should be reported to facilitate accurate interpretation²¹. Despite clear guidance that both relative and absolute measures should be reported in RCTs²², a 2020 review of 200 individual RCTs reported that only 9% presented both relative and absolute measures²³; while 28% reported relative treatment effect only, and 17% reported absolute treatment effect only²³.

Although relative measures can inform differences between groups and directions of association, in the absence of context from absolute measures, they can be used to overstate outcomes - particularly if outcomes are rare²¹.

A hypothetical example can help to put this into context: An relative risk of 50% could represent an absolute risk reduction of 30% (if the absolute risk improved from 60% to 30%), or an absolute risk reduction of 2% (if the absolute risk improved from 4% to 2%)¹⁸.

Inclusion of absolute risk alongside relative risk measures can provide a more complete picture when interpreting data, and assist when discussing results with patients¹⁸

Precision

Where a confidence interval is provided, this can be used to indicate precision of the treatment effect¹⁹. A 95% confidence interval represents the range of values within which we can be 95% confident the true treatment effect lies within¹⁹. A narrow confidence interval indicates a precise estimate of treatment effect, however a large confidence interval should be interpreted with caution – as the true effect can lie within a wide range of values¹⁹.

A key question to help clarify whether a result is adequately precise: Would the decision be the same whether the lower and the upper boundaries of the confidence interval were the truth? If so, then the results can be considered sufficiently precise¹⁸.

Clinical importance

To assess the clinical importance of the treatment effect, consider if the observed treatment effect is likely to translate into meaningful improvements for the patient¹⁹. This may be based on clinical acumen and experience, or established minimally important differences in literature¹⁹.

However, decisions around clinical importance can be complicated by use of surrogate outcomes¹⁸. These may or may not lead to clinically meaningful improvements in outcomes that are important to patients, such as quality of life or overall survival (in oncology, for example)¹⁸.

Statistical significance does not necessarily translate into a clinically important difference¹⁹, as the result should be large enough to be clinically meaningful¹⁷

Could adherence have impacted the outcome?

It is important to consider whether the adherence (or compliance) of study participants to the intervention and control was reported, and whether there was a difference in adherence between groups¹⁸. This is particularly relevant in oncology trials, where adverse events and medication safety profiles can have a considerable impact on adherence¹⁸.

Consider applicability: Could the study’s results apply to real-world patients?

To determine if results of a clinical trial could be relevant to individual patients in clinical practice, consider the applicability of the population and intervention used in the trial, and how well they can be applied to patients in clinical settings^17,19 (Table 1).

Table 1. Assessing applicability of a clinical trial to real-world patients^17-19.

Keep in mind the potential impact of bias

A study is considered biased when a component of its design or execution has systematic impacts on the results that may deviate from the truth²⁴.

Bias can lead to an over- or underestimation of the truth, or can compromise the validity of the study’s findings – even if all other facets of the study were appropriate²⁴

Various types of bias can impact how the results of the study are subsequently interpreted²⁴. These include biases in participant selection, trial performance, detections of outcomes, attrition (withdrawal) of participants, and reporting²⁴.

It has been suggested that when interpreting RCTs, healthcare professionals should carefully consider potential sources of bias, and the impact they may have on confidence in the trial’s results¹⁹. While healthcare professionals may not have sufficient time or resources to perform a full risk of bias assessment on each clinical trial publication, it is important to be aware that these types of bias do exist and can have a considerable impact on the validity of a trial’s findings^24,25.

In conclusion: Deciding whether to use trial results to inform patient management

When interpreting clinical trial results, it is important to remember results are influenced by a variety of factors and should not be taken at face value. Care should also be taken to avoid extrapolating results to patient populations or settings that were outside the context of the trial.

In summary, key considerations when reading a clinical trial publication include:

• Are the design and methods valid?
• What are the study results and treatment effect?
• Is the trial population relevant to real-world patients and settings?
• Are there potential sources of bias in the study that could compromise the validity of the study’s findings?
• Are the results of a sufficient quality and standard to inform your clinical decision-making?

Shared-decision making is an important consideration when applying evidence to individual patients^18,20. This includes consideration of whether the treatment benefits outweigh the potential risks, harms and costs, as well as the patient’s individual preferences and circumstances^17,18,20.

Missed the earlier articles in this series?

Read article 1 in this series - How can flaws in clinical trial design, conduct and reporting impact clinical decision-making?

Read article 2 in this series - Interpreting clinical trial results through a frame of reference: Why take the time?

Further reading for healthcare professionals

Tips on reading clinical trial publications

• Thabane A, Phillips MR, Wong TY, Thabane L, Bhandari M, Chaudhary V, et al. The clinician’s guide to randomized trials: interpretation. Eye. 2022;36(3):481-482. DOI: 10.1038/s41433-021-01866-7
• Sonbol MB, Firwana BM, Hilal T, Murad MH. How to read a published clinical trial: A practical guide for clinicians. Avicenna J Med. 2020;10(2):68-75. DOI: 10.4103/ajm.ajm_186_19
• Bauchner H, Golub RM, Fontanarosa PB. Reporting and Interpretation of Randomized Clinical Trials. JAMA. 2019;322(8):732-735. DOI: 10.1001/jama.2019.12056

Useful explanations on types of bias in clinical trials

• Phillips MR, Kaiser P, Thabane L, Bhandari M, Chaudhary V, Wykoff CC, et al. Risk of bias: why measure it, and how? Eye. 2022;36(2):346-348. DOI: 10.1038/s41433-021-01759-9
• Lewis SC, Warlow CP. How to spot bias and other potential problems in randomised controlled trials. J Neurol Neurosurg Psych. 2004; 75: 181-187. DOI: 10.1136/jnnp.2003.025833

Critical appraisal – where to start?

Critical appraisal checklists have been developed to guide the reader through the appraisal process, prompting them to ask certain questions of the study publication they are appraising²⁶. The type of checklist is tailored to the type of study being appraised, for example if it is a randomised controlled trial or observational cohort study²⁶.

• Refer to BMJ best practice for a concise list of useful critical appraisal tools
• A useful publication with guidance on critical appraisal for pharmacists and other clinicians: Bashir S, Dziemidowicz K. Critical appraisal: how to evaluate research for use in clinical practice. The Pharmaceutical Journal; 2021

Explore medical education and publication summaries on Medthority

Explore a range of independent and sponsored medical education across therapeutic areas including cardiology, immunology, oncology and more. Upskill on key developments via continuing medical education in various formats including Learning Zones, with expert insights from key opinion leaders in videos and podcasts, as well as webinars, quizzes and more. Medthority Learning Zones also feature publication digests, which provide concise summaries of key publications across different fields.

Explore medical education content by specialty on Medthority