Bias in RCTs - part 1
Randomization isn’t enough ... the list must be hidden.
This is the first post in a four-part series on bias in randomized controlled trials (RCTs), to be released over the coming months. Part 1 introduces the foundational concept that treatment groups must be comparable at the beginning of the trial, a goal achieved through proper randomization. This is analogous to runners starting a race in a fair manner — see previous post “RCTs are not immune to bias”. When randomization works as intended then the trial resembles a fair start line between individuals in the treatment and control groups.
Some might say “The trial is randomized and is therefore protected against selection bias and confounding”. Yes, randomization is our best method to protect against selection bias and confounding at the start of the trial! Yet, an important part of critical appraisal is checking for clues that randomization was implemented properly. Selection bias can also occur once the study has begun through selection out of the study (i.e. dropouts) but that is for a different post!
To check for a potentially flawed randomization process, I typically assess three key elements: the randomization scheme, the adequacy of allocation concealment, and the baseline comparability of patient characteristics across groups.
The randomization scheme
First, I look for evidence that a proper randomization scheme was applied to a sufficiently large patient population. Allocation sequence generation or how patients will be assigned treatments, should be done using a random number generator of some kind. I search the trial methods or study protocol to verify that a central ‘randomization office’ using a computerized system was used to for treatment allocation. I also accept other ‘adequate’ methods such as random number tables, online randomization web-sites designed for clinical trials, or pre-generated random sequences sealed in opaque, sequentially numbered envelopes. These approaches are all capable of producing genuinely random treatment assignments. For readers who are interested in understanding more details about the types of randomization, I created a separate post you can check out here.
I stay alert for methods that may appear random but aren't truly so. For instance, assigning treatment based on every other patient, day of the week, admission date, or medical record number can introduce bias. Why? Because these approaches are influenced by underlying systematic factors that may correlate with patient characteristics or outcomes, undermining the integrity of randomization.
How large or important is this potential bias? A paper published in the Annals of Internal Medicine found that inadequate or unclear sequence generation may exaggerate treatment effects by about 11%.1 That’s substantial! In fact, that is comparable to the actual relative relative treatment effect reported in some trials. In other words, an inadequate sequence generation process can make an ineffective treatment appear beneficial. To be clear, I’m not suggesting that randomization itself fails, but rather that the method used to assign treatments may not have been truly random.
Alas, a proper randomization scheme is not enough! It must also be hidden!!
Concealing randomization
The second thing I check is that the randomization process is hidden or concealed. Both the treating clinician and the patient must not be able to predict the next treatment assignment. Adequate methods of concealment include remote or centrally administered randomization. Envelopes containing treatment assignments must be sequentially ordered, sealed, and opaque. Similarly, drug containers must be sequentially numbered and identical in appearance. Envelopes must not be opened, nor drugs administered or dispensed, until after the treatment has been ‘irreversibly’ assigned. Concealing treatment allocation is not as straight forward as it may seem.
The classic example of inadequate allocation concealment is from a RCT of open versus laparoscopic appendectomy.2 The laparoscopic surgery was an annoyance for the resident physicians enrolling patients in the trial because, they needed to call in an attending surgeon if the laparoscopic procedure was done during the night and there was limited operating room availability for the newer laparoscopic procedure. This resulted in residents holding up the semi-opaque envelopes to the ceiling light to reveal the treatment assignment.
During night shifts, residents were able to manipulate treatment allocation by selectively opening envelopes. Can you guess which ones? They chose envelopes assigning patients to open surgery, avoiding laparoscopic procedures that required attending surgeons and more resources. As a result, laparoscopic surgeries were deferred to daytime hours when senior staff and operating rooms were more available. If patient characteristics differed between night and day admissions, this could introduce bias into the trial results. This example highlights the critical importance of using opaque, tamper-proof envelopes, or better yet, centralized randomization, to preserve allocation concealment and prevent selection bias.
How large is the bias introduced with inadequate allocation concealment? It depends on the nature of the outcome being measured. According to a meta-epidemiologic study published in The BMJ, trials with inadequate or unclear allocation concealment showed roughly a 31% relative increase in exaggerated benefit for trials with subjective outcomes.3 Subjective outcomes rely on judgements either by patients (e.g., self-reported symptoms) or clinicians (e.g., assessments of depression or respiratory distress), making them especially vulnerable to bias when allocation concealment is compromised.
For objective outcomes, such as all-cause death or lab-based measures such as blood glucose or hemoglobin, studies with inadequate or unclear concealment showed little difference in results compared to those with adequate concealment.3
Allocation concealment must not be confused with blinding of treatments. They are distinct concepts. The purpose of allocation concealment is to prevent clinicians and patients from knowing the treatment to be allocated before it is assigned. Blinding is about maintaining treatment concealment after randomization and throughout the conduct of the study.
Table 1 check
Third, I check for any striking imbalance in the baseline characteristics of patients in the treatment and control groups. This is typically found in table 1 with a more detailed version in the supplemental appendix. Keep in mind that, randomization doesn’t fail per se as it is possible to get imbalance by chance.4 Studies with a smaller sample size are more likely to have imbalance by chance. This does not cause bias. I am not inspecting table 1 to sort out if randomization did not work, rather I am searching for evidence of the randomization process gone awry.
Here are some clues that I look for which may indicate there were problems with the randomization process include:
Group sizes that differ more than expected from the planned allocation ratio. For example, let’s say that 500 patients were planned to be randomized in a 1:1 fashion to a drug vs placebo; however, when the trial was conducted 295 patients received the drug and 205 received placebo. This large discrepancy raises concern that the study investigators may have selectively chosen patients to the drug arm.
Too many significant baseline differences between groups to be explained by chance. About 5% of baseline characteristics are expected to be statistically different (p-value<0.05) by chance. However, if more than 1 in 20 baseline covariates are significantly different, this might indicate the randomization process was flawed.
Unusually similar baseline characteristics that likely aren't due to chance. This suggests possible data fabrication or a compromised randomization process.
In other words, an uncanny number of imbalances or none at all point toward a flawed randomization procedure!
Should baseline imbalances between treatment groups be adjusted for in the statistical analysis?5 It depends. Ideally, the trial protocol should prespecify which baseline covariates will be adjusted for. In general, adjusting for variables used to stratify the randomization process is beneficial to improve the precision of the estimated treatment effect. Adjustment for baseline covariates in the statistical analysis of an RCT is primarily intended to improve statistical precision and not to correct for bias. However, caution is warranted. If the covariates being adjusted for are mediators in the causal pathway between treatment and outcome, this can introduce bias. I would think this risk would be heightened in trials with a run-in period, although I have not seen or read any empirical evidence to support this assertation.
If you are interested in the idea of how randomization cannot “fail” or when to adjust for baseline characteristics in an RCT, check out these two papers:
Owora AH, Dawson J, Gadbury G, et al. Randomisation can do Many Things – But it Cannot “Fail”, Significance 2022;19(1):20–23.
Holmberg MJ, Andersen LW. Adjustment for Baseline Characteristics in Randomized Clinical Trials. JAMA. 2022;328(21):2155–2156.
Stay tuned for Part 2 of the “Risk of Bias in RCTs” series, coming next month! In the meantime, I’ll be sharing a few more Pharmacotherapy Pearls letters—short reflections and insights about drug-drug interactions in primary care and more!
Peace and kindness,
JM
