Good Science and Good Medicine
Proposing the Observation-enriched Randomized Controlled Trial (ORCT)
When it's not feasible or ethical to do a randomized controlled trial
“What good is a statistically perfect, well-designed trial if nobody shows up?"
INTRODUCTION
 The NCI’s CTAC (Clinical Trials and Translational Research Advisory Committee) met for the 22nd time on March 12, 2014 in their ongoing effort to improve efficiency and effectiveness of cancer clinical trials. A significant portion of the meeting addressed lagging patient accrual numbers. An important slide entitled “Analysis of Accrual for NCI Cooperative Group Phase III Trials Activated 2000-2010” explained that 254 such trials were activated during this period. Of this total 51 are still open and accruing (with 1/5 of them <90% accrued), and for 203 trials accrual has stopped. Among those 203 trials which have stopped accruing patients 119 met their goal by at least 90%, but 84 did not. Fifty-three trials (50 adult and 3 pediatric) described by the committee as “the troublesome trials” were closed solely because of inadequate accrual rate.
A second slide summarized the 254 trials during this period by stating that 24.4% of all adult cancer trials ended with <90% accrual because of inadequate accrual rates. There was significant discussion around the table regarding the human and financial costs of these failed trials as well as the reasons for the poor accrual. Trial arms with significantly different interventions, felt by patents and their doctors to lack equipoise and randomization to very disparate arms were felt to be the main reason for the 24.4% accrual failures. There seemed to be a sense of “nothing can be done; that’s just the way it is”.
We however feel that something CAN be done. Please allow us to explain our proposal for the ORCT (The Observation-enriched Randomized Controlled Trial).
Introducing ourselves and our proposal
 Jim Omel is a medical doctor and survivor of myeloma. As a dedicated patient advocate Jim has long-advocated for a patient's right to choose the arm of the study they would like to participate in - as a way to do what is both right for the patient (honor their preferences, clinical factors, and expectations), and also as a way to overcome the dismal rates of accrual - particularly in randomized trials which must be completed to make progress against cancer.
I am Karl Schwartz, a patient advocate with no formal credentials in study design. I have however twelve years experience in the review of clinical trial concepts and protocols. Jim and I are colleagues in advocacy. We share a passion for making the clinical trial system as efficient as it can be to serve the urgent needs of patients facing cancer.
Here we are seeking input on the ORCT (The Observation-enriched Randomized Controlled Trial) - as an additional tool to consider when comparing treatments for cancer - such as (but not limited to) when the compared interventions have very different risks, or when both treatment protocols can be used off-study.
The ORCT lets the patient either (1) choose to be randomized or (2) pick the study arm they want to be in. Their arm selection may be based on their own expectations, preference, or their unique clinical risk factors. Such decisions will most often be guided by the patient's physician.
Mary, for example, might prefer the study arm that would not put her fertility at risk. John might prefer the treatment arm that appears to have the greater chance to achieve a cure. Tom, having no opinion or expectation about which is better, might well choose to let a computer decide.
We want underscore that we are not advocating for replacing the Randomized Controlled Trial (RCT) design when it's feasible and ethical to use it. We are proposing The Observation-enriched Randomized Controlled Trial (ORCT) as a way do controlled trials when patients are unlikely to accept randomization - when clinical equipoise is lacking.
In particular, we are seeking input from statisticians who are the most qualified to tell us if the ORCT can be a better choice than RCT - and large single-arm studies with historical controls - in select cases, such as when the study arms have very different goals or risk factors making full enrollment infeasible.
Together Jim and I have drafted a proposal for a novel type of controlled clinical trial - a hybrid system - one that we hope can help to reliably answer the study question, solve the enrollment problem, and also address the ethical concern of forcing patients to be randomized - particularly for studies that lack equipoise.
* Equipoise is defined as a study where there is genuine uncertainty regarding which of the compared protocols is superior by patients and physicians.
Illustrating the ORCT
(The Observation-enriched Randomized Controlled Trial)
A ORCT can have any number of arms, but for illustration we will describe a trial containing two arms ... (1) Regular treatment arm, and (2) New Drug arm. An oncologist might present the trial to our patient Mary as follows:
Mary, this is your condition (natural history) …
This is the regular way to treat your condition …
We can do the regular treatment of course.
You have another choice – a clinical trial, comparing the regular way to an experimental approach.
- This is the background on the study drug – the preliminary evidence of efficacy and risks – about which we have less experience and therefore less certainty. The preliminary evidence shows there might be a noticeable benefit over the standard approach. That is the reason we are doing the trial.
If you choose to be in the ORCT trial you will have three choices, Mary:
.gif) |
You can decide to let a computer assign you by chance to one or the other study arm.
Let the computer decide if you, like me, have no idea which is better and are willing to leave the decision to chance.
Doing it this way helps to answer the question more objectively.
|
|
You can choose the regular treatment
|
|
You can choose the study drug
|
However, choosing to be in the study requires extra procedures that would not be part of regular care.
Because being in the study requires that you contribute time, pay for travel expense, and perhaps lose time off work,
and because there is some discomfort for doing these extra tests, you will receive a modest compensation (as for jury duty).
Also Mary, all the expenses are NOT your expense or your insurance company’s. They are paid for by the trial itself ... not by you.
Each of these tests will better help us to help patients with your disease who follow after you Mary.
You will be greatly helping each one of them.
Note: It must be a modest compensation/reimbursement so to avoid selecting mostly patients who
have lower income – who may have different risk factors than the general population with this condition.
ORCT at a glance PDF
Discussion:
|
With this design we will get a mix of participants in terms of selection method (by patient-physician/computer)
and in the number of patients assigned to each arm.
|
|
Depending on the appeal of the study drug and the efficacy and risks of regular care, the study arms
will be out of balance, such as:
15% may choose the regular arm
Many “Marys” will choose this if they want what is known and “safe”.
55% may choose the study drug arm
Many “Marys” will choose this when the regular treatment is not very effective or is very toxic.
They may want a better chance to kill their cancer and will gladly take a “risk” to accomplish it (or vice versa, depending on the preliminary evidence for the study drug, and the efficacy of the regular treatment).
30% may choose to be randomized.
These are the “Marys” who truly do not care. They are also the TRUE MEASURE of equipoise of the trial.
In this example, 30% would receive the regular treatment, and 70% would receive the study drug.
With this distribution the statisticians can tell us how many participants are needed to get a reliable answer
to the study question, or as Jane Perlmutter, Ph.D. suggested, accrual might be limited to the rapidly enrolling cells:
|
|
Treatment A
|
Treatment B
|
Mean
|
|
Patient Chooses
|
Cell 1
|
Cell 2
|
Marginal 1
|
|
Patient is Randomized
|
Cell 3
|
Cell 4
|
Marginal 2
|
|
Mean
|
Marginal 3
|
Marginal 4
|
|
We note that the distribution to the study arms objectively measures equipoise;
and that clinical trials do not require an equal number of patients in each arm.
There are examples of randomized trials that are 2:1
|
We appreciate that there will be a need to provide incentives to the participants who choose the regular treatment in the study, if extra tests are required beyond standard care. To address this issue it may be necessary to limit the number of tests required by participants choosing the regular arm to only those which are needed to compare efficacy and toxicity. This will make their decision to participate compare well to choosing regular treatment off study. These extra tests can be optional and also encouraged by financial incentives - along with providing the rationale for giving the tests. It should be emphasized that these tests are done to help future patients or to more accurately monitor for response and toxicity.
The Current System is not working
We note also that there are many reports describing a crisis in clinical research ... the very low enrollment rate in clinical trials ... indicating that the system is not working for patients, researchers, or for drug sponsors:
The Oncologist, 2008: One in Five Cancer Clinical Trials Is Published:
A Terrible Symptom—What's the Diagnosis? http://bit.ly/18P82T5
-
"half of the unpublished trials have failed to accrue and reach endpoints;
-
Nearly 60% of trials opened for 5 years had fewer than five patients enrolled at each site, and,
for >20% of studies, not a single subject had been accrued.
-
of all NCI phase I, II, and III trials opened and closed between 2000 and 2007,
only 50%–60% achieved minimal stated accrual goals.
-
So, while perhaps only one in five cancer clinical trials is ever published,
of those unpublished, a significant percentage probably died for lack of accrual.
Our online survey of patients shows that randomization is the main reason for declining to participate in a study - and that the recommendation of an oncologist is the primary reason for considering and participating in a study:
|
Interest, attitudes, and participation in clinical trials among lymphoma patients with online access -
ASCO 2009 JCO
|
As expected, the discussion of clinical trials with the patient's oncologist was associated with the highest consideration (85%) and participation (53%) rates, suggesting a need to increase awareness of study protocols among treating physicians so that this discussion can become more routine.
Patient issues and perceptions regarding randomization, study risk, eligibility, and tests and procedures suggest an opportunity to improve enrollment in clinical trials by focusing on these aspects of study design, specifically, attending to the rationale of the protocol as a treatment decision.
Addressing expected concerns about bias
This is the main weakness of the ORCT study design ... as it is for single-arm trials.
We feel that this shortcoming could be mitigated by the following:
|
The ORCT design will be more attractive to patients and referring physicians, allowing for larger studies and faster accrual - larger studies that may be especially important to carry out for the discovery or validation of biomarkers.
|
|
Participating physicians have the greatest influence on which arm the patient will choose.
This source of bias can be minimized by random selection of study doctors.
|
|
The participating physicians can agree to refer consecutive patients to the trial.
Because the ORCT design allows for patient choice, study doctors will likely feel there is less patient
coercion compared to trials which force patients to be randomized.
|
|
The study doctors can capture the reason for choosing one arm or the other -
helping to interpret the outcomes and helping to determine if a larger study is needed.
When a patient is guided by a physician to choose a safer arm of the study because of a specific risk factor (such as retaining fertility) this improves the ethics of referring the patient to the trial. (Good medicine)
|
|
Prognostic indexes, biomarkers, and refined eligibility criteria can be used increasingly to test for imbalances
in the study arms. (Good science)
Statisticians may be able to predefine rules to censor outcomes based on prognostic factors
or apply methods to achieve balance in the observational arms, such as with Propensity Score Methods
As we understand it propensity scoring has to anticipate and account for such confounding variables in order adjust for bias.
Here we ask if the type of disease can influence the reliability of propensity scoring to account for bias when comparing interventions. For example, heart disease could have more variables that may influence the course of the disease, such as diet, quality of sleep, anxiety, and belief in the value of the chosen intervention.
For lymphoma at least, prognostic indexes guide clinical decisions and risk stratification in trials (age, tumor bulk, co-morbidities, stage, and so on). For cancer the variables that may guide a choice of study arms appear less formidable to account for. Also for cancer there is no placebo effect - or observed influence of life style. The main determinant of treatment resistance is the biology of disease, which is often unknown.
|
|
Subset analysis - “a trial within a trial” - can evaluate whether attending physician or patient selection bias
actually has any effect on treatment results.
|
|
Other? We invite all - particularly experts to submit your
|
Regulatory agencies have accepted single-arm studies as the basis for accelerated approval for agents that address an unmet need and in most cases these approvals were validated by further controlled study. Here the unmet need is to make the clinical trial system more efficient and inclusive and sensitive to patient concerns, again, in select cases – where an RCT is not feasible or ethical. Large single-arm studies evaluating approved drugs have helped to guide clinical practice, as have the outcomes of small randomized trials.
… We should not let the imperfect be the enemy of the good. The degree of study bias in a ORCT will depend on the distribution to the study arms and might be mitigated by the methods described previously – or by novel ideas not yet considered. The natural history of the disease in the eligible population is another factor which can influence the need for a randomized control. For some indications the outcomes with the control are fairly predictable, such as in the relapsed setting for patients with unfavorable prognostic markers.
Our understanding is that there is no rule requiring randomization as the sole method of control in a pivotal clinical trial. Indeed, in a study of patients with disease that is refractory to regular treatments, each eligible patient is the control. What is critical is that the findings from the study are judged persuasive by expert consensus – and that the outcomes reasonably guide clinical practice or approval.
So, for example, the ORCT might be a used as an alternative to:
|
large phase II single-arm studies
|
|
the single-arm study for accelerated approval
|
|
the randomized comparison of two interventions with very different risks and approaches
Such as when comparing biomarker-based targeted drugs to regular treatment
|
|
the randomized comparison of two interventions that can each be used off study (comparative effectiveness research)
|
|
a RCT where equipoise is deficient (or controversial) for an otherwise valid study question
|
|
as an alternative to terminating a RCT due to poor enrollment.
|
Our first official responses from statisticians
As noted we are seeking input from statisticians on our proposal. Here we provide official responses received to date.
Most recently, Dr. Don Berry wrote:
You are to be congratulated for expanding the envelope of research approaches. We need fresh ideas for moving into the Brave New World of cancer clinical trials. And yours is one of them. Some recipients of your note will view your suggestion as a waste of time, but I'm enthusiastic about pursuing it.
Especially as biologists slice and dice cancers into ever smaller pieces we must learn how to exploit outcome information from non-randomized patients. There's substantial potential for bias, of course, but RCTs are themselves not immune to bias.
Incorporating randomization while augmenting with information from patient-selection cohorts to the extent the two evince consistent treatment effects can only be positive--when carefully done!
One possible method of analysis is Bayesian hierarchical modeling. A possible design is Bayesian adaptive wherein the extent of borrowing across the two groups is evaluated as data accumulate. In the case of sufficiently different treatment effects there would be little borrowing and so the randomized cohort would have to be larger, perhaps even as large as what is traditional today. There may be a slight statistical penalty to be paid but the potential upside is huge!
For those interested in reading more about PCT trials I suggest the book Optimizing Health: Improving the Value of Healthcare Delivery by Franz Porzsolt and Robert M. Kaplan (Springer: New York, 2006), available at Amazon.
It reviews some actual PCT trials. My assessment is that there tends to be concordance between the treatment effects for the two groups (although not necessarily concordance within subsets having the same treatment, which should not be a requirement anyway). Exceptions tend to be for outcomes such as satisfaction with treatment selected, which is hardly surprising.
By the way, "patient-selected clinical trial" is an unfortunate name because it suggests no randomization, which to me is a non-starter. You need a more descriptive name, one that conveys the notion that you're not replacing randomization but complementing it.
… Brewin and Bradley in BMJ (1989) used the term "partially randomized patient preference trial" (PRPPT). This is appropriately descriptive but the acronym is ghastly.
[Note: We have change the name accordingly to ORCT (Observation-enriched RCT)]
I have two ancillary comments about your proposal. First, I've never appreciated the potential for propensity scores beyond using standard covariate analyses. Second, I don't think the route to changing research practice goes through the ASCO Post.
I suggest that we form a group of folks interested in encouraging the development of PCTs in cancer. Perhaps some of the recipients of this message will volunteer for such a group, either by responding to "all" or just to you or just to me. And I think I can speak for you in saying that we would be grateful to hear negative comments, preferably aired by responding to "all."
Dr. Rademaker wrote:
"Bottom line is that through propensity scores, you can try to adjust for any biases So, there is hope. I have copied Joseph Kang on this email. These are the concepts PDF, doing it would take someone who is familiar with the concepts and has experience working with these designs.
Alfred W Rademaker, Ph.D.
Director, Biostatistics Collaboration Center (BCC)
Professor in Preventive Medicine
Dr. Joseph Kang:
"No one can possibly object that randomization will remove biases due to residual confounding. Indeed randomization is the gold-standard method in causal inference for any clinical studies.
Propensity score methods aim to turn an observational data set into a pseudo-randomized data set where subjects across different intervention arms can be balanced with respect to measured confounders (such as risk factors at baseline). Numerous epidemiologists and biostatisticians (currently 12,482 publications only in NCBI http://www.ncbi.nlm.nih.gov/pmc/?term=propensity score ) have used propensity scores in various studies. They also developed sensitivity analysis methods for the residual confounding issue as well.
Statistical schema for Propensity scoring provided by Dr. Kang PDF
-Joseph Kang, Ph.D.
Northwestern University Feinberg School of Medicine
Dr. Rick Chappell:
"Dr. Kang’s document is a useful approach, but I would add the warning that propensity scores or other adjustments cannot substitute for randomized clinical trials. There will always be the possibility of bias due to some factor which is associated with the patients' choice and with the outcome (a confounder). If this confounder isn't properly incorporated into the propensity score for example, if it isn't measured), then the apparent treatment effect will be biased. However, an patient-selected or other nonrandomized control is still better than no control at all.
It seems to me that the only true measure is to compare your results with an RCT's. So the ORCT could be a preliminary study subject to confirmation from a RCT. Of course, there are always subjective considerations as to why subjects might pick one treatment or another but I wouldn't know how to quantify these.
Rick Chappell, Ph.D.
Professor, Depts. of Statistics and of Biostatistics & Medical Informatics,
Univ. of Wisconsin Medical School
In closing, we acknowledge the statistical axiom that the RCT is the gold standard method to objectively compare treatment protocols. However, it is often not feasible to enroll patients in trials that assign patients to treatment arms by chance. Patients have shown, by refusing participation, that they will not always allow their treatment to be decided by computer or coin flip.
We recognize that the proposed ORCT design will not be appropriate in all circumstances. The ORCT is a tool to consider when clinical equipoise is deficient and therefore accrual is not feasible for a RCT. In such cases we expect that the ORCT will be more reliable as evidence than a large phase II trial using historical controls.
The ORCT can be more appropriate than an RCT when (1) either protocol in the study can be used off study, (2) when the patient's preferences or risk factors make either study arm unappealing or inappropriate, or (3) when the compared study arms have very different approaches or treatment-related risks.
Pat Gavin notes: With the rapid advancement of cancer molecular and treatment knowledge, “personalized medicine” has made the RCT less desirable, and probably even unethical. Targeting of specific cancer patients who are likely to respond to therapy makes a traditional randomized trial nearly impossible to accomplish. Consider the unfairness to a patient whose cancer molecular markers have identified him/her to be more likely to respond to a new drug, when RCT randomization has placed that patient in an arm which will not get this targeted agent.
The ORCT incorporates random assignment to study arms ... when it's an acceptable choice to the patient and referring physician. The more acceptable that choice the closer the ORCT becomes to the gold standard method of patient assignment. The ORCT will encourage participation in the trial because it allows patients to choose their therapy when a clinical risk factor makes one approach more appropriate, or when they have strong preference or aversion to one of the compared interventions.
We submit that the ORCT is clearly superior to any RCT which is never started because it’s judged to be unfeasible … or to any RCT that is terminated because of poor enrollment. We hope and expect that the ORCT provides another way to do good science while practicing good medicine.
We invite comment and guidance - in particular from statisticians in this field.
Thank you for listening and for your input in advance.
Jim Omel, MD.
Karl Schwartz, MFA
Patient research advocates
References
|
Gregory A. Curt, M.D., Senior Editor, The Oncologist and
Bruce A. Chabner, M.D., Editor-in-Chief, The Oncologist, 2008:
One in Five Cancer Clinical Trials Is Published: A Terrible Symptom—What's the Diagnosis? http://bit.ly/18P82T5
|
|
Eur Heart J. 2012: (technical - describes a research method to decrease bias in non-randomized trials.)
Do observational studies using propensity score methods agree with randomized trials?
A systematic comparison of studies on acute coronary syndromes http://1.usa.gov/18Kt7Zo
For this indication, "observational studies using PS methods produce treatment effect estimates that are of more extreme magnitude compared with those from RCTs, although the differences are rarely statistically significant."
|
|
Rev Esp Cardiol. 2011 (technical)
Propensity Score Methods for Creating Covariate Balance in Observational Studies http://bit.ly/18D8BuX
A technical paper that describes a method that could help to make The Observation-enriched Randomized Controlled Trial (ORCT) more reliable - in cases where random assignment to the study arms is not feasible because of strong patient preferences or ethical.
|
|
