Science and Good Medicine
Proposing the Observation-enriched Randomized Controlled
not feasible or ethical to do a randomized controlled trial
“What good is a statistically perfect, well-designed trial if
nobody shows up?"
NCI’s CTAC (Clinical Trials and Translational Research Advisory
Committee) met for the 22nd time on March 12, 2014 in
their ongoing effort to improve efficiency and effectiveness of
cancer clinical trials. A significant portion of the meeting
addressed lagging patient accrual numbers. An important slide
entitled “Analysis of Accrual for NCI Cooperative Group Phase
III Trials Activated 2000-2010” explained that 254 such trials
were activated during this period. Of this total 51 are still
open and accruing (with 1/5 of them <90% accrued), and for 203
trials accrual has stopped. Among those 203 trials which have
stopped accruing patients 119 met their goal by at least 90%,
but 84 did not. Fifty-three trials (50 adult and 3
pediatric) described by the committee as “the troublesome
trials” were closed solely because of inadequate accrual rate.
A second slide summarized the 254 trials
during this period by stating that 24.4% of all
adult cancer trials ended with <90% accrual because of
inadequate accrual rates. There was significant discussion
around the table regarding the human and financial costs of
these failed trials as well as the reasons for the poor
accrual. Trial arms with significantly different interventions,
felt by patents and their doctors to lack equipoise and
randomization to very disparate arms were felt to be the main
reason for the 24.4% accrual failures. There seemed to be a
sense of “nothing can be done; that’s just the way it is”.
We however feel that something CAN be
done. Please allow us to explain our proposal for the ORCT
(The Observation-enriched Randomized Controlled Trial).
Introducing ourselves and our
Jim Omel is a
medical doctor and survivor of myeloma. As a dedicated patient
advocate Jim has long-advocated for a patient's right to choose
the arm of the study they would like to participate in - as a
way to do what is both right for the patient (honor their
preferences, clinical factors, and expectations), and also as a
way to overcome the dismal rates of accrual - particularly in
randomized trials which must be completed to make progress
am Karl Schwartz, a patient advocate with no formal credentials
in study design. I have however twelve years experience
in the review of clinical trial concepts and protocols. Jim
and I are colleagues in advocacy. We share a passion for
making the clinical trial system as efficient as it can be to
serve the urgent needs of patients facing cancer.
Here we are
seeking input on the ORCT (The Observation-enriched
Randomized Controlled Trial)
- as an additional tool to
consider when comparing treatments for cancer - such as (but not
limited to) when the compared interventions have very different
risks, or when both treatment protocols can be used off-study.
The ORCT lets the patient
either (1) choose to be randomized or (2) pick the study arm
they want to be in. Their arm selection may be based on their
own expectations, preference, or their unique clinical risk
factors. Such decisions will most often be guided by the
Mary, for example, might prefer the study arm that would not put
her fertility at risk. John might prefer the treatment arm that
appears to have the greater chance to achieve a cure. Tom,
having no opinion or expectation about which is better, might
well choose to let a computer decide.
We want underscore that we are not
advocating for replacing the Randomized Controlled Trial (RCT)
design when it's feasible and ethical to use it. We are
proposing The Observation-enriched Randomized Controlled Trial (ORCT) as a way
do controlled trials when patients are unlikely to accept
randomization - when clinical equipoise is lacking.
In particular, we are seeking
input from statisticians who are the most qualified
to tell us if the ORCT can be a better choice than RCT - and
large single-arm studies with historical controls - in select
cases, such as when the study arms have very different goals or
risk factors making full enrollment infeasible.
and I have drafted a proposal for a novel type of controlled
clinical trial - a hybrid system - one that we hope can help to reliably
answer the study question, solve the enrollment problem, and
also address the ethical concern of forcing patients to be
randomized - particularly for studies that lack equipoise.
* Equipoise is defined as a study where there is genuine
uncertainty regarding which of the compared protocols is superior
by patients and physicians.
(The Observation-enriched Randomized Controlled Trial)
A ORCT can have
any number of arms, but for illustration we will describe a
trial containing two arms ... (1) Regular treatment arm, and (2)
New Drug arm. An oncologist might present the trial to our
patient Mary as follows:
Mary, this is your condition (natural history) …
This is the regular way to treat your condition …
We can do the regular treatment of course.
You have another choice – a clinical trial, comparing the
regular way to an experimental approach.
- This is the
background on the study drug – the preliminary evidence of
efficacy and risks – about which we have less experience and
therefore less certainty. The preliminary evidence shows there
might be a noticeable benefit over the standard approach. That
is the reason we are doing the trial.
If you choose
to be in the ORCT trial you will have three choices, Mary:
decide to let a computer assign you by chance to
one or the other study arm.
Let the computer decide if you, like me, have no idea
which is better and are willing to leave the decision to
Doing it this way helps to answer the question more
choose the regular treatment
choose the study drug
However, choosing to be in the study requires extra
procedures that would not be part of regular care.
Because being in the study requires that you
contribute time, pay for travel expense, and perhaps lose time
and because there is some discomfort for doing
these extra tests, you will receive a modest compensation (as for
Also Mary, all the
expenses are NOT your
expense or your insurance company’s. They are paid for by the
trial itself ... not by you.
Each of these tests will better help us to help
patients with your disease who follow after you
You will be greatly helping each one of them.
Note: It must be a modest
compensation/reimbursement so to avoid selecting
mostly patients who
have lower income – who may have different risk
factors than the general population with this
ORCT at a glance
this design we will get a mix of participants in terms
of selection method (by patient-physician/computer)
and in the number of patients assigned to each arm.
Depending on the appeal of the study drug and the
efficacy and risks of regular care, the study arms
will be out of balance, such as:
15% may choose the regular arm
Many “Marys” will choose this if they want what is
known and “safe”.
may choose the study drug arm
Many “Marys” will choose this when the regular
treatment is not very effective or is very toxic.
They may want a better chance to kill their cancer and
will gladly take a “risk” to accomplish it (or vice versa, depending on the preliminary
evidence for the study drug, and the efficacy of the
may choose to be randomized.
These are the “Marys” who truly do not care. They
are also the TRUE MEASURE of equipoise of the trial.
example, 30% would receive the regular treatment, and
70% would receive the study drug.
With this distribution the statisticians can tell us how
many participants are needed to get a reliable answer
the study question,
or as Jane Perlmutter, Ph.D.
suggested, accrual might be limited to
the rapidly enrolling cells:
Patient is Randomized
that the distribution to the study arms objectively measures
and that clinical trials do not require an equal
number of patients in each arm.
There are examples of randomized trials that are 2:1
We appreciate that there
will be a need to provide incentives to the participants who
choose the regular treatment in the study, if extra tests are
required beyond standard care. To address this issue it may be necessary to limit the
number of tests required by participants choosing the regular
arm to only those which are needed to compare efficacy and
toxicity. This will make their decision to participate compare
well to choosing regular treatment off study. These extra tests
can be optional and also encouraged by financial incentives -
along with providing the rationale for giving the tests. It
should be emphasized that these tests are done to help future
patients or to more accurately monitor for response and
The Current System is not working
We note also
that there are many reports describing a crisis in clinical
research ... the very low enrollment rate in clinical trials ...
indicating that the system is not working for patients,
researchers, or for drug sponsors:
The Oncologist, 2008: One
in Five Cancer Clinical Trials Is Published:
A Terrible Symptom—What's the Diagnosis?
"half of the unpublished
trials have failed to accrue and reach
Nearly 60% of trials opened for 5
years had fewer than five patients enrolled at
each site, and,
for >20% of studies, not a single subject
had been accrued.
of all NCI phase I, II, and III
trials opened and closed between 2000 and
only 50%–60% achieved minimal stated
So, while perhaps only one in five
cancer clinical trials is ever published,
of those unpublished,
a significant percentage
probably died for lack of accrual.
Our online survey of patients shows that randomization is the
main reason for declining to
participate in a study - and that the recommendation of an
oncologist is the primary reason for considering and
participating in a study:
Interest, attitudes, and participation in clinical
trials among lymphoma patients with online access
As expected, the discussion of clinical trials with the
patient's oncologist was associated with the highest
consideration (85%) and participation (53%) rates,
suggesting a need to increase awareness of study protocols
among treating physicians so that this discussion can become
Patient issues and perceptions regarding
randomization, study risk, eligibility, and tests and
procedures suggest an opportunity to improve enrollment in
clinical trials by focusing on these aspects of study
design, specifically, attending to the rationale of the
protocol as a treatment decision.
Addressing expected concerns about bias
This is the main weakness of the
design ... as it is for single-arm trials.
We feel that this shortcoming could be mitigated by the following:
The ORCT design will be more attractive to
patients and referring physicians, allowing for
larger studies and faster accrual -
larger studies that may be especially important to carry out
for the discovery or validation of biomarkers.
Participating physicians have the greatest
influence on which arm the patient will choose.
This source of bias can be minimized by
random selection of
The participating physicians can agree to
refer consecutive patients to the trial.
Because the ORCT design allows for patient
choice, study doctors will likely feel there is
coercion compared to trials which force patients
to be randomized.
The study doctors can
capture the reason for
choosing one arm or the other -
helping to interpret the outcomes and helping to
determine if a larger study is needed.
When a patient is guided by a physician to
choose a safer arm of the study because of a
specific risk factor (such as retaining
fertility) this improves the ethics of referring the
patient to the trial. (Good medicine)
biomarkers, and refined eligibility
criteria can be used increasingly to test for
in the study arms. (Good science)
Statisticians may be able to predefine rules to
censor outcomes based on prognostic factors
or apply methods to achieve balance in the observational
arms, such as with
Propensity Score Methods
As we understand it propensity scoring has to anticipate and
account for such confounding variables in order adjust for
Here we ask if the type of disease can influence the
reliability of propensity scoring to account for bias when
comparing interventions. For example, heart
disease could have more variables that may influence the
course of the disease, such as diet, quality of sleep,
anxiety, and belief in the value of the chosen intervention.
For lymphoma at least,
prognostic indexes guide clinical decisions and risk
stratification in trials (age, tumor bulk, co-morbidities,
stage, and so on). For cancer the variables that may guide a
choice of study arms appear less formidable to account for. Also for cancer there is no placebo effect - or
observed influence of life style. The main determinant of
treatment resistance is the biology of disease, which is
Subset analysis - “a trial within a trial” - can
evaluate whether attending physician or patient
actually has any effect on treatment results.
We invite all - particularly experts to submit
suggestions or comments
have accepted single-arm studies as the basis for accelerated
approval for agents that address an unmet need and in most cases
these approvals were validated by further controlled study. Here
the unmet need is to make the clinical trial system more
efficient and inclusive and sensitive to patient concerns, again, in select cases – where an RCT
is not feasible or ethical. Large single-arm studies
evaluating approved drugs have helped to guide clinical
practice, as have the outcomes of small randomized trials.
… We should not let the imperfect be the enemy of the good. The
degree of study bias in a ORCT will depend on the distribution to the
study arms and might be mitigated by the methods described
previously – or by novel ideas not yet considered.
The natural history of the disease in the eligible population is
another factor which can influence the need for a randomized control.
For some indications the outcomes with the control are fairly
predictable, such as in the relapsed setting for patients with
unfavorable prognostic markers.
understanding is that there is no rule requiring randomization
as the sole method of control in a pivotal clinical trial.
Indeed, in a study of patients with disease that is refractory
to regular treatments, each eligible patient is the control.
What is critical is
that the findings from the study are
judged persuasive by expert consensus – and that the outcomes reasonably guide clinical
practice or approval.
So, for example, the
ORCT might be a
used as an alternative to:
large phase II
study for accelerated approval
randomized comparison of two interventions with very different
risks and approaches
Such as when comparing biomarker-based targeted drugs to
randomized comparison of two interventions that can each be used
off study (comparative effectiveness research)
a RCT where equipoise is
deficient (or controversial) for an otherwise valid study question
as an alternative to
terminating a RCT due to poor enrollment.
Our first official responses from statisticians
As noted we are seeking
input from statisticians on our proposal. Here we
provide official responses received to date.
Most recently, Dr. Don Berry
You are to be congratulated for expanding the envelope
of research approaches. We need fresh ideas for moving into
the Brave New World of cancer clinical trials. And yours is
one of them. Some recipients of your note will view your
suggestion as a waste of time, but I'm enthusiastic about
Especially as biologists slice and dice cancers into ever
smaller pieces we must learn how to exploit outcome
information from non-randomized patients. There's
substantial potential for bias, of course, but RCTs are
themselves not immune to bias.
Incorporating randomization while augmenting with
information from patient-selection cohorts to the extent the
two evince consistent treatment effects can only be
positive--when carefully done!
One possible method of analysis is Bayesian hierarchical
modeling. A possible design is Bayesian adaptive wherein the
extent of borrowing across the two groups is evaluated as
data accumulate. In the case of sufficiently different
treatment effects there would be little borrowing and so the
randomized cohort would have to be larger, perhaps even as
large as what is traditional today. There may be a slight
statistical penalty to be paid but the potential upside is
For those interested in reading more about PCT trials I
suggest the book Optimizing Health: Improving the Value of
Healthcare Delivery by Franz Porzsolt and Robert M. Kaplan
(Springer: New York, 2006), available at Amazon.
It reviews some actual PCT trials. My assessment is that
there tends to be concordance between the treatment effects
for the two groups (although not necessarily concordance
within subsets having the same treatment, which should not
be a requirement anyway). Exceptions tend to be for outcomes
such as satisfaction with treatment selected, which is
By the way, "patient-selected clinical trial" is an
unfortunate name because it suggests no randomization, which
to me is a non-starter. You need a more descriptive name,
one that conveys the notion that you're not replacing
randomization but complementing it.
… Brewin and Bradley in BMJ (1989) used the term "partially
randomized patient preference trial" (PRPPT). This is
appropriately descriptive but the acronym is ghastly.
[Note: We have change the name accordingly to ORCT
I have two ancillary comments about your proposal. First,
I've never appreciated the potential for propensity scores
beyond using standard covariate analyses. Second, I don't
think the route to changing research practice goes through
the ASCO Post.
I suggest that we form a group of folks interested in
encouraging the development of PCTs in cancer. Perhaps some
of the recipients of this message will volunteer for such a
group, either by responding to "all" or just to you or just
to me. And I think I can speak for you in saying that we
would be grateful to hear negative comments, preferably
aired by responding to "all."
Dr. Rademaker wrote:
line is that through propensity scores, you can try to
adjust for any biases So, there is hope. I have copied
Joseph Kang on this email. These are the concepts
PDF, doing it
would take someone who is familiar with the concepts and has
experience working with these designs.
Alfred W Rademaker, Ph.D.
Director, Biostatistics Collaboration Center (BCC)
Professor in Preventive Medicine
Dr. Joseph Kang:
"No one can possibly object
that randomization will remove biases due to residual
confounding. Indeed randomization is the gold-standard
method in causal inference for any clinical studies.
Propensity score methods aim to turn an observational data
set into a pseudo-randomized data set where subjects across
different intervention arms can be balanced with respect to
measured confounders (such as risk factors at baseline).
Numerous epidemiologists and biostatisticians (currently
12,482 publications only in NCBI
) have used propensity scores in various studies. They also
developed sensitivity analysis methods for the residual
confounding issue as well.
Statistical schema for Propensity scoring provided by Dr.
-Joseph Kang, Ph.D.
Northwestern University Feinberg School of Medicine
Dr. Rick Chappell:
"Dr. Kang’s document is a useful approach, but I would add
the warning that propensity scores or other adjustments
cannot substitute for randomized clinical trials. There will
always be the possibility of bias due to some factor which
is associated with the patients' choice and with the outcome
(a confounder). If this confounder isn't properly
incorporated into the propensity score for example, if it
isn't measured), then the apparent treatment effect will be
biased. However, an patient-selected or other nonrandomized
control is still better than no control at all.
It seems to me that the only true measure is to compare your
results with an RCT's. So the ORCT could be a preliminary
study subject to confirmation from a RCT. Of course, there
are always subjective considerations as to why subjects
might pick one treatment or another but I wouldn't know how
to quantify these.
Rick Chappell, Ph.D.
Professor, Depts. of Statistics and of Biostatistics &
Univ. of Wisconsin Medical School
In closing, we acknowledge
the statistical axiom that the RCT is the gold standard method
to objectively compare treatment protocols. However, it
is often not feasible to enroll patients in trials that assign
patients to treatment arms by chance. Patients have shown, by
refusing participation, that they will not always allow their treatment
to be decided by computer or coin flip.
We recognize that the
proposed ORCT design will not be appropriate in all circumstances. The
ORCT is a tool to
consider when clinical equipoise is deficient and therefore
accrual is not feasible for a RCT. In such cases
we expect that the ORCT will be more reliable as evidence than a large phase II trial using
The ORCT can be more
appropriate than an RCT when (1) either protocol in the
study can be used off study, (2) when the patient's preferences or
risk factors make either study arm unappealing or inappropriate, or (3) when the
compared study arms have very different approaches or
notes: With the rapid
advancement of cancer molecular and treatment knowledge,
“personalized medicine” has made the RCT less desirable, and
probably even unethical. Targeting of specific cancer patients
who are likely to respond to therapy makes a traditional
randomized trial nearly impossible to accomplish. Consider the unfairness to a
patient whose cancer molecular markers have identified him/her
to be more likely to respond to a new drug, when RCT
randomization has placed that patient in an arm which will not
get this targeted agent.
incorporates random assignment to study arms ... when it's an
acceptable choice to the patient and referring physician.
The more acceptable that choice the closer the ORCT becomes to the gold standard
method of patient assignment. The ORCT
will encourage participation in the trial because it allows patients to choose their therapy when a clinical
risk factor makes one approach more appropriate, or when they
have strong preference or aversion to one of the compared
We submit that
the ORCT is clearly superior to any RCT which is never
because it’s judged to be unfeasible … or to any RCT that is
terminated because of poor enrollment. We hope and expect that
the ORCT provides another way to do good science while practicing good
comment and guidance - in particular from statisticians in this
Comment or Question?
Thank you for
listening and for your input in advance.
Jim Omel, MD.
Karl Schwartz, MFA
Patient research advocates
Gregory A. Curt, M.D., Senior Editor, The Oncologist and
Bruce A. Chabner, M.D., Editor-in-Chief, The Oncologist,
One in Five Cancer Clinical Trials Is Published: A Terrible
Symptom—What's the Diagnosis?
Eur Heart J. 2012:
(technical - describes
a research method to decrease bias in non-randomized
Do observational studies using propensity score methods
agree with randomized trials?
A systematic comparison of studies on acute coronary
For this indication,
"observational studies using PS methods produce treatment
effect estimates that are of more extreme magnitude compared
with those from RCTs, although the differences are rarely
Rev Esp Cardiol.
for Creating Covariate Balance in Observational Studies
technical paper that describes a method that could help to
make The Observation-enriched Randomized Controlled Trial (ORCT) more reliable -
in cases where random assignment to the study arms is not
feasible because of strong patient preferences or ethical.