I met Eli Estey on a trip to MD Anderson in Houston. I found him to be a charming and thoughtful guy. He has now moved to the 'Hutch' in Seattle and has penned an interesting article entitled "Do commonly used clinical trial designs reflect clinical reality?" in the journal Haematologica.
We normally rely on randomized phase III clinical trials which compare two treatments using endpoints like overall survival, progression-free survival or time to next treatment. We rely heavily on p<0.05 which means that there is less than a 1 in 20 chance that any difference between the two could have occurred accidentally. In other words we are pretty sure that any positive result is not a false positive. In leukemia trials there is a tradition, which comes from St Jude's Memphis, that we inch forwards in our improvements. Childhood ALL is curable in the majority of cases. Treatment involves the use of several drugs in a particular order over a couple of years. Successive trials made small changes to the protocol that gave small improvements in survival. In order to be sure that these small improvements were not false positives the trials needed to utilize large numbers of patients so that a 'p' value of <0.05 could be obtained. Childhood ALL trials started with drugs that gave a high rate of complete remissions (vincristine and prednisone) so there was a good chance that adding more chemotherapy would improve things, but not by much. Hence, it was necessary to be certain that adding these new drugs did make a difference and were not false positives.
With adults, the problem is not that we are doing pretty well already, but that we don't cure very many at all. Therefore, our problem is not that we have too many false positives; it is that there may be drugs out there that help, but we reject them because we have too many false negatives.
The cure rate for CLL and adult AML is so poor that we are not really interested in drugs that make a small difference - we want agents that make a big difference - we should be willing to accept a higher risk of a false positive.
When trials are designed the numbers needed in each arm are partially determined by the power calculation. Traditionally an 80% power is required. What does this mean? It is actually the false negative rate. It means that one trial in five is discarding a treatment incorrectly because the negative answer was a false one.
Estey argues that this way of doing things is not clinically relevant. On the contrary there have been quite massive improvements in adult leukemia that did not come from these sorts of trials at all. He cites the value of imatinib in CML, CDA in hairy cell leukemia, arsenic trioxide and ATRA in acute promyelocytic leukemia and high-dose ara-C for core-binding factor AML as examples where the effect was so large that it didn't need an enormous trial to detect its value.
I raise this problem for CLL recently in this way. John Byrd recently reported that there are 107 different agents being studied at teh preclinical and clinical level for teh treatment of CLL. The typical phase III trial in CLL requires 800+ patients. There are simply not enough patients to test all those agent singly let alone in combination. WE must find a better way of doing things.
The better way that Estey proposes is the use of Bayesian statistics. Thomas Bayes (c. 1702 – 17 April 1761) was a British mathematician and Presbyterian minister, known for having formulated a specific case of the theorem that bears his name: Bayes' theorem, which was published posthumously. I have read the paper several times but I am none the wiser. Would someone please explain Baysian statistics to me.
8 comments:
Terry...I am of no help in this regard. Statistics as taught to my medical school class was a waste of our time (as well as that of our teacher) and I have suffered from this deficiency throughout my career, depending upon the wisdom (and blessings) from professors such as you.
Hopefully a mathematician/statistician can help us all. We definitely need wothwhile studies which are fairly set up and fairly judged.
DWCLL
Doc,
Honestly, how often is it in a patient's best interests to get into a trial?
If things look really bad for someone, it seems to me that it would be worth it to chance some new treatment. But otherwise, why should he use anything other than whatever treatment is the gold standard?
That's an easy one to answer. Suppose the gold standard treatment gave a remission rate of 10% - that was the case with AML min 1966. Two new drugs made an appearance then; daunorubicin and cytarabine. The only way to get them was in a clinical trial, but no-one knew that they were any good until there was a clinical trial. The trouble with clinical trials is that long before the trial was completed people knew that the new drugs were better than the old. Thr trial was necessary to show how much better. Somehow Bayesian statistics allows you to circumvent these long, costly and wasteful trials. But I haven't figured out the reasoning behindthem yet.
A very rough explanation of Bayesian statistics....
Bayesian statistics uses conditional probability, which is quite confusing even for mathematicians. So as conditions change, the data analysis changes with conditions. Specifically, Bayesians use early results to refine or amend their analysis - the early results change the conditions, which in turn changes the conditional probabilities.
It sounds like a good idea to use first results to refine the hypothesis, but non-Bayesians object that this is changing the rules after the game starts.
Yes, I think I understand that, but could you give a concrete example and answer why non-Bayesians think it is unfair to change the rules after the game as started? Because it isn't a game. Can't it be seen as starting a new game because the old rules didn't fit the situation?
I've taken a number of statistical classes for my degree, but these were basic and intermediate level stat classes.
I will tell you that most people are simply mystified by statistics. They begin to understand when you talk of probabilities in terms of gambling. However, most people would say that after flipping an honest coin 20 times, and getting heads each time, the chances are almost certainly going to be heads the 21st time. Of course, each event is independent, and the chances are even the next toss will be heads.
For a website that sort of agrees with you, see http://www.stat.columbia.edu/~cook/movabletype/archives/2008/04/problems_with_b.html
Non-Bayesians argue that classical statistical methods sufficiently handle changes due to early results.
The real problem in statistics is to handle variability. The simplest statistical situation is when there are only two possible results - zero/one, off/on, yes/no, go/no go. As the number of possible results increases, statistical variance increases exponentially. The classical (non-Bayesian) approach is then to break up the statistical tests into simpler questions which have simpler answers. So clinical trials of drug tests are broken up into increasingly complex phases - the simplest and most easily answered question is at the beginning phase 1. Statistically, variances and thus sample sizes and costs increase with each phase.
Classical statistics thus appeals to common sense. People find it easier to understand a complex issue when the questions are broken down into simplest steps first. And of course accountants accept the idea of starting small and only increasing test costs if early results are promising.
As for not changing the rules of the game or starting a new game with different rules, mathematicians call these situations by the generic term "game" even when the results are deadly serious.
Classical statisticians argue that they are willing to change the rules as long as there is another rule as to how rules will be changed. Similarly, classical statistician argue that starting a new game needs another rule as to when to do it. The absence of a rule about changing the rules or starting over under new rules is often derided as "subjectivity" in Bayesian statistics.
I think most classical statisticians accept the idea of a new game when a patient is diagnosed with acute cancer and short survival. In this situation, there is little concern with chronic drug side effects over a long time. But this is more of a financial than a statistical issue because there is little funding for this - including funding to cover costs of potential lawsuits.
Anon.
That was Professor Gelman's April Fool's joke about Bayesian statistics!
Post a Comment