Dr. Donald Berry, the head of biostatistics at the M. D. Anderson Cancer Center in Houston, has a Ph.D. in his field and long years of expertise in designing and interpreting results of clinical trials.
Dr. David Freedman, a statistics professor at the University of California at Berkeley, has similar credentials. Yet, when they examine data from one of the most widely cited studies of mammography, they come to different conclusions. And therein lies the conundrum: how can experts look at the same data and disagree over their meaning?
The research, known as the Health Insurance Plan Study, or HIP, is one of the first and largest mammography studies. Begun in New York in the 1960’s, it involved more than 60,000 women randomly assigned to have mammograms or not and followed for 18 years. The study found that the breast cancer death rate was about 30 percent higher in women who did not have mammograms. Of about 30,000 who were not screened, 196 died of breast cancer compared with 153 of about 30,000 who had mammograms.
Dr. Berry and Dr. Freedman agree that the study had a great strength: it was done when mammograms were not in general use, so women who were randomly assigned to forgo the screening were unlikely to have it on their own. In later years, when mammograms were easily available, researchers struggled with the problem of women assigned not to be screened who had mammograms on their own anyway. The two also agree that the study had a potential flaw. The researchers began by randomly assigning women to have mammograms or not. But they also decided that they did not want to include women who already had breast cancer. So after the women were assigned, they dropped women who, they later realized from looking at medical records, had had cancer. About 1,100 ended up being dropped — some 800 from the mammography group and about 300 from the control group.
Critics of the study wonder why so many more women in the screening group turned out to have had a diagnosis of breast cancer before the study began. In theory, they say, the numbers should have been roughly equal. As a result, they wonder if some women who had already had breast cancer were wrongly left in the control group.
Dr. Freedman said there was a reason for the imbalance: Sam Shapiro, the study director, had better data on the women in the screened group than in the control group, since he was following the screened women closely, with mammograms and office visits. So he was more likely to notice if they had already had breast cancer. But does this flaw cast serious doubts on the conclusions? Here, the two experts disagree.
Dr. Berry says the trial’s conclusion rests on a difference of 43 deaths from breast cancer after 18 years. And, he says, there were 500 more women excluded from the screened group than from the control group. If just 10 percent of those 500 women died of breast cancer, and if they had remained in the study group where they were originally assigned, that would have been an additional 50 breast cancer deaths. That result, Dr. Berry said, would more than eliminate the positive effect that the study found from screening. For that reason, he said, he is not confident in the study’s conclusion that mammography led to a lower breast cancer death rate. Dr. Freedman said he was confident in the data because of the study’s setup. One reason, he said, is that Mr. Shapiro made a correction to try to make the two groups of women equivalent. If breast cancer was diagnosed, the researchers would examine the patient’s records and if they discovered that she had had breast cancer before the study began, they dropped her from the study.
In the end, Dr. Freedman said, about equal numbers of women in the two groups who were kept in the study developed breast cancer. That, he says, tells him that the two groups were not so different. Dr. Berry said he was not at all comforted by Dr. Freedman’s arguments. “Any bias associated with the women who had been excluded at the start of the study cannot be repaired retrospectively by guessing whether someone had cancer at the time of randomization,” he said. In fact, he added, it can make a bias worse. “Only the screening group had mammograms,” Dr. Berry said. “On second look at a woman’s first mammogram, one might find that breast cancer was present at the time but it had been missed,” he said. So more women might have been excluded from the mammography group after they developed breast cancer.
The two experts agree that there is no way to resolve their disagreement. “It’s amazingly complicated,” Dr. Freedman said.
Dr. Berry said the only way that he could feel comfortable with the data would be if someone could find the women who were excluded from the trial after they were assigned to have mammograms or not, discover how many died of breast cancer, and then ask if those data altered the conclusions. That, Dr. Freedman and Dr. Berry concur, would not be an easy task, and no one, so far, has volunteered to undertake it.
Dr. Freedman and Dr. Berry also disagree about the interpretation of a more recent study, from Canada, that found no benefit from mammograms. Dr. Freedman said the study had some serious problems, and he has less confidence in its conclusions. Dr. Berry says the flaws are minor and the conclusions are credible. Still, neither Dr. Berry nor Dr. Freedman convinced the woman in his life. The two women are over 50 and so of an age when mammography guidelines call for annual screening.
Dr. Berry’s wife has annual mammograms, although she is well aware of her husband’s doubts about their worth. Dr. Freedman’s woman friend has them only every three years, despite Dr. Freedman’s urging that she have them annually.
From the NYT