(Circulation. 2007;116:1596-1601.)
© 2007 American Heart Association, Inc.
Controversies in Cardiovascular Medicine |
From Harvard Medical School, Division of Vascular Surgery, and Beth Israel Deaconess Medical Center, Boston, Mass.
Correspondence to Frank W. LoGerfo, MD, William V. McDermott Distinguished Professor of Surgery, Harvard Medical School, Division of Vascular Surgery, Beth Israel Deaconess Medical Center, 110 Francis St, Suite 5B, Boston, MA 02215. E-mail flogerfo{at}bidmc.harvard.edu
| Introduction |
|---|
|
|
|---|
Response by Samuelson et al p 1601
As it now stands, existing studies leave us with the unfortunate but not unreasonable conclusion that no scientific basis exists for the use of CASs as approved by the FDA, and in the absence of change, there is every reason to doubt the capability of our current system to protect the public from unnecessary risk in the future. Although this article focuses on just 1 example of how our systems are flawed at multiple levels to provide a reliable assessment of CASs and other technology, readers seeking further examples can find a wealth of related information.3–5
| The SAPPHIRE Trial |
|---|
|
|
|---|
The original intent of the study was to randomize 600 to 900 patients with a maximum sample size of 2900. The study was set up with certain statistical objectives, including the method of statistical analysis, the timing of interim analyses, the conditions for termination, and a final test to determine the primary end point. As the study unfolded, however, all of these statistical protocols were violated. The trial was terminated because of a decrease in enrollment; the interim analyses were not performed; and an alternative statistical method was used to determine the end point. After a lengthy description of the original study design, the authors state, "In early 2002, the pace of enrollment in the trial abruptly slowed, because several nonrandomized CAS registries had become available." One such registry was their own, and they chose, as permitted in the SAPPHIRE design, to assign more of the study-qualified patients to their stent registry than to the randomized study. For these investigators, an incentive or requirement no longer existed to determine whether CAS was effective as soon as they had the opportunity to use CAS without such evidence. As an unfortunate consequence, their study was underpowered to answer the question it posed as to whether CAS was inferior to CEA. Despite this shortcoming, the study was published and formed the primary basis for FDA approval.
The primary end point of the trial was the cumulative incidence of death, stroke, or myocardial infarction within 30 days of the procedure or death or ipsilateral stroke between 31 days and 1 year. No significant statistical difference existed with regard to the primary end point at 30 days. At the end of 1 year, the authors, using a statistical analysis that was an alternative to the study design, reported a significant difference in the primary composite end point of 12.2% for CAS and 20.1% for CEA (P<0.05). This difference, in favor of CAS, is highly attributable to the differing incidence of non–Q-wave myocardial infarctions, as determined by a 2-fold elevation in creatinine kinase with a positive MB fraction. No other criteria were required for the diagnosis of myocardial infarction. For this reason, the significance of the diagnosis cannot be compared with other studies of perioperative myocardial infarction, usually using the World Health Organization criteria.6,7 The inclusion of non–Q-wave infarction as an end point equivalent to death or stroke is highly controversial. The authors justified this decision by stating that a perioperative non–Q-wave infarction confirms an increase in the risk of myocardial infarction by a factor of 27 in the subsequent 6 months. That may be true; however, for some reason, that concern was not validated by their own data in this study because no subsequent Q-wave infarctions occurred in the CEA group between 31 days and 1 year despite the higher incidence of "enzyme" infarctions. Another reason for the higher incidence of enzyme infarctions may be that the assays were done more often in the CEA group than in the CAS group because patients were studied every 24 hours while in the hospital and the CEA group on average was in the hospital 1 day longer than the CAS group. Unless the number and timing of enzyme studies were identical between the 2 groups, the data are not valid. Thus, for a number of reasons, the significance of myocardial infarction and the differences between groups represent data that cannot be relied on for any conclusion. The key issues here are stroke and death. However, the study was underpowered to determine noninferiority in outcomes under the conditions defined in the protocol, as was eventually noted by the FDA.
Missing from the SAPPHIRE data is any reference to local or systemic complications of stent placement and administration of intense antiplatelet therapy. Is it possible that no such complications occurred? To put this question in perspective, at least 1 similar registry of CASs exists in which poststent cerebral hemorrhage in a patient on antiplatelet therapy is not counted as a stroke but as "other neurological."8 This may or not have been the case in the SAPPHIRE study, but to have no anticoagulation-related local or systemic problems in a study this large and involving major arterial stent placement via percutaneous femoral artery puncture is distinctly unusual and at least warrants some comment or verification. Also missing from SAPPHIRE is any effort to identify "silent" cerebral ischemia, an outcome at least as relevant as silent myocardial ischemia. Subsequent studies have shown silent infarcts in
40% of patients even when a protective filter is used.9,10 These are hard-copy, irrefutable data that warrant attention. Are these infarcts truly silent? Should we look for cognitive or emotional effects? Does embolization continue after the procedure?
The SAPPHIRE study design also was unique among carotid studies in that no tracking of outcomes based on degree of stenosis took place. What was it, and was it equal in the randomized arms? If the randomization were truly applied to all eligible patients, it would be extremely unlikely that 20% of the patients in the randomized trial had recurrent stenosis as an indication. At the primary institution, the Cleveland Clinic, the reported volume of CEA between 1989 and 1995 was 371 per year.11 During the overlapping interval, 1989 to 1999, the volume of redo CEA was 20 per year, or 5.4%,12 a number that is consistent with other high-volume centers.13 Credibility is strained to say that this fraction is now 20%, and this discrepancy adds fuel to the argument that the opinion-based entry criteria are scientifically invalid. The reason is that recurrent stenosis is a smooth, fibrotic lesion that many believe is highly suited to CAS, with less risk of embolization during the procedure. A study that includes a large number of restenotic lesions may be favorable to CAS. Thus, we are saddled here with the appearance of bias. That bias would have been eliminated with protocol-based entry criteria.
These data were presented to the FDA at a meeting of the Circulatory System Devices Advisory Panel on April 21, 2004.14 The response of the FDA staff reviewers warrants praise and is encouraging. Heng Li, PhD, the FDA statistical reviewer, summarized the randomized trial statistical analysis. The recorded minutes of Dr Lis findings stated, "Dr. Li provided a detailed explanation of the planned statistical methodology and the stopping rule it incorporated. In his analysis, the evidence would not have indicated that the trial should have been stopped (and non-inferiority declared), if the original protocol had been followed."
The minutes continued: "The sponsor made unplanned comparisons between the stent registry and the CEA arm of the randomized study. Because the patient characteristics in the two groups by definition are different, a straightforward comparison is not appropriate. To address this issue, the sponsor used a propensity score method to compare the two groups, attempting to make a post-hoc claim of non-inferiority of the stent registry to the randomized CEA arm." In other words, because the data from the biased-but-randomized arm were insufficient to make the case for CAS, data from the nonrandomized arm were pulled in to an alternative statistical analysis.
Dr Li continued, "In summary, the original group sequential protocol was not followed, and the FDA was not informed of any change in protocol. Any non-inferiority claim based on the sponsors post-hoc propensity score analysis is problematic."
Ronald Weintraub, MD, FDA consultant, reviewed the methodology of the SAPPHIRE pivotal study. He stated, "The sponsors study findings are limited because the pre-specified enrollment plan and study analysis was not carried to completion in the SAPPHIRE randomized study."
Lisa Cannel, FDA lead reviewer, commented, "The sponsor terminated the pivotal trial early, citing too many competing studies, physicians reluctance to randomize, and surgeons unwillingness to refer patients. The competing studies involved CORDISs own devices and were facilitated by CORDIS." Thus, the sponsor (CORDIS) effectively undermined its own study and placed the responsibility on its own investigators.
Despite these concerns, including the misgivings of the FDA statistician, the FDA panel of clinician experts voted 6 to 5 in favor of CASs as a legitimate alternative to CEA in both symptomatic and asymptomatic patients who fall within a broad definition of high risk. With the shortcomings of the study and the complete absence of study design or data comparing symptomatic and asymptomatic patients as identified by the FDA staff, how did this all come about? How does a study with these shortcomings warrant publication in a major medical journal, and how, in spite of this, did the FDA panel approve the device? Such a challenge to logic and science requires that we examine the FDA approval process.
At the FDA, the data were reviewed by an advisory panel consisting of 11 voting members: 6 cardiologists, 2 interventional radiologists, 2 vascular surgeons, and 1 neurologist. All of these people submitted disclosures. Six of the voting members were acknowledged to have current or past interests in firms at issue. These interests were waived because they were in matters not related to the specific agenda. In other words, these panelists had existing interests in the corporate sponsor, but those interests were with products or devices other than CASs. Others have examined the complexities that arise under these circumstances.3,4 It is notable that the FDA itself, under the Prescription Drug User Free Act, is funded by the corporate sponsor. Others have posed this question: Who then is their client, the corporation or the public?15,16
The lead author of the report was an inventor of the Angioguard embolic protection device used in the SAPPHIRE Trial and was a founder of the Angioguard Corp, purchased by Johnson & Johnson in 1999, for $40 million.17 In fact, of the 15 authors of the SAPPHIRE Trial, 10 acknowledged support in one form or another from the CORDIS Corp; in addition, 2 of the authors were employees of the CORDIS Corp. This level of disclosure of relationships with the corporate sponsor was unusually high but does not prove that any effective influence was exerted on the authors. The disclosure standard set by the National Institutes of Health and the Association of Medical Colleges was met.18 The underlying philosophy here seems to be that once the disclosure standard is met, readers can form their own opinions. Unfortunately, however, readers cannot expunge from the literature studies such as SAPPHIRE that do not follow scientific standards and, once published, can be used to create public policy. In this case, the clinical science would have to be of the highest standard to overcome the burden of competing interests; in this case, as will be shown, the science was far from that.
The specific role of the lead author is a little different. The recommendations of a recent roundtable on this very subject, including editors of the American Heart Journal and The Lancet,19 concluded: "The inventor must not be involved with the clinical studies and must be excluded from enrolling patients, analyzing the data, and writing the manuscript." The publisher of SAPPHIRE, however, was not responsible for approval of the inventor as the principal investigator or lead author. That, in fact, was primarily the responsibility of the authors institution and its internal review processes. Nonetheless, the publisher has the admittedly complex task of examining all aspects of a clinical study to ensure the quality of information through which (in this case) public policy was influenced. Why dwell on these concerns about compromised science leading to public policy? Because once a product is unleashed in the medical marketplace, it is extremely difficult to reverse that decision. An entire industry, along with jobs and political influence, has developed around carotid stenting. Before FDA approval, not much was at stake, but now a monetary force has been created. Once created, the industry interest can be protected and maintained by a moving target of slightly new devices, slightly different technology, and new target populations, all of which make it extremely difficult to force the removal of a product from the marketplace.
| Other Trials |
|---|
|
|
|---|
Two recently published European trials favor CEA and raise serious questions about the safety of CAS. The Endarterectomy Versus Angioplasty in Patients with Symptomatic Carotid Stenosis trial was, in fact, a protocol-based randomized comparison of CAS with CEA, an essential scientific criterion not met by either of the above-mentioned trials. The trial was stopped prematurely by its safety committee for reasons of safety and futility. The rate of any stroke or death was 9.6% for CAS compared with 3.9% for CEA, with a relative risk of 2.5 (95% CI, 1.2 to 5.1; P<0.01). The flaws in this study included the limited carotid stenting experience requirements of the interventionalists. In addition, embolic protection devices were not used in the early phase of the trial. Nonetheless, the experimental design and conduct of this study were at a scientific standard well above the US studies. The Stent-Supported Percutaneous Angioplasty of the Carotid Artery Versus Endarterectomy (SPACE) trial20 was a multinational, multicenter, randomized controlled trial of CAS versus CEA with an outcome of any stroke or death of 7.68% versus 6.51% and a relative risk of 1.19 (95% confidence interval, 0.75 to 1.92). These 30-day outcome results were interpreted to fail to confirm noninferiority of CAS; longer-term results are pending. The study was stopped prematurely for funding reasons, and many would fairly argue that, at this point, it is inconclusive. However, it is also reasonable to ask whether approval would have been granted if data from these trials had been presented to the FDA.
| Conclusions |
|---|
|
|
|---|
The bottom line here is that we need well-conducted, scientifically designed randomized trials to get answers about CASs. SAPPHIRE represents a failed opportunity. The only existing randomized trial in this country is the Carotid Revascularization Endarterectomy Versus Stent Trial (CREST),26 a National Institutes of Health–sponsored trial that began long before SAPPHIRE but is moving comparatively slowly now that the FDA has approved CAS and CAS registries. Regrettably, we have an organ of government, the National Institutes of Health, that does not have support from another, the FDA. It is, in essence, science versus commerce and a major flaw in our system. If the FDA had refused to allow any stent registries until CREST was completed, we would now have an extensive body of valid information of which we could be proud and on which safe public policy could be based. Instead, we have settled for unscientific studies designed to win expeditious FDA approval. As a result, CAS was approved for use in symptomatic patients, a subgroup for which we have virtually no data, and asymptomatic patients, a subgroup for which we have flawed data. This can be described as the commercialization of science. It is not what the public deserves from its faith in clinical scientists, its respect for prestigious journals and institutions, and its dependence on regulatory agencies.
In the end, no one can make a strong argument for CASs. To the contrary, the most scientifically valid data, the European studies, are unfavorable to CAS. Unfortunately, CASs, having FDA approval, are already in widespread use despite conflicting data about their safety. Going forward, physicians, editors, institutional review boards, governmental agencies, and readers must be more vigilant and critical of any commercialization of clinical research. The validity of our debate can be no better than the validity of the available data.
It should be noted that the biotechnology industry does play an essential and vital role in the advancement of medical care. As eager as industry might be to see a new proprietary device arrive successfully in the marketplace, no business entity can ultimately benefit from inaccurate information. Clinical scientists funded by industry must walk a difficult ethical line and maintain scientific rigor in the face of countervailing pressures. This balancing act protects both parties. In the case of the rush to FDA approval of CASs, the flawed data and the appearance of impropriety, whether it occurred or not, are such that a disinterested party would not agree that we have data on which public safety can depend. If we adhere to the scientific method and are committed to evidence-based medicine, we should cease debate until we complete a scientifically valid study such as CREST.
In the end, it may be that CASs are as effective as or more effective than CEA or noninterventional medical therapy. It is our role as clinician-investigators to fully use the resources available to us to design and implement sound clinical studies. Until we have done that, we have no basis for supporting the current use of CASs.
| Acknowledgments |
|---|
None.
| References |
|---|
|
|
|---|
"Lies, damn lies, and statistics...." In todays medical literature, it is possible to find "data" to support almost any position. In Endarterectomy Versus Angioplasty in Patients With Severe Symptomatic Carotid Stenosis (EVA-3S) and Stenting and Angioplasty With Protection in Patients at High Risk for Endarterectomy (SAPPHIRE), we have conflicting studies with significant flaws (not unlike most randomized studies). The unassailable fact is that carotid endarterectomy (CEA) is a good operation. However, many years experience and a body of literature demonstrate that many patients are at increased risk for CEA, as exemplified by the patients excluded from the North American Symptomatic Carotid Endarterectomy Trial (NASCET) and the Asymptomatic Carotid Atherosclerosis Study (ACAS) (identified as high-risk patients on the basis of the trial designers surgical experience) and the objective performance criteria (predicted risk of CEA for patients in high-risk carotid artery stenting [CAS] registries) developed by the Food and Drug Administration after rigorous literature review. As a surgeon who has performed CEA procedures over the last 30 years, who participated in NASCET, and who also has performed CAS procedures since 1994 (while participating in most Food and Drug Administration–sponsored trials), I find that nothing is more frustrating than facing a patient in whom the known risk for CEA is excessive and for whom no reimbursable alternative exists, a not uncommon scenario. In 2007, equipoise exists between CEA and CAS. The Cardiac Revascularization Endarterectomy Versus Stent Trial (CREST), like all the other CAS trials, will teach us much about CAS relative to CEA but will not eliminate one procedure or the other. Today, the important question is not "Which is better, CEA or CAS?" but rather "Which procedure is better for a given patient?" and "Are the risk factors excessive for revascularization (with either technique)?"—suggesting a role for best medical therapy.
| Footnotes |
|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Circulation Home | Subscriptions | Archives | Feedback | Authors | Help | AHA Journals Home | Search Copyright © 2007 American Heart Association, Inc. All rights reserved. Unauthorized use prohibited. |