Level I Evidence: Great Boon or Misleading Waste?

By Frank J. Veith, MD

Level I evidence for a particular medical practice or procedure is based on appropriately conducted prospective randomized comparative clinical trials in which one treatment regimen or procedure, often new, is compared with an established standard. Level I evidence is considered by many clinical scientists to be the strongest support for changing clinical practice.

Some purists believe that such practice changes should not occur without level I justification. Yet others are quick to point out the potential flaws in such randomized trials and, on this basis, claim they are unnecessary.

Nowhere is this controversy more apparent than with the introduction of new endovascular procedures that can replace well-established standard open operations such as carotid endarterectomy and abdominal aortic aneurysm repair. The intensity of this controversy is fueled not only by legitimate differences of opinion but also by turf issues, because different specialties may claim the rights to perform the endovascular replacements for open procedures.

Because this intense controversy exists, it is worthwhile to explore the question: Is level I evidence a boon to medical practice or a misleading waste? The short answer is that level I evidence can be both. To understand the reasons why, one must appreciate the potential flaws and weaknesses that can diminish the value of level I evidence and look at some examples.

First is the pace and state of development of the newer technology being evaluated. If the technology is advancing so fast that what is used currently is superior to what was used in the trial, poor level I results with the new treatment no longer apply. This was certainly the case in many of the early carotid stenting trials, and these trial results are no longer applicable.

Another similar flaw that can diminish the value of level I evidence is important progress in the treatment afforded the control group. This flaw is now exemplified by the excellent and often quoted NASCET trial, which clearly showed the benefits of endarterectomy over best medical treatment circa 1990 for symptomatic high-grade carotid stenosis.

One can argue that 2006 best medical treatment with statins and better platelet inhibitors might produce better outcomes in a current control group--maybe even equal to those of endarterectomy. Thus, the applicability of even the finest level I evidence is time limited and lessened by medical progress in either the invasive therapy or the control arm treatment or both.

New, more current level I evidence must be gathered to have valid applicability to current medical practice. In carotid therapy, there is a clear need for new level I evidence trials that include current best medical therapy arms.

A third weakness limiting the value of level I evidence relates to the selection of patients for randomization. An example is the SAPPHIRE trial, which compared carotid stenting with embolic protection to carotid endarterectomy in so-called high-risk patients. This trial demonstrated the equivalence of stenting to endarterectomy in this population, although the adverse event and death rates in both arms of the study were unexpectedly high.

The value of the study was lessened because 70% of the patients were asymptomatic. This raises the question: Might these patients have fared better with best medical treatment? Yet this study was widely cited in the lay media as evidence that carotid stenting was better than endarterectomy, a totally unjustified conclusion.

A fourth flaw relates to the competence of the individuals performing the procedures, in either the new technology or the control old-treatment arms. An example is the recent French trial (EVA-3S) comparing stenting with endarterectomy in symptomatic carotid stenosis patients. This study showed significant superiority of endarterectomy.

However, the experience level of those performing the stenting has been questioned. Was it this or was it the symptomatic lesions that led to the inferiority of stenting? This question remains unanswered. The fact that the recent German (SPACE) trial in similar symptomatic patients showed no clear outcome differences between stenting and endarterectomy does not help to answer the question. However, the German trial had an adverse outcomes rate in stented patients similar to the French trial.

The outcome difference in the two trials was due to a difference in the adverse event rates in the endarterectomy patients, and this remains unexplained. The results of these two excellent trials are provocative but point up the imperfections of even well-conducted trials that seek level I evidence and definitive conclusions.

A number of other considerations raise further questions about the value and applicability of level I evidence. These include the adequacy of the randomization process. Too many exclusions in one arm might indicate investigator or patient biases, which are realities of medicine and life in free societies. Level I data are hard to gather, and good level I data are even harder to get, even by those rare unbiased individuals genuinely seeking unvarnished truth.

There is also the problem of applicability of results to real world medicine. If results are obtained by highly experienced superstar specialists in leading medical centers, can they be extrapolated to everyday medical practice?

Perhaps the best level I evidence should be gathered in medical institutions and practices of all sizes and types. Clearly, this is difficult or impossible to do.

Finally, there are in some prospective randomized trials idiosyncratic flaws that render their results nonapplicable in other circumstances or locales. One example is the well-conducted EVAR II trial from the United Kingdom. This trial compared endovascular abdominal aortic aneurysm repair (EVAR) with best medical management in high-risk patients with aneurysms more than 5.5 cm in diameter and suitable anatomy for EVAR. No difference in outcomes was observed in the two groups.

However, there was a delay averaging 57 days between randomization and EVAR. Nine of the 20 deaths in the EVAR patients occurred from aneurysm rupture during this delay. If this delay had not occurred and EVAR could have been carried out with less than the 9% procedural mortality observed in the trial, EVAR is likely to have been beneficial in these high-risk patients.

Thus, the conclusions of EVAR II might not apply in other countries where delay between diagnosis and EVAR could be minimized and in other circumstances in which lower procedural mortality could be achieved, as it was in other reports of EVAR in apparently comparable high-risk patients.

In light of all these flaws and weaknesses, is there any value to level I evidence?

Of course there is. Solid level I evidence can answer medical questions that cannot be resolved in any other way. Such evidence can be the strongest possible basis for good medical practice. However, level I evidence is neither perfect nor a panacea. Its flaws and limitations must be kept constantly in mind to prevent overinterpretation and overextrapolation of such evidence and to be sure it is valid and applicable to the circumstances at hand. Level I evidence per se is not a substitute for good judgment by individual physicians and the medical profession as a whole.

This judgment must be exercised in the interpretation of randomized prospective controlled studies, in the application of such studies to individual patients, and in the supplemental use of other considerations and classes of evidence in determining the best treatment for a given patient in a given medical setting. Level I evidence can be extremely helpful, but it should not be the Holy Grail of medical decision making.