Without evidence of benefit, an intervention should not be presumed to be beneficial or safe.

- Rogue Medic

How Not to Respond to Negative Research

ResearchBlogging.org

Also posted over at Paramedicine 101 (now at EMS Blogs) and at Research Blogging. Go check out the rest of the excellent material at these sites. There is a new research podcast specifically for EMS – EMS Research Podcast. On episode 2 we discuss several topics, including the research on the RAD-57 non-invasive monitor.

Continuing from The RAD-57 Pulse Co-Oximeter – Does It Work – Part I and from The RAD-57 Pulse Co-Oximeter – Does It Work – Part II. More on the use of the RAD-57.

To the Editor:

Masimo Corporation is the manufacturer of the Rad-57, a multiwavelength pulse carbon monoxide oximeter that measures noninvasive carboxyhemoglobin (SpCO) in the blood. The Rad-57 is the subject of a study by Touger et al1 and an accompanying editorial2 in this edition of Annals. Masimo appreciates the journal’s willingness to publish these comments about the study and editorial.[1]

So far, so good.

Since the introduction of the Rad-57 in 2005, Masimo has received countless reports from clinicians that the device has enabled them to save lives and limit the damaging effects of carbon monoxide poisoning.[1]

Only the second paragraph and Dr. O’Reilly is already wandering into the street corner sales pitch. This is what the alternative medicine charlatans always seem to say.

Countless? Why is it that none of the people making these anecdotal claims are able to count, but they expect us to trust them with our lives?

Given this accuracy specification, approximately 95% of SpCO measurements are expected to be within 2 SDs of the COHb value. If this specification were applied to limits of agreement calculation, the accuracy could be stated as 5.9% to +5.9%. The study by Touger et al1 stated: “We determined a priori that a difference of ±5% carboxyhemoglobin would be considered clinically significant.” No rationale for the ±5% threshold was provided, but we do not believe it is appropriate to initiate a study with expectations greater than the stated performance of a device or drug. In short, there was a strong likelihood the study would result in a negative conclusion by the investigators before it even started.[1]

Accuse the ones performing the study of approaching things with a bias.

So, what did the study state about the reasons for their range?

Finally, the selection of ±5% carboxyhemoglobin as a boundary for acceptable limits of agreement was based on presumed clinical significance but may be considered somewhat arbitrary. The actual limits of agreement (–11.6% to 14.4% carboxyhemoglobin) demonstrated in our study substantially exceeded this value, suggesting that the inference would have been the same even if a larger value, eg, ±10% carboxyhemoglobin, had been chosen.[2]

Dr. O’Reilly is complaining that ±5% is narrower than the ±5.9% that he states should be used as the limit of 2 standard deviations. Dr. Touger points out that ±5.9% would not make the RAD-57 look good. Even ±10% wouldn’t make the Rad-57 look good.

The study results are significantly different from those of other available studies, as well as from Masimo’s internal test data from subjects with 1% to 40% COHb levels. However, the study results by Touger et al1 are discussed as being representative of device performance, and strong conclusions are made by both the authors and the editors according to the study results.[1]

This study was of real emergency department patients (at a burn center with a hyperbaric chamber) being evaluated for possible CO (Carbon monOxide) poisoning. As far as being different from other available studies, that is not true.

Here is another study, which does not encourage faith in the RAD-57 readings.

A total of 36.4% of the patients transported during the study had SpCO documented. Of the 1,017 adults included in this group, 11 (1.1%) had an SpCO >15%.[3]

1% of the patients included in the study had RAD-57 levels above 15%.

Of the 11 patients with a SpCO >15%, 10 were transported to a hospital for which the investigators had institutional review board (IRB) approval to review the patient’ s medical record. Of those 10, none had confirmatory venous carboxyhemoglobin levels. The two patients with an SpCO level of 21% did have a repeat SpCO documented at triage upon arrival to the emergency department. Their repeat levels were 8% and 2%. None of the 10 patients with levels >15% ultimately were diagnosed with and treated for carbon monoxide exposure or toxicity.[3]

The medical records were available for 91% of that 1%.

None of the 10 patients with levels >15% ultimately were diagnosed with and treated for carbon monoxide exposure or toxicity.

But, none of them had blood drawn to check carboxyhemoglobin levels.

Were they misdiagnosed?

More important. Did the researchers let the hospitals know that they were studying the ability of the RAD-57 to identify carboxyhemoglobin?

From the study, the answer appears to be, No.

Do we know if the RAD-57 missed any patients with elevated carboxyhemoglobin?

No.

This study does not appear to have been designed to identify patients missed by the RAD-57.

Data obtained from other studies indicate that there are false positive results using the new technology, but the benefit of identifying true positive CO toxic patients outweighs the burden of false positive results.3 In this cohort, there were at least two patients who had discordant results when a repeat SpCO was obtained, further emphasizing that all positive results obtained with the non-invasive method should be confirmed with a blood carboxyhemoglobin level.[3]

The RAD-57 readings were inconsistent. The patients with high readings were not diagnosed with, or treated for carbon monoxide toxicity. This does not support Dr. O’Reilly’s claims that the RAD-57 is accurate and just being unfairly evaluated.

The RAD-57 may have some role in identifying elevated carboxyhemoglobin levels, but so far nobody can tell what that role is. As I pointed out earlier

Less than half of the patients with elevated COHb were correctly identified.

If we screen a fire fighter for COHb, then we need to keep that fire fighter out of the fire.

Should anyone ever use a low RAD-57 reading to justify returning a fire fighter to a fire?

No.

Masimo stands by its products’ performance and knows that when SpCO-enabled devices are used according to their directions for use, they provide accurate SpCO measurements that provide significant clinical utility, helping clinicians detect carbon monoxide poisoning in patients otherwise not suspected of having it and rule out carbon monoxide poisoning in patients with suspected carbon monoxide poisoning.[1]

That advice from Dr. O’Reilly may encourage us to return fire fighters to an environment that has already made them toxic, but with the mistaken belief that they have carboxyhemoglobin levels of zero, when their carboxyhemoglobin is really very high.

Dr. O’Reilly’s advice is bad for Masimo investors.

Dr. O’Reilly’s advice is bad for patients.

Dr. O’Reilly’s advice misrepresents the research.

Find a way to make the RAD-57 reliable, then sell a lot of them. Right now, it isn’t reliable.

Right now, the RAD-57 is an accident waiting to happen.

Continued in How Not to Respond to Negative Research – Addendum and in How TO Respond to Negative Research.

Footnotes:

[1] Performance of the Rad-57 pulse co-oximeter compared with standard laboratory carboxyhemoglobin measurement.
O’Reilly M.
Ann Emerg Med. 2010 Oct;56(4):442-4; author reply 444-5. No abstract available.
PMID: 20868919 [PubMed – indexed for MEDLINE]

Free Full Text of letter and author reply from Ann Emerg Med with links to Free Full Text PDF download

[2] Performance of the RAD-57 pulse CO-oximeter compared with standard laboratory carboxyhemoglobin measurement.
Touger M, Birnbaum A, Wang J, Chou K, Pearson D, Bijur P.
Ann Emerg Med. 2010 Oct;56(4):382-8. Epub 2010 Jun 3.
PMID: 20605259 [PubMed – indexed for MEDLINE]

Free Full Text Article from Ann Emerg Med with links to Free Full Text PDF download

[3] Non-invasive carboxyhemoglobin monitoring: screening emergency medical services patients for carbon monoxide exposure.
Nilson D, Partridge R, Suner S, Jay G.
Prehosp Disaster Med. 2010 May-Jun;25(3):253-6.
PMID: 20586019 [PubMed – indexed for MEDLINE]

O’Reilly M (2010). Performance of the Rad-57 pulse co-oximeter compared with standard laboratory carboxyhemoglobin measurement. Annals of emergency medicine, 56 (4) PMID: 20868919

Touger, M., Birnbaum, A., Wang, J., Chou, K., Pearson, D., & Bijur, P. (2010). Performance of the RAD-57 Pulse Co-Oximeter Compared With Standard Laboratory Carboxyhemoglobin Measurement Annals of Emergency Medicine, 56 (4), 382-388 DOI: 10.1016/j.annemergmed.2010.03.041

Nilson D, Partridge R, Suner S, & Jay G (2010). Non-invasive carboxyhemoglobin monitoring: screening emergency medical services patients for carbon monoxide exposure. Prehospital and disaster medicine : the official journal of the National Association of EMS Physicians and the World Association for Emergency and Disaster Medicine in association with the Acute Care Foundation, 25 (3), 253-6 PMID: 20586019

.

The RAD-57 Pulse Co-Oximeter – Does It Work – Part II

ResearchBlogging.org

Also posted over at Paramedicine 101 (now at EMS Blogs) and at Research Blogging. Go check out the rest of the excellent material at these sites. There is a new research podcast specifically for EMS – EMS Research Podcast. On episode 2 we discuss several topics, including the research on the RAD-57 non-invasive monitor.

Continuing from The RAD-57 Pulse Co-Oximeter – Does It Work – Part I. More on the use of the RAD-57 non-invasive monitor.

The ad says, Quick and Noninvasive Assessment of CO Levels in the Blood.

I do not disagree with those claims. The problem is not speed.

I don’t mind taking more time, if it leads to greater accuracy.

This study does not encourage confidence in the accuracy of the assessment provided by the RAD-57.

Median laboratory carboxyhemoglobin level was 2.3% (interquartile range 1% to 8.5%), with a range of 0% to 38%. Median RAD carboxyhemoglobin level was 3.0% (interquartile range 0% to 7.5%).[1]

Even with properly trained people and repeating the test, the RAD-57 does not appear to be accurate. Most of the laboratory readings were near zero, so guessing zero for everyone would have been a pretty good guess. A Pet Rock™ can even more consistently provide the same zero reading.

Type of exposure, . . . . . . . . . . . . . . . . . . . . . . . No. (%)
Smoke inhalation . . . . . . . . . . . . . . . . . . . . . . . 68 (57)
Improperly vented fuel burning device. . . . . . 47 (39)
Unknown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 (4)[1]

Most of the people assessed were in the ED (Emergency Department) for smoke inhalation.

One of the uses for this is to assess fire fighters – look at the Masimo ad.

Should anyone ever use a low RAD-57 reading to justify returning a fire fighter to a fire?

No.

If that fire fighter has elevated COHb, that probably occurred while fighting the fire. Even if we screen every fire fighter before the fire, a zero reading on the RAD-57 does not mean that a fire fighter does not already have elevated COHb.

Sending a fire fighter, with a not yet detected elevated COHb, back into the fire is probably only sending that fire fighter back into the same environment that produced the not yet detected elevated COHb.

This is not the way to make good things happen.

A laboratory value of 15% carboxyhemoglobin was selected as the cutoff for determination of these performance characteristics according to proposed use of this RAD level for out-of-hospital triage of patients with possible carbon monoxide exposure in New York City.13[1]

The RAD device correctly identified 11 of 23 patients with laboratory values greater than or equal to 15% carboxyhemoglobin (sensitivity 48%; 95% CI 27% to 69%).[1]

Less than half of the patients with elevated COHb were correctly identified.

If we screen a fire fighter for COHb, then we need to keep that fire fighter out of the fire.

If these results are typical of what could be expected on a fire scene?

This is a big if – but still a much safer if than ignoring these results.

Out of 23 fire fighters with greater than 15% COHb, 12 might be told to go back in and to breathe in more CO.

I am not a lawyer, but I detect some significant legal fees in Masimo’s future, unless they make some things very clear.

The RAD-57 is not appropriate for the screening of large asymptomatic populations, except to come up with a reason to encourage those with elevated COHb readings on the RAD-57 to go to the hospital. Extrapolating from the results of this study, this will only identify about half of those with elevated COHb, so what do we do for the other half with elevated COHb?

The RAD-57 should not be thrown out, but the RAD-57 should never be used to justify discouraging someone from going to the hospital.

The RAD-57 should not be thrown out, but the RAD-57 should never be used to justify sending a fire fighter back into a fire.

Does the RAD-57 work?

Maybe, but this study does not inspire confidence in the RAD-57.

We should not ignore this study.

Continued in How Not to Respond to Negative Research and in How Not to Respond to Negative Research – Addendum.

Footnotes:

[1] Performance of the RAD-57 pulse CO-oximeter compared with standard laboratory carboxyhemoglobin measurement.
Touger M, Birnbaum A, Wang J, Chou K, Pearson D, Bijur P.
Ann Emerg Med. 2010 Oct;56(4):382-8. Epub 2010 Jun 3.
PMID: 20605259 [PubMed – indexed for MEDLINE]

Free Full Text Article from Ann Emerg Med with links to Free Full Text PDF download

Touger, M., Birnbaum, A., Wang, J., Chou, K., Pearson, D., & Bijur, P. (2010). Performance of the RAD-57 Pulse Co-Oximeter Compared With Standard Laboratory Carboxyhemoglobin Measurement Annals of Emergency Medicine, 56 (4), 382-388 DOI: 10.1016/j.annemergmed.2010.03.041

.

Lies, Damned Lies, and Medical Science – Part I

Also posted over at the new podcast specifically for EMS research – EMS Research Podcast. On episode 2 we discuss several topics, including the article Lies, Damned Lies, and Medical Science, by David H. Freedman. This article suggests that medical research is always wrong. Since we were discussing this on a podcast about research, I would have expected much more interest in this critical topic.

Since this is an article about the research done by Dr. John Ioannidis, does the article accurately represent the research of Dr. Ioannidis?

First, let’s look at the evidence provided in the article and the way it is presented.

Just as I was getting the sense that the data in drug studies were endlessly malleable, Ioannidis, who had mostly been listening, delivered what felt like a coup de grâce: wasn’t it possible, he asked, that drug companies were carefully selecting the topics of their studies—for example, comparing their new drugs against those already known to be inferior to others on the market—so that they were ahead of the game even before the data juggling began? “Maybe sometimes it’s the questions that are biased, not the answers,” he said, flashing a friendly smile. Everyone nodded. Though the results of drug studies often make newspaper headlines, you have to wonder whether they prove anything at all. Indeed, given the breadth of the potential problems raised at the meeting, can any medical-research studies be trusted?[1]

There are not many direct quotes in the article, rather there are many interpretations of what Dr. Ioannidis says. This is one of the problems with the article. The article purports to be about the problems with the objectivity of medical research, but the author of the article is constantly inserting his own opinion where we should be reading the words of Dr. Ioannidis.

“Though the results of drug studies often make newspaper headlines, you have to wonder whether they prove anything at all.” – David H. Freedman.

A newspaper headline has nothing to do with science. Newspaper headlines are the interpretation of isolated research by non-scientists for an audience of non-scientists. There is plenty of excellent coverage of the problems with this kind of bad journalism at Gary Schwitzer’s HealthNewsReview Blog.

Do journalists do a good job of reporting science?

No.

Does that mean that medical science is a Lie?

Does that mean that medical science is a Damned Lie?

If we are buying what Dr. Mr. Freedman is selling, then medical science is not just a lie, but a damned lie. After all, Dr. Mr. Freedman is a journalist presenting medical research in a way designed to sell the most copies and make a lot of money to get the most people to read his story.

The reporting of medical science press releases from non-scientists publicizing the research done by scientists is not a significant part of the problem. Only the science is the problem – at least, if we are buying what Dr. Mr. Freedman is selling.

“Indeed, given the breadth of the potential problems raised at the meeting, can any medical-research studies be trusted?” – David H. Freedman.

According to Dr. Mr. Freedman?

Or

According to Dr. Ioannidis?

As I intend to point out further, these two answers are not within six sigma of each other in the same ballpark.

His work has been widely accepted by the medical community; it has been published in the field’s top journals, where it is heavily cited; and he is a big draw at conferences. Given this exposure, and the fact that his work broadly targets everyone else’s work in medicine, as well as everything that physicians do and all the health advice we get, Ioannidis may be one of the most influential scientists alive. Yet for all his influence, he worries that the field of medical research is so pervasively flawed, and so riddled with conflicts of interest, that it might be chronically resistant to change—or even to publicly admitting that there’s a problem.[1]

Scientists publicly admitting that there are problems?

Journalists writing about scientists pointing out the limitations of the research that they publish?

Or

Journalists promoting the new cure for cancer in humans every time something is shown to correlate with less growth of cancer in a single mouse.

The job of a journalist is to ask the tough questions.

The job of a journalist is to understand the material being presented.

The job of a journalist is not to republish a press release from a drug lab, sometimes just cutting and pasting the entire press release into the newspaper, magazine, or web page.

Still, we often get nothing more than press releases forwarded by the news media.

Is this a failure of science?

To be continued in Lies, Damned Lies, and Medical Science – Part II and later to be continued in Lies, Damned Lies, and Medical Science – Part III.

Footnotes:

[1] Lies, Damned Lies, and Medical Science
The Atlantic
by David H. Freedman
Article

.

The RAD-57 Pulse Co-Oximeter – Does It Work – Part I

ResearchBlogging.org

Also posted over at Paramedicine 101 (now at EMS Blogs) and at Research Blogging. Go check out the rest of the excellent material at these sites. There is a new research podcast specifically for EMS – EMS Research Podcast. On episode 2 we discuss several topics, including the research on the RAD-57 non-invasive monitor.

Non-invasive monitoring for CO (Carbon Monoxide) toxicity seems to be the hot topic right now. Annals of Emergency Medicine has made its main paper, from October, on the RAD-57 non-invasive monitor free for everyone. The Doctor’s Channel has a brief video on the RAD-57 paper. The Poison Review has a brief print review of the RAD-57 paper. JEMS (Journal of EMS) has an entire supplement devoted to CO toxicity. On the EMS Research Podcast, we debate this for a while.

So, does the Masimo RAD-57 non-invasive monitor work?

First, is there a need for some kind of non-invasive monitor?

Symptoms of carbon monoxide poisoning are nonspecific and include headache, fatigue, malaise, confusion, nausea, dizziness, visual disturbances, chest pain, shortness of breath, loss of consciousness, and seizures. Although carboxyhemoglobin levels are inconsistently related to the degree of toxicity, in the absence of a clear history, detection in the blood may be the only means of confirming suspected exposure.1[1]

Many of these can be described as flu-like symptoms. Another problem is hypochondria. When people are told that they have been exposed to something, they frequently imagine symptoms. At the other end of the spectrum, we have fire fighters wanting to go back in to fight a fire and denying that they have any symptoms. with such a diverse mix of presentations, it would be nice to have something more portable and less invasive than a hospital laboratory.

If the RAD-57 can do this, that would be a great thing for fire departments, for EMS, and for patients.

Demonstration of adequate agreement between measurements made with the RAD device and standard laboratory measurement is essential before this noninvasive method of carbon monoxide detection can appropriately be substituted for arterial or venous measurement in clinical care.[1]

Essential?

Is there any other method of assessing the validity of the non-invasive measurements of CO levels?

Not yet.

Measurement was obtained with the RAD by clipping the appropriate (adult or pediatric) probe onto the digit for a period of approximately 15 seconds and reading the carboxyhemoglobin value from the handheld device screen. Immediately after this measurement, the device was removed from the finger, replaced on the same finger, and allowed to recalibrate, and a repeated carboxyhemoglobin level was recorded from the device. Arterial or venous blood was obtained with the first RAD measurement and sent to the hospital laboratory in a heparinized syringe for direct measurement of whole blood carboxyhemoglobin, using co-oximetry (Siemens Rapidlab 1200 blood gas analyzer). Clinicians were asked to record whether the RAD measurement was technically difficult and whether the low signal quality indicator on the device was illuminated for each subject.[1]

That seems reasonable.

Let’s skip to the results.

A nice chart, but what does it mean?

The numbers on the left are the RAD-57 readings, while the numbers on the bottom are the laboratory readings. The readings should be the same. The laboratory readings are considered to be accurate. The difference between the laboratory value, or real COHb (carboxyhemoglobin – CO in the blood), and the RAD-57 value is the way inaccuracy is determined. The gray diagonal line in the middle should be about where all of the results end up, if the readings agree (if the RAD-57 is accurate).

For example, 5% from the RAD-57 (draw a horizontal line from 5 on the left) and 5% from the laboratory (draw a vertical line from 5 on the bottom), we would put a dot on the line where the two lines meet. If we take away all the lines and leave the dots, we have this chart of all 120 patient readings.

Two more examples are taken from Table 3. Laboratory and RAD COHb values(laboratory value ≥15% COHb and RAD value <15% COHb). I use the first two.

Zero% from the RAD-57 (draw a horizontal line from 0 on the left) and 19.8% from the laboratory (draw a vertical line from 19.8 [round off to 20] on the bottom), we would put a dot on the line where the two lines meet.

Zero% from the RAD-57 (draw a horizontal line from 0 on the left) and 35.2% from the laboratory (draw a vertical line from 35.2 [round off to 35] on the bottom), we would put a dot on the line where the two lines meet.

Since very few of the dots are going to be exactly on the line, we include a range of acceptable variance. I changed the gray line to a thick red line and put a thin red line on each side, parallel to the thick red line. This allows the number to be higher or lower by 5, for a total range of 10 around the laboratory reading.

This really does not make clear where the problems are likely to be. If the RAD-57 gives a false high reading and we take somebody to the hospital, that is an inconvenience, but it is not likely to lead to a bad medical outcome. The problems are when the RAD-57 gives false low readings that encourage EMS to tell the patients that there is no problem. If these patients do not receive treatment, there may be problems. The chart below shows that area. I drew a line to indicate that the upper left corner that would be included in the allowable variance range.

Even more potentially dangerous may be the patients with RAD-57 readings that are below 5, while their laboratory readings are more than 10. Do we really think that some EMS personnel will not look at these low numbers and tell patients that they have nothing to worry about?

Then there is the problem of all of the zero readings on the RAD-57. Everybody in this study was being evaluated for suspected carbon monoxide poisoning. There does not appear to have been any zero laboratory reading when the RAD-57 had a non-zero reading. There are a bunch of very low laboratory readings, but the only zero laboratory reading(s) coincide with a RAD-57 zero reading(s).

The RAD-57 zero readings are in the Pet Rock™ Area. I could run some wires and a fancy probe to a Pet Rock™. I would obtain the same result the RAD-57 did for all of these patients, who had laboratory readings of up to 35.2.

Lab   =   35.2

RAD-57   =   0

Pet Rock™   =   0

That’s not even close enough for government work.

The Pet Rock™ doesn’t need batteries or maintenance.

Is the RAD-57 that difficult to use properly?

Or

Is the RAD-57 that inaccurate?

Or

Is the RAD-57 both inaccurate and difficult to use?

Or

Something else?

Those are a lot of zeros that should not be zeros.

Continued in The RAD-57 Pulse Co-Oximeter – Does It Work – Part II and in How Not to Respond to Negative Research as well as in How Not to Respond to Negative Research – Addendum.

Footnotes:

[1] Performance of the RAD-57 pulse CO-oximeter compared with standard laboratory carboxyhemoglobin measurement.
Touger M, Birnbaum A, Wang J, Chou K, Pearson D, Bijur P.
Ann Emerg Med. 2010 Oct;56(4):382-8. Epub 2010 Jun 3.
PMID: 20605259 [PubMed – indexed for MEDLINE]

Free Full Text Article from Ann Emerg Med with links to Free Full Text PDF download

Touger, M., Birnbaum, A., Wang, J., Chou, K., Pearson, D., & Bijur, P. (2010). Performance of the RAD-57 Pulse Co-Oximeter Compared With Standard Laboratory Carboxyhemoglobin Measurement Annals of Emergency Medicine, 56 (4), 382-388 DOI: 10.1016/j.annemergmed.2010.03.041

.