.jpg)
–
Also posted over at Paramedicine 101 (now at EMS Blogs) and at Research Blogging. Go check out the rest of the excellent material at these sites. There is a new research podcast specifically for EMS – EMS Research Podcast. On episode 2 we discuss several topics, including the research on the RAD-57 non-invasive monitor.
–
Continuing from The RAD-57 Pulse Co-Oximeter – Does It Work – Part I and from The RAD-57 Pulse Co-Oximeter – Does It Work – Part II. More on the use of the RAD-57.
–
To the Editor:
Masimo Corporation is the manufacturer of the Rad-57, a multiwavelength pulse carbon monoxide oximeter that measures noninvasive carboxyhemoglobin (SpCO) in the blood. The Rad-57 is the subject of a study by Touger et al1 and an accompanying editorial2 in this edition of Annals. Masimo appreciates the journal’s willingness to publish these comments about the study and editorial.[1]
So far, so good.
Since the introduction of the Rad-57 in 2005, Masimo has received countless reports from clinicians that the device has enabled them to save lives and limit the damaging effects of carbon monoxide poisoning.[1]
Only the second paragraph and Dr. O’Reilly is already wandering into the street corner sales pitch. This is what the alternative medicine charlatans always seem to say.
Countless? Why is it that none of the people making these anecdotal claims are able to count, but they expect us to trust them with our lives?
Given this accuracy specification, approximately 95% of SpCO measurements are expected to be within 2 SDs of the COHb value. If this specification were applied to limits of agreement calculation, the accuracy could be stated as 5.9% to +5.9%. The study by Touger et al1 stated: “We determined a priori that a difference of ±5% carboxyhemoglobin would be considered clinically significant.” No rationale for the ±5% threshold was provided, but we do not believe it is appropriate to initiate a study with expectations greater than the stated performance of a device or drug. In short, there was a strong likelihood the study would result in a negative conclusion by the investigators before it even started.[1]
Accuse the ones performing the study of approaching things with a bias.
So, what did the study state about the reasons for their range?
Finally, the selection of ±5% carboxyhemoglobin as a boundary for acceptable limits of agreement was based on presumed clinical significance but may be considered somewhat arbitrary. The actual limits of agreement (–11.6% to 14.4% carboxyhemoglobin) demonstrated in our study substantially exceeded this value, suggesting that the inference would have been the same even if a larger value, eg, ±10% carboxyhemoglobin, had been chosen.[2]
Dr. O’Reilly is complaining that ±5% is narrower than the ±5.9% that he states should be used as the limit of 2 standard deviations. Dr. Touger points out that ±5.9% would not make the RAD-57 look good. Even ±10% wouldn’t make the Rad-57 look good.
The study results are significantly different from those of other available studies, as well as from Masimo’s internal test data from subjects with 1% to 40% COHb levels. However, the study results by Touger et al1 are discussed as being representative of device performance, and strong conclusions are made by both the authors and the editors according to the study results.[1]
This study was of real emergency department patients (at a burn center with a hyperbaric chamber) being evaluated for possible CO (Carbon monOxide) poisoning. As far as being different from other available studies, that is not true.
Here is another study, which does not encourage faith in the RAD-57 readings.
A total of 36.4% of the patients transported during the study had SpCO documented. Of the 1,017 adults included in this group, 11 (1.1%) had an SpCO >15%.[3]
1% of the patients included in the study had RAD-57 levels above 15%.
Of the 11 patients with a SpCO >15%, 10 were transported to a hospital for which the investigators had institutional review board (IRB) approval to review the patient’ s medical record. Of those 10, none had confirmatory venous carboxyhemoglobin levels. The two patients with an SpCO level of 21% did have a repeat SpCO documented at triage upon arrival to the emergency department. Their repeat levels were 8% and 2%. None of the 10 patients with levels >15% ultimately were diagnosed with and treated for carbon monoxide exposure or toxicity.[3]
The medical records were available for 91% of that 1%.
None of the 10 patients with levels >15% ultimately were diagnosed with and treated for carbon monoxide exposure or toxicity.
But, none of them had blood drawn to check carboxyhemoglobin levels.
Were they misdiagnosed?
More important. Did the researchers let the hospitals know that they were studying the ability of the RAD-57 to identify carboxyhemoglobin?
From the study, the answer appears to be, No.
Do we know if the RAD-57 missed any patients with elevated carboxyhemoglobin?
No.
This study does not appear to have been designed to identify patients missed by the RAD-57.
Data obtained from other studies indicate that there are false positive results using the new technology, but the benefit of identifying true positive CO toxic patients outweighs the burden of false positive results.3 In this cohort, there were at least two patients who had discordant results when a repeat SpCO was obtained, further emphasizing that all positive results obtained with the non-invasive method should be confirmed with a blood carboxyhemoglobin level.[3]
The RAD-57 readings were inconsistent. The patients with high readings were not diagnosed with, or treated for carbon monoxide toxicity. This does not support Dr. O’Reilly’s claims that the RAD-57 is accurate and just being unfairly evaluated.
The RAD-57 may have some role in identifying elevated carboxyhemoglobin levels, but so far nobody can tell what that role is. As I pointed out earlier –
Less than half of the patients with elevated COHb were correctly identified.
If we screen a fire fighter for COHb, then we need to keep that fire fighter out of the fire.
Should anyone ever use a low RAD-57 reading to justify returning a fire fighter to a fire?
No.
Masimo stands by its products’ performance and knows that when SpCO-enabled devices are used according to their directions for use, they provide accurate SpCO measurements that provide significant clinical utility, helping clinicians detect carbon monoxide poisoning in patients otherwise not suspected of having it and rule out carbon monoxide poisoning in patients with suspected carbon monoxide poisoning.[1]
That advice from Dr. O’Reilly may encourage us to return fire fighters to an environment that has already made them toxic, but with the mistaken belief that they have carboxyhemoglobin levels of zero, when their carboxyhemoglobin is really very high.
Dr. O’Reilly’s advice is bad for Masimo investors.
Dr. O’Reilly’s advice is bad for patients.
Dr. O’Reilly’s advice misrepresents the research.
Find a way to make the RAD-57 reliable, then sell a lot of them. Right now, it isn’t reliable.
Right now, the RAD-57 is an accident waiting to happen.
–
Continued in How Not to Respond to Negative Research – Addendum and in How TO Respond to Negative Research.
–
–
Footnotes:
–
[1] Performance of the Rad-57 pulse co-oximeter compared with standard laboratory carboxyhemoglobin measurement.
O’Reilly M.
Ann Emerg Med. 2010 Oct;56(4):442-4; author reply 444-5. No abstract available.
PMID: 20868919 [PubMed – indexed for MEDLINE]
–
[2] Performance of the RAD-57 pulse CO-oximeter compared with standard laboratory carboxyhemoglobin measurement.
Touger M, Birnbaum A, Wang J, Chou K, Pearson D, Bijur P.
Ann Emerg Med. 2010 Oct;56(4):382-8. Epub 2010 Jun 3.
PMID: 20605259 [PubMed – indexed for MEDLINE]
Free Full Text Article from Ann Emerg Med with links to Free Full Text PDF download
–
[3] Non-invasive carboxyhemoglobin monitoring: screening emergency medical services patients for carbon monoxide exposure.
Nilson D, Partridge R, Suner S, Jay G.
Prehosp Disaster Med. 2010 May-Jun;25(3):253-6.
PMID: 20586019 [PubMed – indexed for MEDLINE]
–
O’Reilly M (2010). Performance of the Rad-57 pulse co-oximeter compared with standard laboratory carboxyhemoglobin measurement. Annals of emergency medicine, 56 (4) PMID: 20868919
–
Touger, M., Birnbaum, A., Wang, J., Chou, K., Pearson, D., & Bijur, P. (2010). Performance of the RAD-57 Pulse Co-Oximeter Compared With Standard Laboratory Carboxyhemoglobin Measurement Annals of Emergency Medicine, 56 (4), 382-388 DOI: 10.1016/j.annemergmed.2010.03.041
–
Nilson D, Partridge R, Suner S, & Jay G (2010). Non-invasive carboxyhemoglobin monitoring: screening emergency medical services patients for carbon monoxide exposure. Prehospital and disaster medicine : the official journal of the National Association of EMS Physicians and the World Association for Emergency and Disaster Medicine in association with the Acute Care Foundation, 25 (3), 253-6 PMID: 20586019
.
Thanks RM,
This was an excellent series. I have always been one to look at the detailed facts rather than hearsay with regard to making purchase decisions. When these devices first hit the market there was a stampede in many areas to get these toys on the rigs. I did what research I could but found no definitive clinical studies except what Masimo was offering, which I took as suspect simply from a marketing standpoint (why would they show me a study that contained negative results?).
I also consulted with a rehab specialist in our area who sits on several NFPA committees, is a PA specializing in Emergency Services, and is a published author and researcher . He told me that ‘something is better than nothing’ and it (the RAD-57) was a good tool which will get better over time. I noted that this person’s web publications on several pre-eminent Fire and EMS magazines are plastered with Masimo advertisements. (No implications, I’m just saying.)
Due to a lack of evidence and budget constraints, I held off on making any recommendation. I do not request equipment unless I am convinced it will do what is promised consistently.
Lacking the personal time to read, digest, and evaluate the research, I very much appreciate the way you’ve laid it out for us, complete with references. This is a valuable public service.
Over the past several years, I have lamented not having a RAD-57 every time we do a CO alarm in a residence with confirmed CO levels. Most of these are false alarms due to defective detectors. Just last night I had a job where the entire house was filled with fumes from a kerosene heater. The patient had been sleeping for 4 hours in that environment and complained of a crushing headache with 10/10 pain. He had vented the house for 1/2 hour before calling us. We found the house clean, but the patient was clearly not well. I documented everything at the scene and in my turnover report gave specific details of the environment. The physician concurred that they would do a COHg test via a venous blood draw to be sure one way or the other. Sure would like to know how it turns out, whether it was just the fumes, or if it was CO, but I’ll probably never know. None the less, I feel like the patient was better off with the assessment he received than he would have if I had a device telling me he had 0% SPCO.
AT this point, I have no problem waiting for the RAD-57 to be better prepared for prime time. Thanks again for a wonderfully useful series.
UU
I suspect that the problem is in the factory calibration of the device (the algorithms for calculating COHb from spectral information) which I think derives from a mistaken idea of what is important to determine. I suspect the factory calibration is being driven by marketing considerations, i.e. because the marketing people want to put “accurate to X%” on their glossy marketing brochures, rather than listen to those involved in treating patients as to what information they actually need to treat those patients.
The only purpose of measurements like this (and for any diagnostic test) is differential diagnosis which is only useful for differential treatment. A test does not need resolution better than the resolution necessary to provide differential treatment.
What is important to determine in a patient with COHb is not a low percentage deviation from the “true” amount of COHb, but rather what is the absolute amount of COHb within a certain range.
If the “true” COHb level is 20%, getting a reading of 30% or 15% isn’t going to change the treatment, O2 and getting to a HBO2 facility (my guess, but IANAD).
I think the “problem” is that the device is tuned to give the most precise readings in the low COHb range where precision isn’t important because low COHb readings are the most common. Then when those measurements are compared and averaged, the precision at the low end dominates the average, but the precision at the low end isn’t at all important. What is important is the rare go/no go values at the high end.
I don’t know anything about the details of how the device works, other than it uses some kind of spectral stuff, but it is inconceivable to me that the technique couldn’t be made to technically work, and technically work just by changing the software. I think the problem there is that because the device has to be FDA approved, changing the software turns it into a new device that needs new FDA approval even if it works better. I suspect that business considerations are preventing that from happening.
daedalus2u,
My response is at How TO Respond to Negative Research.
In this crazy world of online advertising, immediately after your damning post is… an ad for the Masimo Rad-57!! “Quick and Noninvasive Assessment of CO Levels in the Blood” it reads, and features a photo of a Fireman and a Medic. The Fireman has had a rough day; he’s seated, no helmet, with the Rad-57 clip attached to his finger. But you know in just a moment the Medic is going to tell him he’s A-OK and send him back to work – because the Rad-57 says he’s great! (Maybe…maybe not!)
If you were using the BS-2-HI Detector on that ad, it could very well blow up in your hand.
Steve,
My response is at How TO Respond to Negative Research.