The RAD-57 Pulse Co-Oximeter – Does It Work – Part II

Whitepaper
Background
Exposure to carbon monoxide (CO) raises levels of circulating carboxyhemoglobin (COHb) in the blood. High levels of COHb indicate CO poisoning and require intervention to prevent further damage or death. Traditional testing for COHb involves invasive sampling and laboratory analysis. Up to 50% of hospitals do not have on-site laboratory COHb testing ability1 and Emergency Medical Services personnel cannot measure laboratory COHb levels in the field. Each year, it is estimated that thousands of CO poisoning cases are missed because of a lack of CO testing in emergency departments alone.2

The Masimo Rad-57 device measures noninvasive COHb (SpCO) levels in the blood using multiwavelength spectrophotometry. Since 2005, the Rad-57 has been used inside and outside of the hospital to help clinicians throughout the world assess CO poisoning.

The FDA 510(k)-cleared accuracy specification of the SpCO measurement is ±3% at one standard deviation in a range of 1 to 40%, compared to simultaneous invasive blood draws and laboratory analysis for COHb.3 This means that approximately two-thirds of SpCO measurements are expected to be within one standard deviation of the COHb value, and approximately 95% of SpCO measurements are expected to be within 6% (two standard deviations) of the COHb value.

Accuracy of Noninvasive Carboxyhemoglobin Measurements from the Rad-57: Analysis of a Recent Study by Touger et al

Summary
The Rad-57™ is a safe and effective device for noninvasive carboxyhemoglobin (SpCO®) measurements. However, a recent study of 120 subjects by Touger et al questions the accuracy of SpCO measurements from the Rad-57 device as compared to laboratory carboxyhemoglobin (COHb) levels. The results of the Touger study are significantly different than other available studies of 1,690 subjects and are also significantly different than Masimo’s internal testing of 3,629 measurements. In Masimo’s opinion, based on internal testing of 3,629 measurements, it is not possible for a functioning Rad-57 device and sensor to produce these results when the directions for use are followed and the measurements are compared to simultaneous COHb levels. There are multiple potential reasons for the results reported in the Touger study, including device malfunction, sensor malfunction, finger positioning in the sensor, timing of laboratory COHb measurements, reporting of zero from the device memory when the device actually was unable to measure and displayed dashes, patient motion, and/or external light interference. Over 8,000 Rad-57 devices with SpCO are in use by clinicians worldwide and Masimo has received complaints from less than 1% of customers since the product was introduced. Since the introduction of the Rad-57, Masimo has received countless reports from clinicians that the device has enabled them to save lives and limit the damaging effects of CO poisoning. If clinicians question SpCO readings from the Rad-57 device, they should repeat the test and make sure they follow the directions for use. If they further question the results of the device, they should obtain an invasive blood sample and perform laboratory analysis. Masimo strongly encourages any clinician who questions the accuracy of SpCO measurements from the Rad-57 device to perform their own statistically significant evaluation per the directions for use, and compare the readings to a simultaneous invasive blood sample, analyzed on a laboratory CO-oximeter for COHb.

Rad-57 SpCO Accuracy
Sources of variation in any measurement exist, even between reference methods, and comparing results from multiple devices increases the expected variability of the comparisons.4 Gehring et al performed a study measuring COHb using the same blood sample analyzed on two identical devices from 5 different CO-oximeter manufacturers, and showed up to a 0.8% standard deviation in COHb measurements in the same model device.5 At two standard deviations, this represents a -1.6% to +1.6% limit of agreement with a laboratory COHb device – compared to itself.

When describing SpCO accuracy compared to laboratory analysis of COHb, some studies use bias and standard deviation and other studies use “limits of agreement.” Limits of agreement are a similar concept to expressing accuracy with standard deviations and can be interpreted to mean that when comparing SpCO to invasive carboxyhemoglobin (COHb), about 95% of SpCO measurements would fall within the limits of agreement.

The exact limits of agreement range is calculated as the bias ±1.96 times the standard deviation. For example:
>> At a bias of 0% and standard deviation of 3%, the limits of agreement are -5.9% to + 5.9%, meaning 95% of SpCO measurements were between 5.9% below COHb to 5.9% above COHb.
>> At a bias of +3% and standard deviation of 3%, the limits of agreement are -2.8% to +8.9%, meaning 95% of SpCO measurements were between 2.8% below COHb to 8.9% above COHb.

It is also important to note that up to 5% of patients are expected to fall outside the limits of agreement. In these cases the difference between SpCO and COHb could be more than 5.9%

Overview of Touger Study
The accuracy of SpCO measurements from the Rad-57 are the subject of the study6 and accompanying editorial7 in the October 2010 edition of Annals of Emergency Medicine. The study included 120 subjects and was conducted in the emergency department at Jacobi Medical Center in the Bronx, New York, USA. The sensors that were used in the study were Rev B SpCO reusable sensors. Masimo is currently shipping Rev E SpCO reusable sensors.

Invasive COHb measurements were compared to SpCO measurements. COHb ranged from 0 to 38%; 23 patients were characterized by levels >15%. The mean difference (bias) between COHb and SpCO values was 1.4%. The limits of agreement of measurement differences between SpCO and COHb were -11.6% to +14.4%. SpCO was >15% when COHb was >15% in 11 of 23 patients (reported sensitivity 48%).

A scatter plot of reported comparisons is shown in Figure 1.
COHb (%)
Touger Study Scatterplot
SpCO (%)
20
40
30
20
10
0
0 10 30 40
Figure 1. Touger study scatterplot of Rad-57 SpCO vs. laboratory COHb. The outliers circled in red are significantly different than other available studies of 1,690 subjects and are also significantly different than Masimo’s internal testing of 3,629 measurements.

Whitepaper

The authors also separated subjects with COHb values over 15% and showed the reported Rad-57 SpCO measurements in those cases (Figure 2).

Laboratory COHb (%)
Touger Study Table
SpCO (%)
19.8
35.2
20.5
16.9
16.2
18.3
16.4
22.9
15.1
15.9
18.8
18
0
0
13
14
6
9
14
13
8
3
11
0

In Masimo’s opinion, based on internal testing of 3,629 measurements, it is not possible for a functioning Rad-57 device and sensor to produce these results when the directions for use are followed and the measurements are compared to simultaneous COHb levels

These SpCO measurements were elevated and would cause clinician suspicion but since they were not >15%, they were not included in the calculated sensitivity

Figure 2. Touger study table purported to show inaccuracy of SpCO levels with COHb levels >15%.

The authors reported sensitivity and specificity using a hard cut point of 15% COHb. Outside of the extreme outliers that indicate a malfunctioning device or sensor or not following the directions for use, this type of sensitivity calculation is not clinically relevant to clinicians using SpCO to aid assessment. This is because cases in which SpCO was close to 15% but not >15% were categorized as “false low” SpCO tests. Examples include cases in which SpCO vs. COHb were 14% vs. 16.9%, 14% vs. 16.4%, and 11% vs. 18.8%. These were all categorized as “incorrect” SpCO readings, when in fact all of them would be interpreted as “elevated CO.” Additionally, in each of these cases the SpCO levels were lower than COHb levels. Given that SpCO does not have a negative bias, this indicates a device malfunction, inappropriate use, or blood samples taken at different times than the SpCO measurements.

The authors stated their results were not consistent with other reported results, but did not state the degree to which the results were different.

“The level of agreement demonstrated in the present study is somewhat less than that reported in the only other published reports that included analyses of agreement between measurements made with the RAD device and laboratory values.”

The study authors stated the following limitations of their study:
>> “The study was conducted at a single hospital ED in a hyperbaric referral center. The results may not necessarily be generalizable to out-of-hospital settings.”
>> “The predominantly black and Hispanic composition of our sample potentially limits our ability to draw conclusions about performance of the device in patients with lighter skin pigmentation.”
>> “Although RAD measurements were reported in all 120 subjects enrolled in our study, technical difficulty was reported in 12 cases. Analysis of agreement, performed with and without inclusion of these 12 cases, did not substantially change the results or inference about acceptability of agreement.”

Rad-57 SpCO Accuracy
Summary of Published Studies on SpCO Accuracy
The results of the Touger et al study are significantly different than at least five other studies of 1,690 subjects. Each of those studies showed narrower limits of agreement than Touger et al (as shown in Table 1).

Table 1. Results of Independent Studies comparing SpCO to COHb
Authors Year Setting # of Subjects
Limits of agreement for
SpCO vs. COHb (%)
Bias (%) Precision (%)
Mottram et al8 2005
Respiratory
Department
31 -1.5 to +5.5 2.0 1.8
Coulange et al9 2008
Emergency
Department
12 -6.4 to +3.4 -1.5 2.5
Kot et al10 2008
Hyperbaric
Center
49 -7.9 to +8.9 0.5 4.3
Piatkowski et al11 2009 Burn Center 20 -1.5 to +7.8 3.2 2.4
Roth et al12
Pending
Publication
Emergency
Department
1,578 -3.4 to +9.4 3.0 3.3
Suner* et al2 2008
Emergency
Department
64 -15.9 to + 7.5 -4.2 5.9
Touger et al6 2010
Emergency
Department
120 -11.6 to +14.4 1.4 6.6
* The reported mean time difference between SpCO measurements and venous blood draws used for COHb measurements was 67 minutes, preventing the results from being used to examine SpCO accuracy.

The only study with results similar to Touger et al is by Suner et al It is important to understand that the Suner study was conducted primarily as a prevalence study to show the potential for missed elevated COHb levels in the emergency department, not as a study to assess SpCO accuracy. In the Suner study, the average time between the SpCO measurements and venous blood draws used for COHb measurements was 67 minutes.

Time (min)
A Washout On Air
COHb (%)
400
20
15
10
5
0
0 200 600 800

B Washout On Oxygen
Time (min)
COHb (%)
400
20
15
10
5
0
0 200 600 800

Figure 3. The half-life of COHb exponentially changes, whether on air (A) or 100% oxygen (B).13

The average half-life of carbon monoxide is reported to be 74 minutes with 100% oxygen and approximately 3 hours at room air.14 As shown in Figure 3, COHb exponentially changes over time, meaning that even a short time between COHb and SpCO measurements can invalidate the comparison.13 This prevents the Suner et al results from being used to examine SpCO accuracy. After eliminating the Suner et al study from consideration, there are no studies with results similar to Touger et al.

One other study was conducted by Dr. Steven J. Barker and colleagues in ten healthy subjects, with comparisons between SpCO and COHb showing a -1.2% bias and 2.2% standard deviation (-5.5 to +3.1 limits of agreement).15 However, since Dr. Barker is a member of Masimo’s Board of Directors and Chairman of Masimo’s Scientific Advisory Board, the results are not included in Table 1.

Whitepaper

Masimo Internal Test Reports on SpCO Accuracy

Masimo has performed a large amount of comparison testing in the development and validation of SpCO.

COHb (%)
N = 3,629
Bias = 0.2%
Precision = 2.5%
SpCO (%)
45
40
35
30
25
20
15
10
5
5 10 15 20 25 30 35 40 45
0
0
Figure 4. Masimo’s internal test data comparing SpCO to COHb.

A summary of Masimo internal test data is shown in Figure 4 and Table 2. Combining all test results over 3,629 measurements, the bias of SpCO measurements compared to COHb measurements is 0.2% with a 2.5% standard deviation. Expressed in limits of agreement, 95% of SpCO measurements would fall between -4.7 to +5.1% of COHb measurements. When examining all 3,629 measurements, 100% of the measurements fall between -11.6% +8.0% of a laboratory COHb level. In only 23 subjects with COHb >15%, Touger et al reported four SpCO measurements more than 12% below COHb, including SpCO measurements reported to be 13%, 18%, 20%, and 35% below COHb. However, in 3,629 measurements from internal testing, Masimo has never observed a single instance of an SpCO measurement more than 12% below COHb.

Table 2. Masimo Internal Test Data for SpCO vs. COHb

Sensor* Year # of Subjects
# of
Comparisons
Limits of Agreement for
SpCO vs. COHb (%)
Bias (%) Precision (%)
Rev B 2007 93 1,225 -4.4 to +6.0 0.8% 2.6%
Rev C/D* 2008 56 1,464 -4.7 to +5.1 0.2% 2.5%
Rev E 2009 62 940 -4.4 to +4.6 0.1% 2.3%
Combined 211 3,629 -4.7 to +5.1 0.2% 2.5%
* Multiple sensor design enhancements have been implemented since the initial SpCO release. Not all enhancements have been characterized by a formal sensor revision but the table reflects the relative changes in performance over time. Note that not all sensor revisions result in accuracy enhancements.

“In Masimo’s opinion, based on internal testing of 3,629 measurements, it is not possible for a functioning Rad-57 device and sensor to produce these results when the directions for use are followed and the measurements are compared to simultaneous COHb levels.”

Rad-57 SpCO Accuracy

Potential Reasons for Discrepancies Between Touger et al Results and Other Studies

The significantly different results obtained by Touger et al create a scientific need to further investigate the methods used in the study, supported by the editorial statement that “Little information about these potential confounders is provided in the Touger report.” The significant deviation in the results reported by Touger et al1 from previous studies simply do not allow any definitive conclusions to be made, as expressed in both the study and the editorial

Internal test data do not always match real-world test data for a variety of reasons including subject type, pathophysiology, and adherence to manufacturer directions for use in sensor placement and device operation. The possible reasons for the discrepancy between the Touger results and the results of other investigators and Masimo’s internal testing include multiple items addressed in the directions for use of the device and sensor, including:
>> Device malfunction
>> Sensor malfunction
>> Inappropriate finger positioning in sensor (finger placement away from the digit stop will result in incorrect SpCO measurements)
>> Using an inappropriately-sized sensor for size of subject’s finger
>> Timing of SpCO and COHb measurements not being simultaneous (because the half-life of COHb is 74 minutes on 100% oxygen and it drops exponentially)
>> Elevated methemoglobin (which can be ruled out by measuring SpMet,® noninvasive methemoglobin, with the Rad-57)
>> Patient motion
>> External light interference

Customer Feedback on SpCO Measurements with the Rad-57

Over 8,000 Rad-57 devices are in use by clinicians worldwide and Masimo has received complaints from less than 1% of all customers over a 5 year period. Upon further investigation of these complaints, the dominant reason for reported SpCO inaccuracy was due to not following the directions for use.

Importantly, since the introduction of the Rad-57, peer reviewed clinical studies and case studies have shown the clinical value of SpCO.16-19 Masimo has received countless reports from clinicians that the device has enabled them to save lives and limit the damaging effects of CO poisoning. In one of many testimonials Masimo has received, Skip Kirkwood, Chief of the EMS Division in Wake County, North Carolina, stated: “We believe that all 50+ people in the hotel would have been dead at dawn if it were not for this lifesaving intervention from Masimo.”

Field Assessment of Rad-57 SpCO Measurements

Masimo stands by its products’ performance and knows that when SpCO-enabled devices are used according to their directions for use, they provide accurate SpCO measurements that provide significant clinical utility, helping clinicians detect CO poisoning in otherwise unsuspected patients and rule out CO poisoning in patients with suspected CO poisoning. Hundreds of hospitals and thousands of Fire and Emergency Medical Services organizations around the world use the Rad-57 every day to help them improve patient outcomes and reduce the cost of care. Masimo welcomes additional investigation into the accuracy and clinical application of its SpCO measurement in both emergency department and other clinical settings, for the benefit of patients and caregivers.

Masimo encourages any clinician who questions the accuracy of SpCO measurements from the Rad-57 device to perform their own evaluation and compare the readings to an invasive blood sample and perform laboratory analysis. Masimo asks in return, with the mutual goal of improved patient care, for the ability to appropriately train all operators of the device and its directions for use.

Whitepaper

As much as Masimo stands behind its product and its accuracy specification, it should augment and not replace the caregivers assessment of the patient. If clinicians question SpCO readings from the Rad-57 device based on their inspection of the patient, they should repeat the test (following the directions for use). If the caregiver further questions the results of the device, they should obtain an invasive blood sample and perform laboratory analysis.

Conclusion

The Rad-57 is a safe and effective device for SpCO measurements. Except for one study, multiple published studies have confirmed the accuracy of SpCO. In addition, 3,629 measurements from Masimo’s internal testing do not show the results reported in just 120 tests by Touger et al Masimo does not believe Touger et al results represent the true performance of the Rad-57. Masimo stands behind the Rad-57 and offers to help any institution do their own evaluation to prove to themselves the stated accuracy of Rad-57. The Rad-57 has been effectively used by caregivers to help patients and emergency personnel, but no device should replace the caregiver’s observation of the patient. Use of the of Rad-57 with SpCO has enabled caregivers around the world to save many lives that would otherwise not have been saved.

References

1. Hampson NB et al. Carboxyhemoglobin measurement by hospitals: implications for the diagnosis of carbon monoxide poisoning. J Emerg Med. 2006; 31(1):13-16.
2. Suner S, Partridge R, Sucov A, et al. Non-invasive pulse CO-oximetry screening in the emergency department identifies occult carbon monoxide toxicity. J Emerg Med. 2008;34(4):441-50.
3. Masimo Corp. Masimo SET Rad 57 Pulse CO-Oximeter. 510(k) summary of the safety and efficacy. Available at http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPMN/pmn.cfm?ID=16285. Accessed August 10, 2010.
4. Bland JM, et al. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1:307-310.
5. Gehring H, et al. Hemoximetry as the gold standard: Error assessment based on differences among identical blood gas analyzer devices of five manufacturers. Anesth Analg. 2007;105:S24-30.
6. Touger M, Birnbaum A, Wong J. Performance of the Rad-57 pulse CO-oximeter compared with standard laboratory carboxyhemoglobin measurement. Ann Emerg Med. 2010. June 2. Article in press.
7. Maisel WH, Lewis RJ. Noninvasive measurement of carboxyhemoglobin: How accurate is accurate enough? Ann Emerg Med. 2010. June 2. Article in press.
8. Mottram CD, Hanson LJ, Scanlon PD, et al. Comparison of the Masimo Rad 57 pulse oximeter with SpCO technology against a laboratory CO-oximeter using arterial blood. Resp Care. 2005; 50(11):252. (Abstract).
9. Coulange M, Barthelemy A, Hug F, et al. Reliability of new pulse CO-oximeter in victims of carbon monoxide poisoning. Undersea Hyperb Med. 2008; 35(2):107-11.
10. Kot J, Sicko Z, Goralczyk P. Carbon monoxide pulse oximetry vs direct spectrometry for early detection of CO poisoning. Anestezjol Intens Ter. 2008; 40(2):75-8.
11. Piatkowski R, Budaj-Fedeka A, Scislo P, et al. A new tool for the early diagnosis of carbon monoxide intoxication. Inhal Toxicol. 2009;21(13): 1144-7.
12. Roth D, Herkner H, Schreiber W, Hubmann N, Laggner AN, Havel C. Bias and precison of noninvasive CO-oximetry in daily clinical use. Pending publication.
13. Bruce M, Bruce E. Analysis of factors that influence rates of carbon monoxide uptake, distribution, and washout from blood and extravascular tissues using a multicompartment model. J Appl Physiol 2006; 100:1171-1180.
14. Weaver LK, Howe S, Hopkins R, Chan KJ. Carboxyhemoglobin half-life in carbon monoxide-poisoned patients treated with 100% oxygen at atmospheric pressure. Chest. 2000 Mar;117(3):801-8.
15. Barker SJ, Curry J, Redford D, et al. Measurement of carboxyhemoglobin and methemoglobin by pulse oximetry. Anesthesiology. 2006; 105:892-7.
16. Nilson D et al. Noninvasive carboxyhemoglobin monitoring: screening emergency medical service patients for carbon monoxide exposure. Prehosp Disast Med. 2000; 25(3)L253-256.
17. Roth D et al. Victim of carbon monoxide poisoning identified by carbon monoxide oximeter. 2009. epub.
18. Bledsoe BE et al. Use of pulse co-oximetry as a screening and monitoring tool in mass carbon monoxide poisoning. Prehospital Emergency Care. 2010. 14:131-133.
19. Raising Medical Professionals’ Awareness of Carbon Monoxide Poisoning. UK House Of Commons All Party Parliamentary Gas Safety Group. January 2009.

6126-6588A-1110 © 2010 Masimo Corporation. All rights reserved.
Masimo International
Tel: +41 32 720 1111
info-international@masimo.com
Masimo U.S.
Tel: 1 877 4 Masimo
info-america@masimo.com