Without evidence of benefit, an intervention should not be presumed to be beneficial or safe.

- Rogue Medic

How did this happen? – Research

Part of the problem with research is the same as the problem with all prediction. We are not good at it. We remember the things we were right about, but we conveniently forget the things we were wrong about.

Nassim Nicholas Taleb[1] describes part of the problem by reversing the situation. Do not try to predict the future, but try to predict the past. Imagine an ice cube placed on a table and try to imagine the way that it will melt. What will the result look like? The result will be a puddle of water.

Now, don’t imagine the situation as one of predicting the future. Imagine you are faced with a puddle of water like the one from the first example. Now, try to imagine what that puddle came from. Was it an ice cube? Was it condensation from a cold glass? Was it something entirely different that produced this puddle? If it did come from a piece of ice, was it an ice cube, a small ice sculpture containing the same amount of water, did some of the water evaporate before you saw the puddle? Is the substance that forms this apparently clear puddle actually water?

When performing research we need to try to control variables, so that we know as much as possible, what happened at each step of the experiment. If we put an ice cube down on a table, then leave, and come back and see a puddle, do we know that the puddle is the result of the ice cube melting? 

No, we do not. Is it reasonable to assume this? Yes, it is a reasonable thing to assume, but research is not about assuming things that are likely. Research is about controlling for all variables, especially those that can easily be controlled for.

When performing research, we need to control everything that we can reasonably control. If we are going to see what happens to an ice cube, we need to sit and watch the ice cube melt. Or we can record the events, so that we can examine the events later. If we are sitting there, watching the ice cube, and somebody comes along, puts the ice cube in a glass with a lot of other ice, puts the glass in the same spot, and the condensation from the glass forms the puddle, we need to know this. This would completely change our results. 

Is this far fetched? It does seem to be, but how do we know until we perform the experiment? If we assume things because we think we already know what is going to happen, then we are fooling ourselves. We can assume all sorts of things, just because the seem like common sense. That is not research. Anybody claiming that it is research is wrong. Unfortunately, this kind of carelessness is not uncommon in EMS research.

Do we really care about an ice cube? No. At least, I do not. This is just a very simple example of how we can assume things in research that will lead to a result that is worse than worthless. Why worse than worthless? Because the resources that could have been used to perform a valid experiment have now been wasted. Because the patients who have been exposed to the experimental treatment will never know if they might have benefited from the study – and neither does anyone else.

Of course the investigators will claim that they were able to demonstrate all sorts of useful information, but this is only because they are incompetent. We should not encourage them.

Actually, we should punish them.

Let’s look at the biggest problem of EMS research – quality.

All sorts of criteria are examined, when performing EMS research. Rarely examined is the quality of the providers participating in the study. Are they typical for the organization? A large enough study can take care of that. Are the providers in the organization representative of excellent, or even just competent, EMS ability? 

To many people, just asking that question is an insult. This should give you a hint of what the answer is for that organization. If they are not constantly questioning their quality, how do they have any idea?

Just because an EMS organization is questioning their quality, does not mean that they are asking the right questions, but it is a good start.

What should we ask?

Was this result from an EMS organization with aggressive medical oversight? Requirements for OLMC (On Line Medical Command) permission are not an example of aggressive oversight. This would better be compared to the Wizard of Oz pulling all kinds of levers and making loud noises to create an impressive spectacle. A spectacle that does not have any substance and is supposed to disguise the reality. We need to avoid the smoke, the mirrors, the man behind the curtain, and look for real indicators of quality.

What can we look for to indicate that an EMS organization is able to provide the kind of quality oversight that would recommend them as a site for evaluation of trauma triage criteria?

If the question is – Can EMS safely triage trauma patients by physical assessment, rather than by mechanism criteria? – then these are some of the questions we should be asking – 

Do they have feedback from the trauma center about patients transported to the trauma center?

If not why not?

HIPAA does not forbid this. 

Do they have feedback from the local hospital about patients transported to the local hospital? 

If not why not?

Again, there is no HIPAA problem, here. If anybody is claiming HIPAA, they are telling you a lie. 

Is the medical director following up on all of the trauma alerts, potential trauma alerts, mechanism alerts, patients who should have gone to a trauma center (which is a huge can of worms on its own – does an ICU admission mean a sick patient or a clueless ED?), . . . ?

If not why not?

Are there continuing education classes available?

If not why not? 

Are opportunities available to spend time in the trauma center performing assessments?

If not why not?

If you believe that Dr. Scalea is correct, that we should not be frugal when it comes to people’s lives (I do agree with this), then why aren’t we making sure that we have excellent EMS providers taking care of these patients? 

Are only trauma center patients deserving of excellent care?

Are only trauma patients deserving of excellent care?

Medical patients require much more critical judgment by EMS. They will only be flown to Shock Trauma when they also have a trauma complaint. Are they unimportant?

Certainly not.

What about feedback on medical patients?

How many pneumonia patients receive furosemide from EMS?

How many pneumonia patients receive furosemide from the hospital?

All that crackles is not CHF.

CHF should not be treated by EMS with furosemide anyway.

This is another way of finding the organizations that not only should not be participating in research, but should not be participating in EMS. 

But this is not trauma and we are only interested in trauma.

Not true. CHF patients are victims of trauma, too. Medical decision making does not limit itself to trauma, or medical, or IV skills, or intubation skills, or . . . . We need to look at the capabilities of those we seek to use as our example of EMS providers. The mediocre need not apply.

If the assessment skills are not common, we need to improve them, so that they become more common. We cannot throw our arms up and whine about it being impractical. This high quality is demonstrated in some EMS organization. 

High quality is not too expensive. If it is considered too expensive, then the organization should be limited to BLS care only.

Narrative Fallacy –

Narrative Fallacy I

How did this happen? – Research

Narrative Fallacy II

CAST and Narrative Fallacy

C A S T and Narrative Fallacy comment from Shaggy

Some Research Podcasting Comments

Shaggy Comments on Some Research Podcasting Comments.

Spine Immobilization in Penetrating Trauma: More Harm Than Good?

EMS EdUCast – Journal Club 2: Episode 43

Education Problems, Autism, and Vaccines

Footnotes:

[1] The Black Swan: The Impact of the Highly Improbable
By Nassim Nicholas Taleb
A must read book. If you have anything to do with risk management, then uncertainty/randomness/the unexpected are important parts of what you do. He deals with them better than anyone else. Too many misunderstand his writing, perhaps because they cannot abandon their own biases and accept their lack of control of events. While I find his prose to be awkward (perhaps he does not appear to be awkward, when compared to my writing, so maybe it is just me), his conclusions are essential to the understanding of risk management. Risk management people include any of us who treat patients.
Article about The Black Swan.

.

Trackbacks

  1. […] at explaining their results wander into narrative fallacy, which I have written about here, here, here, here, here, here, here, here, here, and […]

Speak Your Mind