–
Does an exam evaluate competence?
Absolutely not, but we spend this week’s podcast making excuses for using exams, especially the worst exams (multiple choice exams), to do create the illusion of evaluating competence.
Jim Hoffman, Josh Knapp, Bob Sullivan, and I discuss testing on EMS Office Hours.
Do EMS Exams Really Gauge Provider Competency?
Josh is attending the EMT-CC (EMT- Critical Care) course at UMBC (University of Maryland Baltimore County), so part of the discussion is why we need to have this as an add-on
The recent article by Kelly Grayson – Occupy EMS is discussed. That article will be my topic tomorrow.
–
We can come up with more complex multiple choice questions, but the problem is that a multiple choice test is the wrong way to evaluate understanding.
Multiple choice is about recognizing the answer that was written with the intention of being the correct answer. Other choices are written with the intent of misleading us so that we choose one of them, because of some similarity to the correct answer.
We cannot ask the person why he chose that answer, because one of the purposes of multiple choice questions is to prevent interaction between the person being tested and the examiner(s).
–
It is pointed out that not everything is gray, as if this is a justification for applying a black and white guess test to medicine, which is almost never black and white.
–
It is suggested that since doctors will take multiple choice tests as part of medical school, multiple choice tests are valid.
Doctors also complete years of residency after completing medical school. If the multiple choice exam were enough to evaluate competence, doctors would not need to continue to be supervised and evaluated during years of residency.
If a doctor fails to complete his residency successfully, can he claim that he has passed a multiple choice test and that the residency clearly is missing the competence that he demonstrated on the multiple choice test? đł
–
What about nurses? They take a multiple guess test.
Nurses generally start in med/surg and only progress out as they demonstrate the interest in doing more, but nurses will be precepted in these more acute care settings.
–
It is also suggested that the hospital and ambulance time during paramedic school justifies using an irrelevant test, because this has already been evaluated.
If that is the case, we need to throw out the irrelevant test.
If the test is useless, no matter how much we think it is a part of a larger theme.
If the test does not do what it is supposed to do, it is useless.
–
Multiple choice requires that the one (as if there could be just one) correct answer be presented as one of three, or four, or five, or mix and match guesses. The correct answer is always among the guesses presented (even if the correct answer is none of the above).
–
Pick one answer and remember your answer. This is a test. The answer is at the end.
Is it improbable that anyone would be able to pass one of these multiple guess tests by guessing?
The evidence is abundant in the bad EMS providers that we see passing these tests.
Ask them some questions and they will demonstrate their lack of understanding of patient care.
Do this as they leave the test.
Do this a day later.
Do this a week later.
Do this a month later.
Do this a year later.
There will be a lot of people who passed the multiple guess test, but who are not able to demonstrate understanding of the medical topics that the multiple guess test claims to be evaluating.
We all know this, so why do we pretend otherwise?
We don’t know what else to do, because we believe in the test.
Does the test prevent dangerous people from becoming paramedics?
Of course not.
Does the test work?
Of course not.
The test is purely a ceremony of passage that should be viewed exclusively as the superstitious ritual that it is.
–
The answer to the multiple choice question above is at the end.
How many of you knew the answer?
How many of you were able to figure it out?
We can automatically rule out Eeeny, Meeny, and Miny, because they are not relevant to EMS.
Larry and Moe are critical to good patient care, but not the best choices.
Shemp came before Curly. Shemp came after Curly. A couple of good reasons for choosing Shemp, but –
It’s always Curly.
You can never go wrong with Curly.
There is no such thing as too much Curly.
This is EMS lore and will stand up to rigorous validation.
Ask a hundred EMS educators.
It’s Curly.
Ask anyone.
It’s Curly.
However, if the patient is not a Three Stooges fan, or if the patient is a Three Stooges fan, but one of those adherents of a deviant sect that does not acknowledge the greatness of Curly, then this information may not help the patient.
Validation of EMS exams has more to do with popular opinion, than with medical evidence. The lack of a well designed, randomized, placebo-controlled trial comparing Stooges is not a fatal flaw for this test question.
The odds of guessing the correct answer are one in seven. It would be impossible for a million monkeys taking this test to come up with the correct answers by chance, at least that makes sense if we do not understand coincidence and probability.
A 14% chance of coming up with the correct answer by chance, but I would be surprised if any group scored less than 90% correct on this question.
If the questions do not evaluate what they are supposed to evaluate, does the test matter?
.
I seriously did guess Curly, because he was the best stooge lmao!!
Multiple choice tests are a compromise. They give a less accurate assessment of actual knowledge; in exchange they are must faster and more efficient to administer to large numbers of people. As a result, they should only be used for screening or “weeding out” purposes.
In college, most lower-level classes in science disciplines use multiple choice tests; but are never found in the advanced classes for the students who intend to make a career of that discipline. In that vein, multiple choice exams would be fine to administer to students before starting paramedic school, to “weed out” those with insufficient knowledge to become paramedics. Unqualified people would still be admitted; but more accurate tests and assessments can be used on the smaller pool of students.
Ideally, NREMT would not test students at all. The assessments and tests (including internships and clinicals) at the individual schools would be acceptable for entry level as a paramedic. In that mindset, NREMT would concern itself with assessing the SCHOOLS, to make sure they produce quality graduates and eliminate the “degree mills”.
The NREMT is forced to evaluate applicants by testing, since they are a certifying agency. Part of the NREMT’s accreditation requires they ensure the competence of their applicants. Until there is a better way to do that than a multiple choice test, the NREMT will continue to administer that test. Without a test of some sort, there is no NREMT.