Without evidence of benefit, an intervention should not be presumed to be beneficial or safe.

- Rogue Medic

Experts, Forecasting, Causality, and Complexity

 

A huge problem in medicine is the failure to see where problems will come from, even when those problems appear to be obvious in hindsight.

The field of economics shares this problem. A perfect example is the recent release of the transcript from the August 2007 Federal Reserve Board meeting. This is 136 pages of stuff that even I am not interested enough to do more than skim through.[1] There is too much good medical research, and some good books, already on my to do list. However the New York Times paid someone to look through this and these are some of the insi

However, there is a lot to think about in just the press release. This was about a year and a half after Alan Greenspan had stepped down as the inscrutable, indecipherable (but not inconceivable), philosopher king of government financial oversight. The Maestro.
 

President George W. Bush presents the Presidential Medal of Freedom to Alan Greenspan, on November 9, 2005 in the East Room of the White House.[2]

 

The markets did well. Few understood what Chairman Greenspan was saying and it appeared that opacity was the goal. People looking for cause and effect assumed that the cause of the success of the markets was Alan Greenspan.

His successor was Ben Bernanke, who did try to make policy less obscure, but otherwise tried to carry on the Greenspan traditions.
 

Economic growth was moderate during the first half of the year. Financial markets have been volatile in recent weeks, credit conditions have become tighter for some households and businesses, and the housing correction is ongoing. Nevertheless, the economy seems likely to continue to expand at a moderate pace over coming quarters, supported by solid growth in employment and incomes and a robust global economy.[3]

 

The market was looking good at the time. The Standard & Poor’s 500 index is one of the best indicators of the state of the market. Here is where the market was at the time of the meeting.
 

Image credit. Click on images to make them larger.
 

There were already serious problems with the economy – mortgage defaults (especially sub-prime), falling housing prices (which the government subsidies helped raise), and recent multi-trillion dollar losses in various markets.

Chairman Bernanke had some odd things to say for someone who is supposed to understand the economy.
 

On the positive side, the repricing of risk and the reevaluation of underwriting standards seem appropriate.[1]

 

The price of risk was completely out of line, so a move in the direction of something rational would be appropriate, if it were more than a token move.
 

It is an interesting question why what looks like $100 billion or so of credit losses in the subprime market has been reflected in multiple trillions of dollars of losses in paper wealth. So it’s an interesting question about what is going on there.[1]

 

When s huge section of the investing community has invested heavily in one area, eventually they will have nobody left to sell to – or even worse, they start believing that what they have been selling is really worth what they have been telling people it is worth. The only bank to not lose a lot of money was Goldman Sachs, but that appears to be because some of their people copied the trades that John Paulson[4] was buying from them.
 

Obviously, the markets right now are not functioning normally.[1]

 

And the chart below shows what happened after the meeting (big red dot) up until January 13, 2013.

Was the experts’ unanimous endorsement of steady economic expansion over the coming year, or two, at all close to what happened?

Did the experts understand the economy or the many factors affecting the markets?

How could the experts forecast steady economic expansion as we were entering an economic collapse?
 

From around 1,500 at the time of the meeting, down to under 700 a year and a half later, then taking almost four years to slowly return to where the market was at the beginning of this continuing gradual expansion.

The market was cut in half. Then it lost some more.
 

We like to deceive ourselves.

Nassim Taleb wrote about this in a book that did a much better job of anticipating the collapse, and he published that book months before this meeting, while things were looking much better –
 

We attribute our successes to our skills, and our failures to external events outside our control.[5]

 

True of many in medicine.
 

It has been more profitable for us to bind together in the wrong direction than to be alone in the right one.[5]

 

That is also a problem in medicine.
 

You cannot ignore self-delusion. Lack of knowledge and delusion about the quality of your knowledge come together-the same process that makes you know less also makes you satisfied with your knowledge.[5]

 

Again, true in medicine.
 

Well, a gentleman called Alan Greenspan, the former chairman of the U.S. Federal Reserve Bank, went to Congress to explain that the banking crisis, which he and his successor Bernanke helped cause, could not have been foreseen because it “had never happened before.” Not a single member of congress was intelligent enough to shout, “Alan Greenspan, you have never died before, not in eighty years, not even once; does that make you immortal?”[5]

 

We only seem to consider what we have already seen – as if that is somehow a limit beyond which no bad things can happen. So we are surprised when this imaginary limit, that only exists in our heads as an anchoring effect,[6] is exceeded.
 

Science alone of all the subjects contains within itself the lesson of the danger of belief in the infallibility of the greatest teachers in the preceding generation … Learn from science that you must doubt the experts. As a matter of fact, I can also define science another way:

Science is the belief in the ignorance of experts.
– Richard Feynman.

 

This is echoed by Nassim Taleb –
 

The problem with experts is that they do not know what they do not know.[5]

 

When experts discourage us from questioning them, and do not provide good evidence to support their positions, those experts deserve criticism.

Footnotes:

[1] Meeting of the Federal Open Market Committee on August 7, 2007
Released Jan. 18, 2013
FOMC
Free Full Text Download in PDF format from the Federal Reserve Board.

[2] Alan Greenspan
Wikipedia
Article

[3] Federal Open Market Committee Statement
Release Date: August 7, 2007
Press Release

[4] John Paulson
Wikipedia
Article

[5] The Black Swan
Nassim Nicholas Taleb
April 17, 2007
Wikipedia entry about the book

[6] anchoring effect
Skeptic’s Dictionary
Definition

.

Comments

  1. Very strange Tim – I was just Googling “black swan” theory yesterday! It’s also good to see that your already wide-ranging interests (EBM, quack medicine, creationists…) are expanding in the new year.

    Skepticism of the experts is all well and good, but it’s problematic what we should replace it with. Feynman can talk about the “ignorance of experts,” but he was an acknowledged expert in quantum mechanics. It becomes circular at this point; you have to learn something about a topic in order to validly question the orthodoxy, and at some point you hopefully develop mastery in the subject.

    I’m not sure that the “problem with experts” that Taleb describes is unique to experts. As another paradigm-breaking physicist put it, experts are people who have earned their stripes by making mistakes in their field, not by always being correct. Who else can we depend on to know what we don’t know? How can the non-experts avoid the traps that experts fall into? I

    • Brooks,

      Very strange Tim – I was just Googling “black swan” theory yesterday! It’s also good to see that your already wide-ranging interests (EBM, quack medicine, creationists…) are expanding in the new year.

      I have been writing about topics covered by Nassim Taleb since the beginning of the blog. I have not written on that much lately.

      How did this happen? – Research

      Risk Management – Why is it so difficult?

      Narrative Fallacy II

      Critical judgment, and out failures in critical judgment, are topics that are important and need to be better covered in EMS and medicine in general.

      Skepticism of the experts is all well and good, but it’s problematic what we should replace it with. Feynman can talk about the “ignorance of experts,” but he was an acknowledged expert in quantum mechanics. It becomes circular at this point; you have to learn something about a topic in order to validly question the orthodoxy, and at some point you hopefully develop mastery in the subject.

      The point is not that experts do not know what they are talking about, but that experts can easily make assumptions that are not reasonable and are not questioned because the people making the assumptions are experts.

      In science, there is no requirement that a paper be written by someone with any credentials. Some journals may use that as an excuse to not publish a paper, but there is nothing in the scientific method about credentials, or years of experience, or age, or ideology, or dogmatism.

      these are prejudices that we introduce into the scientific process because of our lack of integrity.

      Science is neutral.

      Experts are not good at being neutral. That is why the scientific method is about taking pains to eliminate all possible forms of bias that may influence the outcome. Too many variables are excused as things that would not affect the outcome, even though the expert is only assuming that it would not affect the outcome.

      IST-3 was horribly flawed in many ways – especially by having family members grade the outcome after knowing whether their family member received the active treatment or the placebo. This was excused by one of the biggest experts in the field. Rather than state that this study is abuse of research subjects (since nothing much will be learned from a badly done study), a waste of money, and an example of expert opinion leading to blind acceptance of nonsense.

      Do I need to be an expert to see the flaws in this study?

      A high school student should be able to reject this as not worthy of publication.

      I’m not sure that the “problem with experts” that Taleb describes is unique to experts.

      This is not unique to experts, but experts may be less likely to be questioned on their assertions.

      Look at the way IST-3 was accepted by the Lancet. The Lancet is reported to have peer review, but how did anyone competent approve this for publication?

      So, to test the prevailing intellectual standards, I decided to try a modest (though admittedly uncontrolled) experiment: Would a leading North American journal of cultural studies . . . publish an article liberally salted with nonsense if (a) it sounded good and (b) it flattered the editors’ ideological preconceptions?

      The answer, unfortunately, is yes.

      – Alan Sokal – A Physicist Experiments With Cultural Studies.

      If experts were in some way less likely to allow their biases to mislead them, this would not be a problem.

      This is easily demonstrated when an expert discusses something outside of his field of expertise. Does the expert acknowledge the lack of expertise in other fields? Experts are humans, and as humans are good at persuading themselves that they know more than they really do.

      As another paradigm-breaking physicist put it, experts are people who have earned their stripes by making mistakes in their field, not by always being correct. Who else can we depend on to know what we don’t know? How can the non-experts avoid the traps that experts fall into?

      The idea is not that experts are always, or even usually, wrong.

      The idea is that we should not trust experts any more than we trust novices. Experts need to demonstrate objectivity in assessment of the evidence as much as anyone else.

      Expertise should not be trusted.

      Expertise should be questioned.

      True experts should not have any problem with being questioned.

      .