Good Models and Bad Models

By jpeter2 from Pixabay

Good Models and Bad Models

By: Michael Ashton | Mon, Mar 13, 2017

I have recently begun to spend a fair amount of time explaining the difference
between a “good model” and a “bad model;” it seemed to me that this was a reasonable
topic to put on the blog.

The difference between a good model and a bad model isn’t as obvious as it
seems. Many people think that a “good model” is one that makes correct predictions,
and a “bad model” is one that makes bad predictions. But that is not the case,
and understanding why it isn’t the case is important for economists
and econometricians. Frankly, I suspect that many economists can’t articulate
the difference between a good model and a bad model…and that’s why we have
so many bad models floating around.

The definition is simple. A good model is one which makes good predictions if
high-quality inputs are given to the model
; a bad model is one in which
even the correct inputs doesn’t result in good predictions. At the limit,
a model that produces predictions that are insensitive to the quality of
the inputs – that is, whose predictions are just as accurate no matter what
the inputs are – is pure superstition.

For example, a model of the weather that depends on casting chicken bones
and rat entrails is a pretty bad model since the arrangement of such articles
is not likely to bear upon the likelihood of rain. On the other hand, a model
used to forecast the price of oil in five years as a function of the supply
and demand of oil in five years is probably an excellent model, even though
it isn’t likely to be accurate because those are difficult inputs to know.
One feature of a good model, then, is that the forecaster’s attention should
shift to the forecasting of the inputs.

This distinction is relevant to the current state of practical economics because
of the enormous difference in the quality of “Keynesian” models (such as the
expectations-augmented Phillips curve approach) and of monetarist models. The
simplest such monetarist model is shown below. It relates the GDP-adjusted
quantity of money to the level of prices.

M2/GDP and GDP Deflator 1913-2013

This chart does not incorporate changes in money velocity (which show up as
deviations between the two lines), and yet you can see the quality of the model:
if you had known in 1948 the size of the economy in 2008, and the quantity
of M2 money there would be in 2008, then you would have had a very accurate
prediction of the cumulative rate of inflation over that 60-year period. We
can improve further on this model by noting that velocity is not random, but
rather is causally related to interest rates. And so we can state the following:
if we had known in 2007 that the Fed was going to vastly expand its balance
sheet, causing money supply to grow at nearly a 10% rate y/y in mid-2009, but
at the same time 5-year interest rates would be forced from 5% to 1.2% in late
, then we would have forecast inflation to decline sharply over that
period. The chart below shows a forecast of the GDP deflator, based on a simple
model of money velocity that was calibrated on 1977-1997 (so that this is all

GDP Deflator, 4q Average 2006-2016

That’s a good model. Now, even solid monetarists didn’t forecast that inflation
would fall as far as it did – but that’s not a failure of the model but a failure
of imagination. In 2007, no one suspected that 5-year interest rates would
be scraping 1% before long!

Contrariwise, the E-A-Phillips Curve model has a truly disastrous forecasting
history. I wrote an article
in 2012
in which I highlighted Goldman Sachs’ massive miss from such a
model, and their attempts to resuscitate it. In that article, I quoted these
ivory tower economists as saying:

“Economic principles suggest that core inflation is driven by two main factors.
First, actual inflation depends on inflation expectations, which might have
both a forward-looking and a backward-looking component. Second, inflation
depends on the extent of slack (or spare capacity) in the economy. This is
most intuitive in the labor market: high unemployment means that many workers
are looking for jobs, which in turn tends to weigh on wages and prices. This
relationship between inflation, expectations of inflation and slack is called
the “Phillips curve.”

You may recognize these two “main factors” as being the two that were
thoroughly debunked
by the five economists earlier this month, but the
article I wrote is worth re-reading because it describes how the economists
re-calibrated. Note that the economists were not changing the model inputs,
or saying that the forecasted inputs were wrong. The problem was that even
with the right inputs, they got the wrong output…and that meant in their
minds that the model should be recalibrated.

Core CPI Inflation, Actual versus Predicted

But that’s the wrong conclusion. It isn’t that a good model gave bad projections;
in this case the model is a bad model. Even having the actual data –
knowing that the economy had massive slack and there had been sharp declines
in inflation expectations – the model completely missed the upturn in inflation
that actually happened because that outcome was inconsistent with the model.

It is probably unfair of me to continue to beat on this topic, because the
question has been settled. However, I suspect that many economists will continue
to resist the conclusion, and will continue to rely on bad, and indeed discredited,
models. And that takes the “bad model” issue one step deeper. If the production
of bad predictions even given good inputs means the model is bad, then perhaps
relying on bad models when better ones are available means the economist is

P.S. Don’t forget to buy my book! What’s
Wrong with Money: The Biggest Bubble of All
. Thanks!

You can follow me @inflation_guy!

Enduring Investments is a registered investment adviser that
specializes in solving inflation-related problems. Fill out the contact form
at and
we will send you our latest Quarterly Inflation Outlook. And if you make sure
to put your physical mailing address in the “comment” section of the contact
form, we will also send you a copy of Michael Ashton’s book “Maestro, My Ass!”

Michael Ashton

Michael Ashton, CFA

Michael Ashton

Michael Ashton is Managing Principal at Enduring
Investments LLC
, a specialty consulting and investment management boutique
that offers focused inflation-market expertise. He may be contacted through
that site. He is on Twitter at @inflation_guy

Prior to founding Enduring Investments, Mr. Ashton worked as a trader, strategist,
and salesman during a 20-year Wall Street career that included tours of duty
at Deutsche Bank, Bankers Trust, Barclays Capital, and J.P. Morgan.

Since 2003 he has played an integral role in developing the U.S. inflation
derivatives markets and is widely viewed as a premier subject matter expert
on inflation products and inflation trading. While at Barclays, he traded
the first interbank U.S. CPI swaps. He was primarily responsible for the creation
of the CPI Futures contract that the Chicago Mercantile Exchange listed in
February 2004 and was the lead market maker for that contract. Mr. Ashton
has written extensively about the use of inflation-indexed products for hedging
real exposures, including papers and book chapters on “Inflation and Commodities,” “The
Real-Feel Inflation Rate,” “Hedging Post-Retirement Medical Liabilities,” and “Liability-Driven
Investment For Individuals.” He frequently speaks in front of professional
and retail audiences, both large and small. He runs the Inflation-Indexed
Investing Association

For many years, Mr. Ashton has written frequent market commentary, sometimes
for client distribution and more recently for wider public dissemination.
Mr. Ashton received a Bachelor of Arts degree in Economics from Trinity University
in 1990 and was awarded his CFA charter in 2001.

Copyright © 2010-2017 Michael Ashton

All Images, XHTML Renderings, and Source Code Copyright ©
Published at Mon, 13 Mar 2017 09:09:58 +0000

About the author


Leave a comment: