Here's why diversification can be an investor's worst enemy

Ask your investment advisor: Why do I need to diversify my portfolio? To improve my investment returns? No. The right answer is to reduce risk.

Indeed, investment advisors, portfolio strategists, and Nobel laureates in economics have spent enormous amounts of time figuring out the best strategies to reduce investment risk. They’ve all concluded that the best way to do this is by diversifying holdings into different asset classes, like stocks, bonds, gold, hedge funds and other strategies meant to smooth out returns.

But does it work? No.

What diversification does is reduce volatility. Diversification does indeed smooth out investment returns, but that’s a psychological decision, not an investment decision. As a result, asset allocation diversification does not help investment performance, it hurts it.

For both professional investors and novices, your single biggest fear is being caught in a major stock market decline of 40% or more, which is why portfolio managers and advisers diversify into bonds and other assets to reduce the volatility of the portfolio.

Based on a fresh analysis my colleagues and I have made, diversification hurts your long term investment performance big time.

First point: if we diversify out of fear of a market crash of 40% or more, how often do these occur? Surprisingly, they have occurred only three times in the last 100 years.

Is it smart to tailor a portfolio to protect against an event this rare? Some will be quick to point out that since the timing of these major declines is unknown, what happens if they occur close to one another? A fair point since two of the three major meltdowns of the century occurred less than 10 years apart, 2000-2001 and 2008-2009. The third was much earlier, in 1973-1974, so plenty of time to recover from that one if you stayed invested.

We decided to test the hypothesis of staying invested through thick or thin. Let’s agree on when the worst possible time might have been to invest in the U.S. stock market in recent times. The obvious conclusion was to invest just before the two major crashes in 2000-2001 and 2008-2009 and stay invested through both crashes. How much would you have lost if you stayed invested until now?

For our purposes, we used the S&P 500 as a proxy for the stock market.

So, on Dec. 31, 1998, our hapless investor decided to plunk his life savings, and his future, into the stock market with a solemn promise to stay invested until his retirement some 25 years later. We observed the value of the index on Jan. 1, 1999 and assumed our investor kept his promise and stayed invested through two of the most horrible stock market periods in history.

We checked his results on December 31, 2019. On the day he invested at the end of 1998, the S&P 500 index stood at 1,229. On December 31, 2019 the index was 3,230.

Our investor’s portfolio, far from losing money, had increased more than 292% (dividends reinvested). But at what psychological cost? During this period, our investor would have had to endure a peak to trough market decline of -50% in 2000-2001, and a peak to trough decline of -53% in 2008-2009. (dividends reinvested, this becomes -45.5% and -50.7% respectively).

Few investors can endure that level of pain.

So, history suggests that the single best way to make a great deal of money is to invest in the stock market for the long term. We’ve all heard that said for decades but even if I can demonstrate convincingly that for an investor under 40 years old a portfolio invested 100% in the S&P 500 index will serve very well, no one believes it and no one follows that advice.

Here’s why: While the stock market has returned close to 10% a year on average with dividends reinvested for over 100 years, it hasn’t been a smooth ride.

There have been decades when the market returned little or nothing, and decades when the annual returns were in the teens. To cite a few examples, if you invested in the index in the decade 2010-2019, your total return would be a gain of 256%, or 13.5% annualized. This is the kind of return most investors would dream of.

It gets better: in the decade of 1990-1999, the total 10-year gain would have been 432%, or 18% annualized and dividends reinvested. That’s “happy dance” time.

It isn’t always this rosy, of course. In the not-so-distant past, 2000-2009, your decade long investment resulted in a loss of -9%, or a loss of -0.9% a year on average for ten years. Enduring a painful period like that requires Zen-like patience.

Good investment advisors all tend to use similar formulas to diversify portfolios, an exercise aimed solely at reducing the extremes in investment performance. As such, the role of the investment advisor is much more akin to that of a psychological or spiritual advisor than an investment expert, which they usually are anyway.

The best investment advisors are not those with the best performance. Indeed, to get high returns you must assume high risks, which is not in the clients’ best interests. Successful advisors are those who can convince their clients to stay invested through feast or famine, reducing the risk that the client will divest holdings entirely out of sheer fear of the market.

The takeaway is that the single best investment strategy to make a fortune is to overcome the emotional side of investing. Buy the S&P 500 index, add to it periodically, and don’t look at it for several decades. If you can do that, a long history of stock market performance says that your success is assured.

—Peter Tanous is the founder and chairman of Lynx Investment Advisory in Washington, D.C. He is the author of several books, including “Debt, Deficits and the Demise of the American Economy” and “The 30-Minute Millionaire,” both with CNBC’s Jeff Cox.