Since the Great Recession ended, many assumed that once the housing market recovered, the economy would finally grow at a healthy pace. But as history has shown, this is a myth. The direction of the housing market doesn’t say a whole lot about the health of the overall economy and where it’s headed.

It’s not that home prices and sales aren’t important. After all, the vast majority of Americans’ wealth is still tied to the homes they own, as opposed to other assets, such as the stock market. And so the thinking goes: If home prices rise, homeowners would feel richer and therefore spend more, which would in turn, drive economic growth.

While that may generally be true, the housing market isn’t always so neatly linked to GDP, says Dean Maki, U.S. chief economist at Barclays. It also doesn’t help that the housing market makes up a much smaller share of the U.S. economy today than it did before the market collapsed: Whereas residential investment peaked at 6.3% of GDP in the middle of 2005, it has spiraled down to 2.7% today.

MORE: How the Publicis-Omnicom deal started as a joke

Just because the housing market grows at a healthy clip, it doesn’t mean the economy will. At various points throughout history, housing either rebounded or boomed while the economy didn’t grow as quickly and vice versa. Last year, the housing market recovered in an unexpectedly big way while the economy slogged along; residential investment grew 12.1%, but GDP grew only 2.2%. Also, between 1987 through 1999, residential investment grew faster at an annualized rate of 5.3% than GDP at 4.5%.

By contrast, there were periods when the housing market actually shrunk, even as the economy grew: Between 1965 and1967, residential investment contracted at an annualized rate of 5% while GDP grew 5.1%.

“The point is not that housing doesn’t matter, but there are times when other factors matter a lot more,” Maki says.

For 2014, residential investment is expected to grow 10.3%, while the economy will likely grow only 2.2% as tens of thousands of federal employees face furloughs and cuts to government programs, Barclays forecasts. This is better than no growth, but the imbalance is hard to dismiss.

MORE: Investors who love QE should fear Larry Summers

Higher taxes and spending cuts that kicked in earlier this year will drag down growth, reducing GDP by 0.6 % this year, according to estimates by the U.S. Congressional Budget Office. However, canceling the spending cuts could add between 300,000 to 1.6 million new jobs by the end of 2014; GDP could also be 0.2% and 1.2% higher, according to a separate report by the CBO that looked into the implications of eliminating the spending cuts starting in August and all of 2014.

So for all the attention paid to the housing market, it doesn’t capture the economy Americans face today.