The U.S. labor force has dropped by six million. Workers began to disappear during the Great Recession, but it has continued even after the economy has improved. So what’s going on?

One answer is demographics. Back in 2006, a paper from a number of Fed economists, predicted the current labor force participation rate, which averaged 63.3% in 2013 — the lowest rate in three and a half decades — to the single decimal point. The paper mostly ascribed the drop to retiring baby boomers and the rise of university students.

Critics contend that explanation is lacking. The dropouts aren’t just youngins and retirees. The labor force participation rate has been dropping for workers aged 25 to 54. And in fact, in the past year, the number of dropouts from that category has been increasing. The real reason, those critics say, is the so-called Obummer economy. Despite what you think, there hasn’t been a recovery, at least not as large as some think. And more and more people are deciding it will never get better and dropping out of the workforce. So, not good.

MORE: The dim future for stock prices

Here’s the good news: The demographic explanation might not have as big of a hole as some may think. When analyzing the data, most people stick to the 25 to 54 age group because that’s the one that the Bureau of Labor Statistics makes readily available. But that’s a big and diverse cohort of the working population.

Break down the data a little further, and you get a different picture. Of the 6 million dropouts since the beginning of the Great Recession, just over 500,000, or 8%, are 35 t0 44 years old, according to Heidi Shierholz at the Economic Policy Institute. And this age bracket represents a very large segment of the overall workforce. So the number of prime-aged workers giving up is not that large, especially as a percentage of the entire group of labor force dropouts.

And since baby boomers are now, on average, in their late 50s, a large portion of the dropouts in the 45 to 54 category are probably on the upper end of that scale, which is around the normal time people start thinking about retirement.

That’s not to say that demographics can explain it all. Nearly half of the drop in the labor force came from workers who were 35 or less. And all of the economists that I talked to for this piece said that they are convinced that the persistently poor economy is a large factor in what is driving the dropout rate. “You would expect to see the biggest response to cyclical factors at the extremes,” says Steven Davis, a top labor economist at the Chicago Booth School of Business. Those are the people who can most easily opt to drop out.

MORE: Nest founder: Why I sold to Google

What’s more, the number of workers not counted in the government’s numbers is increasing. There aren’t very many measures of the shadow economy, but the economists who have examined it believe there has been a sizable uptick in undocumented work in the past few years, as manufacturing and other once-stable and union jobs have disappeared. And it’s not just drug sales and gambling; in fact it’s less so these days. While there’s no great data on this, it stands to reason that a growing portion of the underground economy’s workers are middle-aged people who have started their own businesses or have consulting gigs and might not be reporting that work to the government.

That, of course, is not the best outcome, either. The government ends up losing out on tax revenue for one. But it’s not nearly as bleak as the “Obummerists” would like you to believe.