For March, economists on average expected employers would add about 220,000 workers to their payrolls. In fact, the number came in at 88,000.

Baker says few people should have been caught off guard. He saw a number of signs that indicated the jobs numbers were sure to drop. Worse, Baker says Wall Street remains too optimistic about the job market, still expecting a rebound later in the year. Baker doesn’t see that.

MORE: Why the Fed can’t help the long-term unemployed

Instead, he thinks the economy will average job growth of around 100,000 for the rest of the year, better than March, but not enough to continue to bring down the unemployment rate. “I don’t see a good picture for the rest of the year,” says Baker. Here’s what else Baker had to say about the job market for the rest of the year, and why he thought a slowdown was in the cards.

Why is hiring slowing?

First of all, the weather. If we don’t get hit by a snowstorm, it makes a difference at the beginning of the year. Hiring looks better than usual. And that’s what happened. But all that is doing is taking away from jobs that usually get filled later in the year, when the weather improves.

Second, everyone is talking about the big rebound in housing. But it’s not really translating into this boom in construction. It’s up, and now accounts for about 2% of GDP, but that’s still way down from where it was. Residential construction spending had been 6% of the economy before the recession.

MORE: Investor frenzy over housing has peaked

Lastly, consumption remains down. Before the recession, the savings rate was essentially zero. Now it’s been around 4%. That’s a big difference and a huge cut in consumption.

But consumer spending appears to be up, even though everyone thought it would drop after the payroll tax increase. Isn’t that a sign that the U.S. consumer is back?

There are other things going on that are making the consumption picture look better than it is. At the end of last year, a number of companies paid special or larger than usual dividends. That extra investment income is what has been driving up consumption recently. But those dividend payments didn’t go to most Americans, just the wealthy. So I think if you compared the recent sales of upscale retailers to Wal-Mart (WMT) you would see a very different picture.

That’s how you get to 100,000 jobs a month for the rest of the year?

We are only looking at an economy that is growing at 2% a year. And that’s before the drag from the sequester deal, which will begin to slow the economy and the job market in April. So I would be shocked if the economy were to grow much more than that. And that’s about what the average Wall Street economist thinks GDP will do. Yet, on the other hand those same economists are predicting job growth of more like 200,000 a month. That’s not a huge disconnect, but the gap doesn’t make sense to me. Seems very optimistic to me.