During a speech at the Council on Foreign Relations last week, Fed governor Jeremy Stein was asked where he sees economic growth coming from, and he said consumers were one of the main reasons he was optimistic. “Consumers generally seem to be showing some signs of strength.”

They have seemed like a resilient lot. Despite a payroll tax increase, government furloughs, and the rest of the sequester cuts, consumers in general appear to be spending more this year. But that may be an illusion.

MORE: Prepare for a slowdown in the housing recovery

We got a glimpse of that last week. On Wednesday, the government said the economy grew 1.8% in the first quarter. That was down from an earlier estimate of 2.5%. The bulk of the reason for the revision was consumer spending. Americans didn’t open their wallets as wide or as often as originally thought.

What’s more, much of the spending was on big ticket items. In fact, take out a jump in spending on utilities, a quirk of a colder winter than a year ago, and purchases of everyday items, like meals, were nearly flat.

On Thursday, the government said consumer spending picked up in May but again showed a similar picture. Much of the increase in spending came from purchases of cars and home improvements. Take those out and there wasn’t much growth.

Economist Dean Baker, of the Center for Economic and Policy Research, suspects what is really boosting the consumption numbers is not that the finances of average Americans have improved, but instead what we are seeing is a boost from the special dividends companies paid out at the end of last year, in order to avoid the tax hikes of the fiscal cliff. Those went mostly to wealthy Americans. The rising stock market has, too, mostly served to improve the finances of the more well-off. That may explain why big ticket items are rising faster than spending in general.

The concern is that the further we get away from when those dividends were paid out, along with the fact that the stock market has recently flattened out, the big increase in spending from wealthy Americans might subside as well.

MORE: Is J.C. Penny a better bet than gold?

Ken Perkins, who runs Retail Metrics, says luxury retailers have done better than other stores even since the early days of the recovery. But he says the gap between retailers like Tiffany (TIF) that cater to wealthy Americans and Target (TGT) has been growing this year. Perkins says Wal-Mart (WMT), for instance, has been adding more and more non-discretionary items into its store mix in an effort to boost sales.

“It speaks directly to the fact that Wal-Mart’s customers don’t have a lot of money to spend outside of household necessities,” says Perkins.

For a while, when economists talked about consumer spending in this recovery, they talked about the barbell. Discount stores and luxury stores were doing well. Now the bar only seems to be bulging at one end, and even that might not last.