The national average for a gallon of regular gasoline is at its lowest level in four years, giving U.S. consumers billions more to spend this holiday season, but that extra money is being offset by higher food costs, which account for a bigger chunk of most consumers’ budgets. The latest sales reports issued by major retailers, including Macy’s (M), Gap (GPS) and Michael Kors (KORS), have come in lighter than expected, signaling lower gas costs aren’t resulting in greater spending at the malls.

Greg Foran, head of Wal-Mart’s (WMT) U.S. division, said lower gas prices were “helping” as the discounter makes its way through the holiday season. But at the same time, grocery comparable sales at the division are flat, despite the higher prices of meat and dairy, as shoppers traded down to less expensive items in those high-inflation categories.

Lower gas prices — down about 9% compared to last year — are adding a few billion to U.S. consumers’ 2014 holiday spending budget, Customer Growth Partners President Craig Johnson told Fortune. But with food accounting for some 15% of household budgets, shoppers will overall have $5 billion less in their pockets, according to Johnson’s estimates. That might explain why some retailers were talking down how much of a benefit lower prices at the pump might yield.

“We’ve seen some relationship in terms of discretionary spending and gas prices,” Macy’s Chief Financial Officer Karen Hoguet told analysts this week on a call to discuss the department store’s sales decline last quarter.

“But I think customers are choosing to spend their disposable dollars in different ways, and that’s part of the reason why we are not more optimistic about the lower gas prices,” she added, noting that consumers are spending more on cars, healthcare, electronics and home improvement, leaving little upside for Macy’s.

And in any case, shoppers still have a voracious appetite for deals, meaning they won’t go hog wild just because of a few more dollars in their pockets.

“Despite the fact they have more money to spend because of lower fuel prices, they’re still very savvy and want to make sure they’re spending their money wisely,” J.C. Penney’s (JCP) CEO Myron Ullman told analysts on a call. The retailer also reported disappointing sales last quarter.

The higher price for dairy and meat is particularly problematic. The beef and veal index has surged 17% since the beginning of the year through September, while prices for dairy-related products have increased for 10 of the past 11 months, according to the Commerce Department.

Food makers such as Kraft (KRFT) and Mondelez (MDLZ) have reported soft volume as they’ve boosted the prices of their products to help offset the higher commodity costs.

Mondelez, the maker of Ritz crackers and Oreos, said recently it has raised prices across “all of our categories.” The company recently told analysts that prices rose most significantly for chocolate and coffee as a result of the steep increase in both cocoa and green coffee costs.

Kraft has also moved aggressively on pricing, saying costs have risen for cheese, meats and other commodities. As a result, consumers are still feeling pinched.

“The 60% of households that make $50,000 a year or less are still quite strapped,” said Kraft Chief Executive Tony Vernon in a recent conference call.