But with interest rates at record lows since December 2008, that model hasn’t work so well. Community banks have seen profits erode. And that will likely continue, given the Federal Reserve’s announcement in January that it would keep interest rates ultra low for at least another two years.

Recently, Fed Chairman Ben Bernanke told community bankers not to worry, that the central bank’s low-interest rate policy would benefit them over time. After all, the thinking is that it would encourage households and businesses to borrow more and therefore help boost the overall economy. And that would eventually ripple to support community banks.

Small bankers aren’t convinced, however. Unlike bigger banks that make money on trading and fees, most community banks rely heavily on net interest margins – the difference between interest earned from loans they make minus interest they pay on deposits. What’s more, the little that banks used to earn on fees has been virtually wiped out by new financial regulations. And demand for loans, while it has risen some recently, remains weak.

It’s a squeeze many smaller banks face – from Connecticut-based Simsbury Bank & Trust to Massachusetts-based Granite Savings Bank.

“It’s kind of like death by a thousand cuts,” says Norm Seppala, CEO and president of the 128-year-old Granite Savings Bank that serves a town of roughly 6,000 people northeast of Boston. “Low interest rates might theoretically eventually help the little banks, but in order for that to happen there needs to be renewed confidence in the economy.” Sepalla adds it’s one of the cheapest times to borrow money and yet his bank (which has about $72 million in assets) is lending only roughly half of what it is able to.

MORE: The global growth that saved Citi

To be sure, the health of community banks has strengthened somewhat in recent months. In his speech to community bankers earlier this month, Bernanke pointed that profits have risen for the past several quarters. And although the proportion of bad loans remains high, the quality of loans has at least stopped deteriorating.

Net interest margins at institutions with less than $1 billion in assets (what the FDIC considers community banks) rose slightly to 3.86% during the third quarter of 2011, compared with 3.83% for the same period during the previous year, according to the Federal Deposit Insurance Corporation, which regulates community banks. What’s more, margins are somewhat higher than they were during the end of 2008 when the Fed lowered rates close to zero.

But the improvement is small relative to where margins have historically been, says Camden Fine, president and CEO of the Independent Community Bankers Association of America.

“Should the near zero interest rate policies of the Fed persist, together with new restrictions on credit and tighter credit standards overall, the net interest margins at community banks will continue a slow and steady decline,” Fine says.

MORE: Bank investors better get used to low returns

Overall, margins have been on a steady decline since the early part 2000 – around the time that the burst of the tech bubble and the September 11, 2001 attacks threatened to undermine the economy. Then Fed Chairman Alan Greenspan embarked on a series of interest cuts that brought down the federal funds rate to 1% by 2004.

At the end of 2008, amid the height of the financial crisis, net interest margins fell to their lowest levels in 20 years, Fine says. Since then, margins have bounced up and down and remain roughly 20% lower from levels in 2003.

And while the latest data shows margins at 3.89%, many parts of the U.S. are seeing much lower levels. For instance, Sepapala at Granite Savings reported a margin of 2.9%.

A margin of at least 4% would be ideal, community bankers say. That’s where they need to be to take them back to where they were for the greater part of the 1990s. Needless to say, given the Fed’s plans, it’s hard to see how they’ll ever return to such levels.