Bernanke’s remarks roiled markets around the world.

U.S. Treasury bonds continued falling, with the 10-year yields hovering at 2.41% — the highest point in two years. Because mortgage rates tend to follow the 10-year, the borrowing costs of home loans will almost surely rise. For the first time in about a year, the 30-year mortgage rate jumped above 4% to 4.15% during the first week of June, according to the Mortgage Bankers Association.

Some might worry higher mortgage rates could derail recovery of the housing market, but that will depend how quickly rates rise. And if history tells us anything, paying more is unlikely to hurt homeowners and buyers. Here’s why:

Interest rates alone don’t drive home prices.

If the past few economic recoveries say anything about interest rates, it’s that the costs of taking out a home loan have almost no bearing on home prices.

Across 20 cities tracked by the Standard & Poor’s Case-Shiller home price index, prices in May posted the biggest gains since 2006; the index rose 10.2% during the first three months this year.

The rebound has been driven by factors beyond borrowing costs, particularly as banks have tightened lending standards. For one, job growth has helped make families more willing to buy. Even though unemployment remains relatively high, employers have added jobs for 32 months in a row.

MORE: Bernanke won’t blow up bond market

Also, the number of homes for sale remains unusually low for various reasons — there’s been little construction in the last few years; many who still owe more on their mortgages than their properties are worth have been reluctant to sell at a loss; and the number of foreclosures sold at deep discounts and short sales have declined.

True, higher interest rates do make buying a home more expensive; the rule of thumb is that for every one-percentage-point increase in mortgage rates, homebuyers pay 10% more. If interest rates rise too rapidly it could certainly slow home sales.

More broadly, though, a slower rise isn’t likely to stall the housing market’s recovery, experts say. During the last housing boom, interest rates were markedly higher than today’s rate: The 30-year mortgage hovered between 5.5% to 7% between 2005 and 2006.

Higher rates aren’t likely to stall homebuilders either. During recoveries following recessions in 1982 and 1991, interest rates were also far higher — 17.3% and 9.5%, respectively, says Patrick Newport, economist with IHS Global Insight. And yet, housing starts during both periods ran about an annual rate of 900,00 — roughly levels seen today, he adds.

It’s (mostly) a mortgage-less recovery, anyway.

Home sales have steadily risen, but investors armed with cash have played an unusually large role. They’re buying up foreclosed properties on the cheap — at times, known to compete with first-time buyers in full-on bidding wars, as The New York Times highlighted recently.

The percentage of homes bought with cash surged in many U.S. cities. Nearly a third of all homes sold in Los Angeles during the first three months this year were bought with cash, compared with a mere 7% in 2007. The same story has played out in Miami, where 65% of homes sold were cash deals, compared with 16% six years ago.

MORE: How to teach Americans to save? Treat them like children

It’s easy to see the imbalance between investors and individual buyers. Record-low interest rates have spurred a refinance boom, where borrowers re-negotiate their existing mortgage to lock in the market’s lower rates.

While higher rates have dampened the number of refis, they still make up a bulk of all mortgage applications. In May, refinance applications made up 73% of all mortgage applications, according to the Mortgage Bankers Association.

Mortgage rates are still plenty low.

Even with the recent jump, mortgage rates are still super slow — close to half-century lows, as the Wall Street Journal noted Wednesday.

How big of an additional cost you’ll incur with higher rates will largely depend on the price of your dream home, but  as the Journal has calculated, a 4.15% rate for a 30-year mortgage works out to an additional $50 a month for a $192,800 home — the median price in April of a previously owned home.

And recall, mortgage rates today are about what they were in the spring of 2012, when many thought the cost of borrowing was still amazingly low. Interest rates will likely continue rising. Whether it derails the housing market’s recovery will depend how gradually, experts say. By late 2014, the rate for a 30-year mortgage is forecast to rise up to 5.6%, says Celia Chen senior director at Moody’s Analytics.

Even then, that’s still lower than the average annual rate of 5.87% in 2005 and 6.41% in 2006 for 30-year mortgages recorded by Freddie Mac.