The number of people looking to refinance their home loan has plummeted recently. According to the Mortgage Bankers Association, the number of borrowers filing refinance applications fell 15% last week. That was the third week in a row that the MBA’s refinance activity index has dropped, which is the first time that has happened all year.

The reason is obvious. Interest rates are rising. That makes it less attractive to refinance your mortgage because you’ll save less. Still, the steep drop-off in mortgage activity has caught some by surprise, especially since the MBA’s mortgage finance index just hit an all-time high in early May.

MORE: Mortgages are hedge funds investment du jour

Freddie Mac (FMCC) chief economist Frank Nothaft has been predicting a 10% drop in overall mortgage activity in 2013. The MBA’s prediction is for a 20% drop. But in the past month, refi activity, which has made up the bulk of home loans, has plunged 40%. “Look at how fast refis are dropping, and volumes will continue to fall,” says Christopher Whalen, a veteran bank analyst at housing finance firm Carrington Investment Services. “This is going to be a big story.”

That’s particularly true when it comes to the big banks’ bottom lines. Mortgage refis had been one of the few bright spots in banking these days. Low interest rates have been squeezing how much banks can make off loans. But that drop in profitability has been made up partly on volume, and nowhere has that been more true than in the mortgage business.

What’s more, banks had been benefiting from a weird quirk in the bond market. Interest rates on debt backed by Fannie Mae (FNMA) and Freddie Mac loans last year fell faster than mortgage rates. When banks sell their loans to Fannie and Freddie, as they have with most loans since the financial crisis, they make the difference between what they can lend at and what Fannie and Freddie can borrow at, minus a small fee the two giant mortgage insurers take. Last year, that spread had opened up to 2.2%, which was the largest in years. Now the spread is back down to 1.1%.

MORE: Jamie Dimon: Prepare for more volatility

For the time being, bank executives say they are not too worried. And so far investors seem to be buying that. Despite the recent rise in interest rates, shares of Wells Fargo (WFC), JPMorgan Chase (JPM) and Bank of America (BAC) are up in the past month.

At an investor conference on Wednesday, Wells CFO Tim Sloan said he doesn’t see the pool of people wanting to refinance their mortgage disappearing anytime soon. He said an improving economy and revival in the housing market means more people will be eligible for refis. Others have also pointed to the government’s modification programs as a continued source of refinance activity.

But of all the banks, Wells stands to lose the most if the refi bust continues. The bank is by far the largest financier of mortgages in the U.S., and has benefited mightily from that in the past year or so. In the first quarter, about $5.4 billion of the firm’s revenue, a quarter of its total sales, came from mortgages. Nearly half of that was from fees generated by mortgage refis, which tend to be very profitable.

Right now, the MBA’s mortgage index is 30% lower than where it averaged a year ago. If that continues, it could translate into a loss of $800 million in revenue a quarter for Wells. The bank would probably avoid much of that hit in the second quarter, since rates have only been rising in the past month, and mortgages typically take two to three months to close. Also, some of the drop in refis could be offset by more fees due to the fact that more people are buying homes and getting new mortgages.

MORE: The refinance boom has peaked

But that’s only if mortgage rates stay where they are. Many people think interest rates could continue to climb, as the Federal Reserve pulls back from its bond buying program. If the 10-year bond was to rise another 1%, not unlikely given how historically low rates have been, bank analyst Keith Murray of Nomura says the rule of thumb is that mortgage refi activity could drop by another 40%.

And yet, shares of Wells are up 5% in the past month and 17% so far this year. Yes, nothing to worry about here.