Bidding wars and record-high home prices have pushed many prospective buyers to extremes, with some ultimately regretting their new purchases or giving up on house hunting altogether.

But through it all, one specific type of homebuyer has been spared from the worst of this year’s real estate troubles: vacation home shoppers. 

During the pandemic, wealthier vacation home buyers from large cities have generally migrated to suburbs and smaller towns, where they have competed against locals with lower incomes. Because of the disparity in wealth, the locals often end up being outbid. 

For those who can afford it, buying a second home has almost become a trendy pandemic-era pastime. Demand for vacation homes surged during the pandemic once white-collar employees were allowed to work remotely. Backed by then-low interest rates and pandemic-era savings, many sought second homes in the mountains, near the seashore, or in the suburbs. 

The number of people buying second homes in locations that can more comfortably accommodate remote work was up 87% in January relative to pre-pandemic levels, eclipsing the 42% increase in demand for primary residences nationwide, according to a report in March by online real estate site Redfin. 

But now, shoppers seeking deals on vacation homes are dealing with a new challenge: borrowing rates that are rising faster than any time in history. And it already looks like they’re being scared away from real estate.

Vacation home buying cools down

Demand for second homes slipped sharply in March, the second month in a row it has done so, according to a new report by Redfin. The decline in demand suggests that even vacation home buyers are being priced out by rising mortgage rates.

“The pandemic-driven surge in sales of vacation homes is coming to an end as mortgage rates rise at their fastest pace in history, causing some second-home buyers to back off,” Redfin deputy chief economist Taylor Marr said in the report.

Mortgage rates sat at historic lows during most of the pandemic, but the Federal Reserve has since begun raising rates in an effort to combat high inflation. The average 30-year fixed mortgage interest rate is currently 5.14%, up from 3.38% a year ago. 

And it isn’t just borrowing rates. Starting April 1, the Federal Housing Finance Agency raised up-front fees for loans on second homes from 1% to 4%, significantly increasing their total cost.

The combination of higher mortgage rates and record-high home prices suggest that the rush to buy second homes is slowing, said Redfin’s Marr. He added that buying a vacation home is starting to look “more like a burden than a good investment” for many prospective buyers.

Will there be relief for other buyers?

While Redfin’s report shows that demand for second homes is softening, the prospects for buyers looking to change their primary residence, or younger buyers entering the housing market for the first time, are slightly better.

A big reason behind the strained housing market for most buyers is the low inventory of affordable homes over the past two years. In many smaller cities or rural areas where city dwellers began moving early in the COVID crisis, new arrivals during the pandemic have priced many locals out.

Home prices in seasonal towns were up 20% year-over-year in February, well above the 13% price growth in non-seasonal towns, according to Redfin. 

“People coming from New York or California had a much higher budget than locals, and were probably able to outbid them in a bidding war,” Sheharyar Bokhari, a senior economist at Redfin, told Fortune. “Prices have been growing sharply in traditional vacation home locations.”

So now that demand for second homes is dropping, does that mean that inventories will grow, and that the market for owner-occupied homes will return to more normal conditions?

Inventory may increase, but Bokhari warns that the same forces causing the demand for vacation homes to drop is pushing more buyers in general out of the market.

“Homes might be sitting on the market longer and there’ll be more homes for people to look at, but obviously not many people can buy them with these expensive interest rates, now almost above 5%,” Bokhari said. “That’s the reason homes are sitting on the market longer.”

Despite the lower demand, Redfin does not expect vacation home sales to return to pre-pandemic levels anytime soon. In fact, despite the latest decline, demand for second homes is still 13% higher than before the pandemic.

Bokhari noted that around one in four second-home buyers pay in cash, and are therefore not affected by mortgage rates or increased loan fees. That should help, to a point, with keeping up some demand amid higher mortgage rates. And as long as employers allow remote work, it’s unlikely that white-collar city dwellers will sell their country homes in large numbers.