A few years ago, houses were suddenly one of Wall Street’s hottest investment ideas. A number of funds popped up to snatch up residential real estate in the wake of the foreclosure crisis. Some of the funds were run by investors who had bet on a housing bust just a few years earlier. Others were run by some well known real estate investors, like Barry Sternlicht, who created the W hotels (HOT), and Tom Barrack, the California real estate developer who once owned Michael Jackson’s Neverland Ranch. Private equity giant Blackstone Group (BX) also got in on the action, ultimately spending more than $9 billion on 47,000 homes.

But now, there seems to be a reasonable chance that Wall Street’s housing bet may go bust.

Earlier this week, two of the bigger residential real estate investment funds, Sternlicht’s Starwood Waypoint Residential Trust (SWAY) and Barrack’s Colony American Homes, announced they were merging, in part because neither of them were big enough to make it on their own. The terms of the deal should give other housing market investors pause. The merger values Colony at $1.5 billion, or about 50% of what it paid for its 18,000 homes and other assets, minus debt.

Indeed, average investors have not turned out to be as excited about owning collections of homes as these Wall Street players thought they would. Shares of Starwood Waypoint have fallen 3.5% in the past year. The stock of American Homes 4 Rent (AMH), another large publicly traded residential housing REIT, is down 5% this year, and is only slightly above where it IPOed at more than two years ago.

From the beginning, a number of housing experts suggested that maintaining and collecting rent on tens of thousands of homes would be a logistical nightmare. Real estate legend Sam Zell warned against the strategy and said he was sticking to apartment buildings. Starwood Waypoint appears to proving that caveat out. According to its latest filing with the Securities and Exchange Commission, the company has spent about $2.2 billion on houses. Yet, its roughly 12,000 homes appear to be bringing in just $67.5 million in annual rent. That’s a paltry return on investment of just 3% a year. Investors would do better in a corporate bond. Throw in interest and depreciation expenses, and Starwood is on track to lose nearly $13 million a year.

Wall Street’s bet on housing could still pay off. Rents across the U.S. in August rose 3% in the past year, according to the Labor Department, about three times as fast as other prices. And given how low interest rates are currently, the REITs don’t have a long way to go before they are generating an attractive return. But apartment developers see this too, and they are building as well. That could put downward pressure on rents. And as banks loosen up mortgage requirements, more people may be able to buy instead of rent.

All this may make it tough for Wall Street investors and firms who put money into residential real estate to cash out. Blackstone has put its housing investments into a new company called Invitation Homes and is eyeing an IPO, but it reportedly had put off the deal until the market for housing REITs improves. It could be waiting a while. As anyone who has bought a fixer-upper knows, getting in is a lot easier than getting out.