“Low interest rates could become a source of instability down the road,” said Summers, a prominent economist and former top advisor to President Obama. Summers was speaking on Thursday to a packed auditorium of hedge fund industry professionals at SALT, the annual industry confab being held at the ritzy Bellagio hotel and casino in Las Vegas.

Summers told the crowd that he shared concerns that low interest rates could be causing new bubbles. That’s why, Summers said, he has long favored government spending on jobs programs rather than stimulus engineered by the Fed.

What’s more, Summers said that the Fed’s policies are likely making the income inequality problem in the U.S. worse, by helping wealthy Americans who hold the majority of stocks, more than the rest of the country. “A policy that works by pumping up asset prices is not going to be egalitarian,” said Summers.

MORE: The Fortune Interview: Larry Summers

Summers recently called Thomas Piketty’s recent book on income inequality Capital in the 21st Century a “profoundly important contribution,” but said he thought Piketty’s solution to it – more progressive taxes – would be hard to pull off.

Last year, Summers was in a two-way race, with Janet Yellen, to be the next head of the Federal Reserve. Summers eventually bowed out. At the time many said his view on Fed policy was very similar with Yellen, who has been a chief architect of the Fed’s plan to keep interest rates as low as possible for longer than normal.

Yellen ended up getting the top job at the Fed, and has since been cutting back the U.S. central bank’s so-called QE bond buying program. That could cause some long-term interest rates to go up. But it still could be a year or more before the Fed decides to raise the short-term interest rates. That could leave enough time for Summers’ worries to materialize.

Author and economist Nassim Taleb, who was also on stage with Summers, said he was concerned that we could have another financial crisis when interest rates rise. Summers countered by saying that was unlikely. Banks are in much better shape than they were a few years ago, he explained.

“The major financial institutions are much better capitalized than in the past,” says Summers. “Will they always be? It’s a good thing to question.”

MORE: Jeremy Grantham: The Fed is killing the recovery

Taleb said he thought the financial system had not really been fixed since the financial crisis, and that we were setting ourselves up for more, and larger bailouts. Summers said the answer was more capital and closer monitoring of the banks, and not some alternative.

“In your plan we have a system with two types of financial firms. One would be like a utility that can fail and be bailed out, but the government gets to set those banks’ salaries and basically say what they can do,” said Summers. “Then there’s the let’s call it the ‘let it rip’ part of the financial system, and hope it works out.”

Summers said he thought the first group of firms, the ones that chose government control, would be small. And the second group would end up being so risky that we would end up having to have bailouts anyway.

“I’m for punishment,” said Taleb. Oddly, that comment drew wide applause from the crowd largely made up of Wall Streeters.