Hedge fund manager David Einhorn has built a reputation on his ability to crater a company’s stock.

His prescient short calls such as Allied Capital and Lehman Brothers had been announced in previous years at the annual Sohn Investment Conference in New York. And so, as he took the took the stage at the event after the U.S. stock market closed on May 4, audience members were eagerly anticipating what would come next.

In a 15-minute presentation, he laid out why industrial company Caterpillar Inc should trade at half its current value. The next day, the stock opened lower, but was down just 1 percent.

After a decade in which hedge fund assets nearly doubled, there are several signs that the golden era for the $2.9 trillion industry may be on the wane, draining the influence of fund managers who once considered themselves masters of the universe.

Once able to command hefty fees by routinely beating the Street, hedge funds are now facing a storm of unsatisfied clients who are demanding either steep discounts or withdrawing their funds entirely, leaving some in the industry wondering whether the pain is just beginning.

The same day that Einhorn spoke in New York, the chief investment officer for the California State Teachers Retirement System called the typical hedge fund fee model “broken.” A few weeks earlier, the New York City Employee’s Retirement Fund announced that it was pulling out of hedges completely.

Einhorn is a prime example of the industry’s troubles. Last year, his Greenlight Capital lost 20 percent, marking the fund’s first down year since the 2007-09 financial crisis and only its second-ever annual loss since its inception in 1996.

Greenlight investors have expressed their disappointment by pulling money, helping shrink assets to roughly $9 billion, down some $3 billion from only eight months ago.

Einhorn, who declined to comment for this story, acknowledged the tough new environment when he wrote to investors in January “we have never had a year where so little went right,” and pledged the firm would “concentrate on trying to make better returns.”

In part, the industry is a victim of its own popularity.

With an estimated 10,046 global hedge funds and funds of hedge funds according to Hedge Fund Research Inc (HFR) – near the record high in 2014 – investors and analysts say that many managers are chasing the same ideas, driving spreads downward and lowering potential returns.

At the same time, extraordinarily low global interest rates are keeping a lid on returns from arbitrage strategies that attempt to exploit differences between global economic policies, for example.

Daniel Loeb, who runs $17.5 billion Third Point, said pointedly in a recent letter to investors “as most investors have been caught offsides at some or multiple points over the past eight months, the impulse to do little is understandable.”

Last year the average hedge fund lost slightly more than 1 percent, according to HFR, but with billionaire stars like Larry Robbins, the founder of Glenview Capital Management, and William Ackman, the founder of Pershing Square Capital Management, nursing double-digit losses.

So far this year the average manager – across all strategies – is again down nearly 1 percent, while the broader U.S. stock market is up, albeit slightly.

As a result, hedge funds are under increasing pressure to lower their typical fee structure, which charges 2 percent of assets invested plus 20 percent of gains, or face investors pulling assets in favor of other illiquid investments ranging from private equity to real estate, or even low-cost market index funds.

Incentive fees – the amount of investment gains that a hedge fund manager can take of client gains – fell by an average of 5.3 percent between the end of 2011 and year-end 2015, according to HFR, and some wealth advisors say they expect it to fall further.

“When you have too much capital chasing the same ideas and deals, it’s going to narrow the return,” said Darell Krasnoff, senior managing director at Bel Air Investment Advisors, who has halved his allocation to hedge funds over the last five years and moved money into commercial real estate instead.

“There’s not that many people who can justify 2 and 20 (percent) out there,” he said.

Overall, investors pulled $15.1 billion out of hedge funds in the first three months of this year, according to data tracker HFR, marking the first quarters of consecutive outflows since the financial crisis.

To be sure, institutional investors such as pensions and endowments have few safe places to put their billions of dollars with stocks at relatively expensive levels and many bonds yielding little to nothing. And recent surveys of large investors showed that many plan to increase their allocation to hedge funds.

The average allocation among U.S. public pension funds to hedge funds is up slightly this year, to 9.2 percent of assets from 9 percent the year before.

Yet public criticism has grown more common, even from within the financial industry.

A day before Einhorn’s speech, commercial insurer AIG Inc blamed poor hedge fund performance for its lousy quarterly earnings results and said it would be pulling $4.1 billion it has invested in the industry.

The day before that, billionaire investor Steven Cohen, who now manages $11 billion for so-called family office Point72, told an audience at the Milken Institute Global Conference in Los Angeles that the days of funds routinely generating 30 percent annual returns were over.

“Frankly, I’m blown away by the lack of talent,” he said.