What you may not realize, however, is that portfolio managers often outsource the voting decision one step further, to a third-party firm that holds zero fiduciary responsibility. That so-called proxy adviser is likely to be either Institutional Shareholder Services, whose parent company, MSCI, creates the indexes underlying many ETFs, or San Francisco-based Glass Lewis, owned by the $130 billion Ontario Teachers Pension Plan. Together the two firms control an estimated 97% of the global proxy advice market and have the ears of institutional investors holding more than $30 trillion in assets.

But there is a growing backlash against this duopoly. Companies such as J.P. Morgan Chase, Pfizer, and Walt Disney have recently begun to challenge the proxy advisers, complaining that their policies are often not in the best interest of shareholders. And many institutional investors have recently taken pains to assert that they do not blindly follow the firms’ recommendations.

In December the Securities and Exchange Commission held a hearing to examine the influence of proxy advisers. Research shows that the firms’ voting recommendations — for or against a proposal — can swing up to 30% of votes cast. That sway has put companies, and the directors who sit on their boards, increasingly at the mercy of the proxy advisers — especially because ISS recommends firing directors if they don’t implement proposals that won a majority of votes. “The two firms are effectively creating de facto corporate governance standards in America,” says former SEC chairman Harvey Pitt. “It’s hard to tell on whose behalf they are making these recommendations or what the benefit is.”

Ironically, it was the SEC itself, under Pitt’s leadership, that inadvertently fueled the rise of ISS and Glass Lewis. In 2003 the agency created Advisers Act rule 206(4)-6, requiring investment advisers to disclose their proxy voting policies and votes to clients. The regulation was designed to prevent investment firms’ conflicts of interest from interfering with their fiduciary obligations. Practically speaking, though, it thrust onerous new levels of paperwork onto institutional investors. Enter ISS, which had been in the proxy advice business since 1985, and Glass Lewis, which formed in 2003.

Today ISS and Glass Lewis sell proxy reports for more than 37,000 shareholder meetings to thousands of institutional investors, who pay up to $1 million a year to subscribe (depending on their holdings). Their off-the-rack voting policies come with the SEC’s blessing in the form of a “no action” letter, promising to absolve money managers of any conflicts of interest. With the volume of proxy proposals swelling in recent years — the total for S&P 500 companies increased 28% from 2009 to 2013 — many institutional investors say the proxy services would be too expensive and complicated to replicate in-house. “You hire an outside service so you don’t miss any proxies you have to vote,” says Don Yacktman, the renowned manager of the $13.9 billion Yacktman Fund and a subscriber to Glass Lewis.

Over the years, though, Yacktman and others have grown skeptical of the proxy advisers’ recommendations. The firms fail, critics say, to understand the nuances of companies and instead apply the same set of governance guidelines to all of them. “One of the problems that ISS and Glass Lewis end up with is they tend to be one-size-fits-all, and there are times when one size doesn’t fit all,” says Yacktman. He remembers when, in 2004, the proxy firms recommended withholding votes for the reelection of Berkshire Hathaway CEO Warren Buffett to the board of Coca-Cola because Berkshire subsidiaries had deals with Coke, creating a theoretical conflict. “To us that was laughable — to think that Warren Buffett wouldn’t be positive on a board of directors,” says Yacktman.

Academics have begun examining whether the proxy firms’ advice benefits shareholders — and there’s evidence that it actually hurts. For example, a recent study at Penn State’s Smeal College of Business looked at the proxy voting records and market performance of some 2,200 mutual funds over a five-year period. Funds that voted with ISS virtually 100% of the time underperformed funds that differed with ISS most often by 4.3 percentage points over that span.

ISS and Glass Lewis shrug off such findings, saying that their job is to reduce long-term risk for shareholders by improving corporate governance, and they point to the stellar credentials of the analysts making their voting recommendations. Glass Lewis’s team, for instance, includes a former U.S. comptroller general. But interestingly, the firms doubt that the economic impact of their recommendations can even be measured: “It would be an impossible undertaking,” says Glass Lewis CEO Katherine Rabin.

Vanguard’s head of proxy voting, Glenn Booraem, says that he frequently diverges from proxy advisers’ recommendations. Boards know best, he says, and should disregard inappropriate proposals. “At the end of the day, we aren’t advocating for governance for governance’s sake,” he says, “but for long-term value for our fund shareholders.” Those shareholders should pay more attention to their proxies.

This story is from the February 24, 2014 issue of Fortune.