Famed investor Warren Buffett, in a New York Times opinion piece and elsewhere, has been making the case that tax rates, which are set to go up on capital gains and dividends on January 1st, won’t affect the market. He says the ultra-rich forever pursue the best investment opportunities, no matter what the tax rate is. So raising rates wouldn’t cause people like him, or even those slightly less wealthy, to invest any less.

Many economists, though not all, seem to disagree. In general, economists believe investors are willing to pay more for investments with higher returns, and less for lower returning ones. Higher taxes would lead to lower after-tax investment returns. So all else being equal, you would expect investors would pay less for stocks if taxes go up, because after-tax returns will go down.

“I don’t see how tax rates could not be relevant,” says Daniel Feenberg, who is a researcher at the National Bureau of Economic Research and co-authored a study on capital gains taxes with Larry Summers, a former key economic advisory to President Obama. “Why did Buffett spend his career looking for good investments if investors don’t care about their returns?”

MORE: How do you spell relief on Wall Street? SEC gridlock

Even Ben Graham, another investing legend and Buffett mentor, seemed to suggest that higher capital gains taxes would cause stock prices to fall. He once testified in front of Congress that he liked the capital gains tax because it discouraged speculation. Graham, who viewed himself as an investor, not a speculator, also thought it allowed people like him to buy stocks at lower prices, because it kept others out of the market.

Buffett’s response is that in the real world that’s not how things work. On Monday night, I ran into Buffett at a cocktail party for the release of a new book on the investing legend edited by Carol Loomis, which includes pieces written by her and other members of the staff of Fortune (See The wit and wisdom of Warren Buffett). Buffett had just knocked over a plate of appetizers that a waiter was bringing around, creating a break in his conversation that allowed me to swoop in with my question about whether investors would really be discouraged to invest if tax rates are higher. “What are the alternatives?” Buffett asked. “Where are they going to go?”

Buffett’s point is this: Investors can’t get the absolute highest return. They can only get the highest return of the options that are offered to them. And even with higher capital gains taxes, stocks, at a time when most bonds are yielding less than 3%, are still the best option investors have.

MORE: Wall Street isn’t backing Jack Lew for Treasury

Recent behavioral economics studies seem to back up Buffett’s notion that taxes may not matter when it comes to the way people invest. Leonard Burman, an economics professor at Syracuse University and fellow at the Tax Policy Institute, has found that people in places like New York, where capital gains are taxed at the state level, invest no differently than those in places like Florida, which has no state capital gains tax. Older people, too, don’t seem less likely to avoid selling their investments that generate the most capital gains, even though their heirs would not have to pay those taxes when they die. “When you look at the data there doesn’t seem to be a whole lot of people who are tax sensitive,” says Burman.

What’s more, large pension funds and 401(k) retirement accounts are exempt from capital gains taxes. So there will be at least some investors who won’t be affected by an increase in the capital gains rate.

The problem with Buffett’s argument is that it is of the “this time is different” variety. And it’s not one to last for long. Eventually, higher interest rates could tempt people to leave the stock market for bonds, savings accounts or other investments, which will have higher yields. The fact that individuals will have to pay higher taxes as well on capital gains will likely hasten that march out of the market.

What’s more, on average, it does appear that investors pay less for stocks in times when capital gains taxes are high, and vise versa. Over the past 100 years, the S&P 500 has had an average price/earnings ratio of 14 in times when the capital gains rate is greater than 24%, which is about what it could max out at if fiscal cliff tax changes go into effect. In years in which the cap gains tax has been less than 24%, the S&P 500’s p/e has averaged nearly 19.

Of course, that could all be a coincidence. In the end, it’s hard to say whether taxes matter that much to stock market returns. They probably do, but not as much as other things, like the state of the economy or interest rates. Burman says he believes it matters, but the amount might be small.

What you can say is this: Currently, the stocks in S&P 500 have an average p/e of 16.5. In part because of that, a lot of people, including Buffett, have made the argument in the past year that now is a good time to buy. A higher capital gains rate is one more reason to believe stocks are not as cheap as they appear.