So far the answer from the bond market has been quite a lot. The yield on 10-year Treasuries spiked to 2.3% Wednesday after the Federal Reserve chairman Ben Bernanke indicated that, yes, the bond stimulus will come to an end. Not immediately, or all at once. But Bernanke said if the economy remains on its current path he expects the Fed to curtail bond purchases later this year, and stop completely by mid-2014 if the unemployment rate hits 7%.

With Wednesday’s move, bond yields have risen 0.8 percentage points in about a month. That might not sound like a lot, but the move translated into a 8% drop in prices for the 10-year. Bond investors aren’t used to those kind of losses.

MORE: Why Bernanke can’t win

Some, though, are hunkering down for much worse. A stream of boldface names, including Warren Buffett, Jamie Dimon, and Goldman Sachs’ (GS) CEO Lloyd Blankfein and COO Gary Cohn, have told us to watch out for rising interest rates. Most of them have focused on what will happen when the Fed stops buying. With that much buildup, it’s no surprise that investors have rushed for the gates now that Bernanke has signaled the final countdown. The problem is there’s no real evidence the Fed’s moves will blow up the bond market, at least not if investors keep their heads.

You can blame the Fed itself for some of the frenzy. Last year, the Fed said it wouldn’t raise short-term interest rates, which are near zero, until the unemployment rate hits 6.5%. There is no target for when the Fed will end its quantitative easing program. That’s created some confusion in the bond market and caused interest rates to move up faster than if the Fed had laid out when they would stop buying bonds and how much at a time. The Fed came closer Wednesday for setting down some guidance as to when QE would end, but Bernanke qualified his 7% target, saying the Fed would adjust its view on the program based on the economic outlook. “Not having consistent guidance has been a mistake,” says Vincent Reinhart, a Morgan Stanley strategist and former top Fed economist.

MORE: Bond investors have already been pulling out before the Fed meeting

But what is also true is that stream of debt doom worriers (which I have to say at times has included me) has made the Fed and its buying seem more important to the bond market than it may actually be. The Fed owns just under $2 trillion in Treasury bonds. That’s less than 20% of the nearly $16 trillion in U.S. debt.

Still, much of that is not traded regularly. Banks, sovereign wealth funds, and other large investors own a similarly big amount of U.S. bonds as Bernanke & Co. And they aren’t likely to sell even if prices drop. The Fed, too, says it has no plans to sell off its own holdings.

MORE: Don’t believe Alan Greenspan’s bullish case for stocks

Currently, the Fed is adding $85 billion a month to its bond portfolio. Of that, slightly more than half, $45 billion, in going into Treasuries. The rest is going into mortgage bonds.

How does that compare to what’s being sold? In May, Uncle Sam issued $184 billion in debt that won’t come due for a year or more. In April, the Treasury sold $282 billion in similar debt. So the Fed is not exactly cornering the market with its bond purchases. And most Treasury bond auctions continue to be oversubscribed.

“There are other natural buyers of U.S. government debt that will step in that have been crowded out by the Fed,” says Shyam Rajan, a U.S. rate strategist at Bank of America Merrill Lynch.

Yes, retail investors have been pulling their money out of bond funds. But those mutual funds are a much smaller portion of the market than they were back in the mid-1990s. What’s more, the amount of new debt is likely to shrink in the coming months. The CBO and others recently predicted that an improving economy and higher tax revenue could cause the U.S. deficit to fall to around $650 billion this fiscal year, down from $1.1 trillion. If that’s true, the Fed could theoretically cut its bond purchases back and still have the same impact that it is currently having by buying $45 billion a month.

“With the government dramatically reducing amount of issuance, the reduced buying from the Fed could still have the power to keep rates low,” says Scott Colyer of Advisors Asset Management.

When you look at the daily trading volume for the bond market, the Fed’s importance is even smaller. An estimated $350 billion of Treasury bonds are bought and sold each day. Spread the Fed’s $45 billion in purchases over the course of the month, and that works out to just 0.4% of overall Treasury bond market activity on a daily basis.

And that’s if the Fed were to stop all of its purchases at once, which it isn’t likely to do. More likely, the Fed would cut back bond purchases gradually, $5 to $10 billion at a time. That pullback could be spread between the Treasury and mortgage markets.

“My anticipation is the that the end of QE will be like going to the dentist,” says long-time fed watcher Bob Eisenbeis of Cumberland Advisors. “The anticipation is worse than that actual visit.”