Update 11:00 pm
FORTUNE — In life there are few coincidences, and this one probably isn’t either: The day Knight Capital Group’s computers nearly blew up the market and lost the firm $440 million in 45 minutes is the same day that the New York Stock Exchange (NYX) launched a new trading system that was, in part, meant to take business away from Knight (KCG).
For the past half decade or so, there has been a tug of war over who completes the buy and sell orders for stocks that average investors like you and I make. It used to happen in the pits of the NYSE. These days, almost none of the trades that folks like you and I make ever get to the exchange. Instead, they get cut off, diverted into the computer systems of Knight or its main competitors Citadel, Citigroup and UBS, which match those with the millions of other orders they collect.
MORE: Should we listen to a bond king trying to time stocks?
And the pace at which these firms have been able to divert traffic from the NYSE has been accelerating. In 2009, about 15% of all trades took place away from the NYSE. Now about a third of all the trades in NYSE-listed shares happen elsewhere.
It’s not clear why this battle over individual stock trades is so pitched. Knight pays brokers for its so-called order flow. And it guarantees that individuals get a slightly better price than what they would get at the exchange. Those stock trades get fed into Knight’s computers, which use lightning fast trading algorithms to figure out how to make money off the orders the firm has just paid up for. This is, in part, the high frequency trading that you have heard about.
Some say that market makers provide a service. Others say Knight and others seek out the orders of individual investors because they view those orders as so-called dumb flow and easier to trade against. What is clear is that Knight and others have figured out how to make money off the stock trades of you and me in ways that we can’t detect but we probably pay for somehow. Eric Scott Hunsader, who runs trading research firm Nanex, estimates market makers have been able to generate $5 billion in profits rapidly trading the orders of individual investors and others in the past seven years.
MORE: Wall Street’s hottest investment idea: Your house
On Wednesday, the same day that Knight lost $440 million, the NYSE launched its own computer driven trading system, called the Retail Liquidity Program, that the exchange hopes will reclaim some of the trading volume it has lost to market makers. NYSE hopes RLP will create more competition among traders and brokers and market makers so that more of those orders get filled at better prices on the exchange. The new system also offers financial incentives for brokers to complete their orders on the exchange, similar to the payments long made by Knight and others that lured trades away from NYSE.
Knight says the computer problems it ran into had to do with NYSE’s new trading system, but it didn’t say what. Tellingly, all of the stocks that Knight’s computers did bogus trades in were listed on the NYSE. It’s likely that Knight tried to upgrade its own algorithm to allow its computers to do an end around the NYSE’s new system. But it messed up somehow. Instead, Knight’s computer system, launched on the same day as the NYSE’s, went on a trading frenzy, buying and selling millions of shares on its own shortly after both systems were switched on when the market opened at 9:30 Wednesday morning.
Normally that shouldn’t have produced any real losses. These weren’t actual orders, so Knight’s system should have just been buying and selling to itself. But that’s not how the world of high frequency trading works. When other traders, i.e. computer systems, saw the spike in activity, they jumped in too.
MORE: NYSE: Hey stock picking on us
Knight disabled the faulty algorithm by 10:15. But by then the damage was done. Knight was out $440 million. Dozens of stocks, including Warren Buffett’s Berkshire Hathaway (BRKB), had gyrated up and down, and our faith in the market was shaken once again.
In theory, we should all benefit from this competition, being able to trade at cheaper and cheaper prices. But in practice the “price improvements” that Knight and now NYSE offer are fractions of a fraction of a penny. At best, what we are getting in return is a market that is less stable. At worst, we are getting a system that is picking our pockets.
If this isn’t a clear case where we need regulators to step in, I don’t know what is.
Update: The original version of this story said the NYSE’s computers provide price improvements. In fact the NYSE’s new platform hopes to provide better pricing through increased competition.
