The recent Libor fiasco is no different. The banks’ alleged manipulation of the key benchmark interest rate may have cost municipalities, hospitals and other large non-profits as much as $600 million a year, according to one expert.

Nonetheless, because the Libor fixing mostly resulted in lower rates, a number of commentators have argued that it might have been a “victimless crime.” We’ve heard that about Wall Street’s misdeeds before.

Remember Abacus, the designed-to-fail Goldman Sachs mortgage bond. Some said Goldman wasn’t at fault because it sold the bond to a large German bank that, unlike individual investors, should have understood what it was buying. But the German bank didn’t hold onto the doomed investment. It was repackaged into seemingly safe investments that were sold to Cedar Rapids, Iowa and others.

MORE: A Barclays split would resurrect Lehman

Last year, JPMorgan Chase and UBS paid hundreds of millions of dollars to settle allegations that they colluded with other banks to inflate the fees they charge municipalities to manage their money. Rolling Stone recently wrote a story about how the court case of two key figures in that bid-rigging scandal went mostly unnoticed.

Take a closer look and it appears the real losers in the Libor trading case are cities across America. Baltimore is the lead plaintiff in a class action suit against Barclays and a number of other banks that claims the city lost money due to Libor manipulation.

Financial contracts based on Libor played a key role in the financial troubles that led to the bankruptcy of Alabama’s Jefferson County.

A recent report from a coalition of labor and community groups called the Refund Transit Coalition claims that Libor manipulation may have cost twelve transit authorities around the country, including those in New York and Boston, nearly $100 million.

Nassau County’s comptroller said the Libor manipulations might have cost his county as much as $13 million on deals related to $600 million of outstanding bonds.

Experts say that hospitals and colleges are also likely to emerge as big losers in the Libor scandal.

“Municipalities are desperate not to raise taxes,” says Robert Fuller, who advises cities on bond deals. “So when they see a way to save money they are more likely to jump at it these days. I think city financial managers were outwitted by Wall Street. Were they cheated? I don’t know.”

MORE: Libor Scandal: More Evidence for the Volcker Rule

How municipalities and other non-profits came to be the biggest Libor losers, while other borrowers many have not been harmed or even came out ahead, is a somewhat complicated financial tale. Starting in the late 1990s, Wall Street bankers began approaching strapped local government officials with a way to save money. Municipalities had long borrowed money at fixed rates, like you would for say a 30-year mortgage. Variable rate debt – the adjustable rate mortgages of the muni bond world – though, carried lower rates, typically as much as one percentage point, at least initially. So a city could save as much as $1 million a year on a $100 million bond offering if it raised money by selling a variable rate bond, rather than a fixed one. The catch of course was that rates on variable debt could go up.

Wall Street had a solution for that as well – interest rate swaps. Swaps are insurance against rising interest rates. If interest rates were to go up, the swap contract would pay out, offsetting the municipality’s rising borrowing costs. If interest rates went down, the municipality would owe money to the bank on the swap, but that extra cash would be offset by falling borrowing costs. The result was that Wall Street bankers told municipal officials that they could effectively fix their borrowing costs at the lower variable rate. Many went for it. Experts say the municipalities and other large non-profits now have as much as $300 billion in outstanding swaps contracts, up from close to nothing a decade ago.

Things, though, didn’t work out as planned. The rates on the debt were based on something called the Sifma municipal bond index, which is named after the industry group that maintains the index and tracks muni bonds. And that’s what municipalities should have bought swaps based on.

MORE: The fall of Bob Diamond

Instead, Wall Street sold municipalities Libor swaps, which were easier to trade and quickly becoming a gravy train for the banks. And normally that would have worked out just fine. Sifma and Libor had historically moved together, at least until Wall Street got into the game of manipulating Libor.

That’s when the problems started. In 2008 and 2009, Libor rates, in general, fell much faster than the Sifma rate. At times, the rates even went in different directions. During the height of the financial crisis, Sifma rates spiked. Libor rates, though, continued to drop. The result was that the cost of the swaps that municipalities had taken out jumped in price at the same time that their borrowing costs went up, which was exactly the opposite of how the swaps were supposed to work.

How much the swaps ended up costing the municipalities isn’t clear. Peter Shapiro, who is one of the country’s top financial advisers to municipalities, says that most of the suits against the banks assume that the manipulation caused Libor to be on average 0.30 percentage points lower than it should have been. Municipal swap contracts were typically based on 67% of Libor. That means the Libor manipulation caused municipalities to pay 0.20% more than they should have. With $300 billion in swaps outstanding, the manipulation could have cost municipalities as much as $600 million a year. The Baltimore suit claims the Libor manipulation dates from August 2007 to May 2010, but the settlement between Barclays and regulators says that bank was submitting false rates as far back as 2005.

What’s more, it appears banks, even as they were losing money on mortgage bonds during the financial crisis, positioned themselves to benefit from the falling Libor rates. A 2010 study of Libor rates conducted by professors at the University of California and the University of Minnesota, found that Citigroup’s interest rate revenue, which includes money the bank made on lending as well as swaps, more than doubled in the first three months of 2009. Revenue at JPMorgan and Bank of America jumped in the same time as well.

“On the trading floors of Wall Street it was very much understood that these swap deals carried additional risks,” says Robert Brooks, a finance professor at the University of Alabama who has studied derivative contracts. “Clearly, that was not understood in halls of municipal offices around the country.”