It was assumed that if Libor rates had been pushed lower, as appeared to be the case, borrowers would have benefited. Even some municipal finance chiefs who had likely lost money on interest rate swap contracts said they weren’t sure they were net Libor losers. Lower rates may have saved them money elsewhere. If you can’t identify a victim, is it a crime?

The new government charges show that traders regularly pushed lending rates up as well as down. In an electronic chat with a fellow trader, Thomas Hayes, one of the UBS (UBS) traders charged by the Justice Department with fraud, wrote in early February 2007, “3m [three-month] libor is too high because I have kept it too high.”

MORE: Four fixes to the libor scandal

Some have argued that the Libor scandal had nothing to do with the financial crisis. That now seems less true as well. The housing market crashed in part because people could no long pay their mortgages. If Libor manipulation added to borrowers’ hardships, then it was a factor too. The Libor scandal proves once again that banks sold risky products, in this case mortgages tied to Libor, and then gamed the system in order to make sure they would profit. Isn’t that precisely what’s at the heart of what caused the financial crisis?

The other thing we got from the Justice Department’s Libor charges, which has been sorely missed in the financial crisis investigations, were some actual villains. Hayes ran a global rate-rigging crew, driven by bribes and promises of profits. Along with Hayes, the U.S. charged former UBS trader Roger Darin with criminal conspiracy. Hayes was also charged with wire fraud. As with the charges against Barclays (BCS), it’s amazing once again just how many people knew Libor was being manipulated and were involved. The U.K.’s Financial Services Authority, which also charged UBS with fraud, said that more than 40 individuals at UBS were involved, including managers and senior managers.

Hayes apparently got traders at other firms to go along, according to the Justice Department. In at least one instance, Hayes apparently got individuals at so-called inter-dealer brokers, which execute transactions between banks, to display fake quotes in order to trick other traders into moving the market in the direction he wanted.

MORE: Citigroup may have been the biggest Libor liar

Still, even knowing that borrowing rates were pushed up as well as down and that this was a vast conspiracy, it’s hard to determine who was hurt and by how much. But that makes it more commendable that regulators pushed forward with their investigation and brought charges related to Libor manipulation. Financial markets are more diffuse than ever. And market manipulation is all about figuring out how to take as little as possible from as many people as possible in order to not get detected. When we are all equal losers, it’s hard to see it.

So regulators were smart to focus on the winners and know that the losers were surely there as well. Hayes wrote to one of the other traders involved in his scheme, “Think of me when you are on your yacht in Monaco.” Hopefully, many Wall Streeters are thinking of Hayes right now.