The idea that American television viewers should be free to buy just the TV channels they want has always proven a pipe dream. It’s a silly idea, cable and satellite operators have convinced politicians and regulators: selling channels in packages funds a wider variety of programming, actually leaving consumers better off.

Regardless of the merits of that argument­ — whether cable industry pricing does more to benefit cable companies’ customers or their shareholders — it hasn’t mattered as a practical matter. Over the past decade, cable companies have always gotten their way in Washington, easily batting down laws and regulations aimed at forcing them to sell TV channels a la carte. But while the pricing system of the cable business has proven impervious to government attack, it’s starting to show signs of weakening from within. As more and more cable channels pile higher and higher fees on subscribers, the foundation of the industry’s hefty profits may fracture.

The critical question: How high can ESPN and other networks raise their “subscriber fees” before it makes sense for one major cable or satellite company to tell them to go jump in a lake? Right now ESPN charges Comcast (CMCSK), DirectTV (DTV) and their competitors about $55 per subscriber per year to carry the main ESPN channels. But the sports network, which is owned by Disney (DIS), just signed a huge new deal to carry Monday Night Football through 2021 at a staggering cost of $15 billion. There’s no question ESPN is going to use that cost to justify further hikes in its subscriber fees.

The incentive to tell ESPN to shove off may already have arrived, as the reliably blunt chief executive of Dish Network (DISH), Charlie Ergen, mused in an earnings call earlier this year: “I guess strategically, I would say it this way, if you got 3 competitors and they all have the sports programming and only 15% of the people actually watch the sports programming…to not carry sports programming would have a great strategic advantage for certain customers.”

To the cable industry, that’s heresy. The notion that dropping unwatched channels would be an advantage for customers is precisely the idea that the industry has fought for years. (According to studies the cable lobby cites, allowing subscribers to by channels individually would result in both less choice and higher prices.)

There would be obvious disruptions as angry sports lovers left a distributor who gave up sports channels, Ergen admits. Presumably sports lovers would drop the cable or satellite system quickly, while it would take longer for the non sports-lovers to realize they can save money by moving to the sports-less competitor. Says Ergen: “I think there might be some short short-term pain, but they probably do pretty well long-term.”

In the earnings call Ergen was careful to keep his discussion hypothetical, and a Dish Network spokesman is quick to emphasize that the company is not planning to drop sports from its lineup.

Dish is currently in a longterm contract with ESPN. The company won’t say exactly what it pays in subscriber fees, or when the deal expires.  When the contract does come up for renewal if Dish was to refuse to pay fees for its 14 million subscribers and drop ESPN, the company would save roughly $750 million a year.