Sure, the IRS isn’t the easiest foe for an American citizen to evade, but roughly half of the taxes most of us pay are levied by state and local governments, and because U.S. citizens can freely move among these jurisdictions, these governments are forced to compete with each other to be appealing places for businesses and wealthy individuals to set up shop.

This dynamic is in the headlines again this tax season, after news broke that hedge fund titan David Tepper of Appaloosa Management was moving his billions in income from high-tax New Jersey to Florida, which has no income tax at all. And the move has New Jersey officials worried. Frank Haines, a budget and finance officer with the Office of Legislative Services told a State Senate committee earlier this month, “We may be facing an unusual degree of income-tax forecast risk,” this year, citing Tepper’s planned move as prime cause for concern.

How much Tepper’s move to Florida would cost New Jersey is tough to say for sure. According to Haines, personal income tax comprises roughly 40% of New Jersey’s tax base, and one-third of that revenue is brought in from less than 1% of taxpayers. And Tepper could well be the highest-earning New Jersian, as Forbes estimates that he brought in $1.2 billion in income in 2015. Because of charitable deductions and other account maneuvers, Tepper likely wouldn’t have paid the top New Jersey state income tax rate of 9% on all that income—which would amount to more than $100 million—but it’s safe to say that the move could cost the New Jersey State government tens of millions of dollars over the course of the next several years.

Critics on the left and the right have used New Jersey’s conundrum to make their favorite political points. On the right, you have those arguing that Tepper’s decision is proof that New Jersey’s tax regime is onerous and forcing the most talented folks to leave the state. Those on the left are pointing to Tepper’s move as another attempt by the wealthy to shirk paying their fair share to society.

But whatever opinion you have on whether the wealthy pay too much in tax or not enough, New Jersey’s tax problem is just a more extreme version of what many state and local governments will have to contend with as growing income inequality slows economic growth and shrinks tax bases. A 2014 report from Standard & Poor’s, which closely tracks and rates the creditworthiness of state and local governments, argued that growing income inequality will wreck havoc on state and local revenue in years to come. It found:

• From 1980 to 2011, average annual state tax revenue growth fell to 5% from 10%; meanwhile, the share of total income for the top 1% of earners doubled;

• State tax revenue trends have also become more volatile as progressive tax states have come to rely more heavily on capital gains from top earners.

• Regardless of a state’s tax structure—be it income-tax or sales-tax reliant—the pace of revenue growth is declining across the spectrum.

• Reasons for rising income inequality aside, the disparity is contributing to weaker tax revenue growth by weakening the rate of overall economic expansion.

• It’s unlikely that states can fully correct for both slower and more volatile tax revenue growth by adjusting their tax policies.

This trend will not make life easier for either side of the debate over taxes. Those who think the rich are already taxed too much must confront the fact that the wealthy are the only group in the country who are consistently seeing their income rise. Those who want to tax the rich more will have to deal with wealthy’s ability to flee liberal, wealthy states like New Jersey for places with a smaller government ethos.

Meanwhile, states and local governments will have to respond by cutting services. We’ve already seen this happen to a large degree in the realm of higher education, as states have severely cut their funding of colleges and universities. This has only shifted the debate over college costs to the federal stage, where national Democratic figures like Bernie Sanders are calling for increased taxes and spending on the federal level to make up for this shortfall.

In other words, rising income inequality has made tax competition between states more fierce, but it hasn’t exactly taken the wealthy off the hook in terms of what society expects they contribute.