The bigger shock in the letter, though, came when the bank informed Lehagre he could be a “potential tax evader,” because he had never filed income tax returns with the Internal Revenue Service. Lehagre had been born in the U.S. to an American mother and French father, but had not lived there since he was 18 months old. He does not speak English (he spoke to Fortune via ChatGPT translation) and has never worked in the U.S.

Lehagre, now 39, is a so-called accidental American, or someone born in the U.S. who has not spent much, if any, time there. These accidental Americans may have no connection to the country, but are still on the hook to file and pay U.S. taxes.

Soon after finding out about his unwanted dual citizenship, Lehagre founded the Association of Accidental Americans, which lobbies for an end to citizenship-based taxation in favor of taxation based solely on residency. He would renounce his U.S. citizenship, but after years of stress and campaigning, he refuses to pay the fee to do so. “There’s no question of me paying $2,350,” Lehagre says.

Every year, thousands of Americans—accidental and otherwise—decide they’ve had enough. Tired of what they view as greedy U.S. tax collectors and a meddlesome bureaucracy, they take the most drastic of steps—tearing up their passport and renouncing their U.S. citizenship altogether.

Exactly how many do so is difficult to quantify. The IRS publishes a quarterly list of Americans who have expatriated, but it is not up to date, and many of the people whose names were published this year actually renounced their citizenship more than a year ago. Still, more have been doing so in recent years. According to a 2020 survey of government data, 5,816 Americans renounced their citizenship in the first half of that year, a more than 10-fold increase from the last six months of 2019. The IRS did not immediately respond to a request for updated figures.

Those who try to renounce their citizenship quickly discover the process is harder—and more expensive—than it looks. It is often a yearslong process that could drain thousands of dollars, or more, depending on the individual’s situation. And the ex-American could also face unintended consequences, lawyers say—like not being allowed to even visit the U.S. ever again.

The ultrawealthy on the move

Americans renounce their citizenship for any number of reasons. Some, like Lehagre, have no actual ties to the U.S. beyond their parents. Others do so for political reasons. But there is an increasing cohort of the ultrawealthy who have been searching for greener pastures abroad because of what they consider an overly onerous U.S. tax code, immigration experts say.

Still, renouncing citizenship is very rare; the ultrawealthy are more likely to acquire second citizenships or residencies in places like Portugal or Malta than give up their American passports completely.

“It is such a significant move, and once you do that you can’t walk it back,” says Peter Roskam, federal policy team leader at BakerHostetler and a former Republican congressman. “It’s not like you can renounce your citizenship for this year. It’s gone. In a world that is less and less stable, having that blue passport is a pretty wonderful thing.”

But it does happen, says David Lesperance, an attorney who helps wealthy Americans give up their U.S. tax status. Lesperance previously told Fortune that many of the richest citizens are seeking a “backup plan” owing to the perception of increasing political and social instability in the U.S. Some fear the rise of anti-Semitic, anti-LGBTQ, and anti-minority sentiment in some parts of the country.

Others do so as a reaction to U.S. “double taxation,” or the fact that the U.S. is one of the only countries that taxes citizens no matter where they live in the world. Though few Americans living abroad actually pay double the taxes after credits and exclusions (though some do), the hassle—and the high filing fees—are enough to catalyze some into renouncing their citizenship. Others may also be trying to get ahead of other tax and estate changes they fear will be pursued by Democratic presidential administrations.

“While tax benefits are not the biggest driver of interest, these benefits do play a role, and the United States’ double taxation policy adds an extra layer of complexity for U.S. nationals,” says Armand Arton, president of Arton Capital, a global citizenship financial advisory services firm. For many Americans living abroad, “the United States will always be home, [but] a double taxation policy pushes many to consider renouncing their American citizenship in favor of better deals found abroad.”

More likely, though, is the renunciation by accidental Americans like Lehagre, says Max Reed, a cross-border tax lawyer based in Canada.

“It’s not like there’s this tidal wave of people moving out of the U.S. and giving up their citizenship,” says Reed. “The vast, vast majority of those are long-standing non-U.S. residents.”

One of the big drivers of these Americans surrendering their passports since 2010 was the implementation of the Foreign Account Tax Compliance Act, or FATCA. This federal law requires not only all foreign financial institutions to report customers with a connection to the U.S. and their assets to the U.S. Treasury Department, it also requires U.S. citizens and green card holders to report foreign financial assets each year to the Internal Revenue Service.

The law was meant to crack down on tax evasion, but many Americans living abroad have found the compliance burden to be untenable. It also led many foreign banks to decline services like mortgages and basic bank accounts to Americans because they don’t want the reporting headaches.

How to renounce citizenship

To voluntarily give up American citizenship, Americans must first secure citizenship in another country. There are a number of ways to do this depending on the country, including through birthright, family ties, or marriage; for the wealthy, there are citizenship by investment programs that make it much easier, assuming they have the funds. Some countries offer other avenues, like buying a home or through work.

Then, the soon-to-be-former American will head to a U.S. embassy to take the oath of relinquishment. There, she will also need to pay all past due taxes, surrender her passport, and pay the $2,350 fee (the State Department recently announced that will drop to $450). Though it seems fairly straightforward, the whole process can take years to complete.

It gets trickier for wealthy Americans. The U.S. may impose an exit tax if your net worth exceeds $2 million or your average annual income exceeds a certain threshold. The exit tax is on worldwide assets, and is levied to ensure those soon-to-be-ex-Americans have fulfilled all of their tax obligations, including on income and capital gains. Retirement accounts like IRAs are liquidated, and tax is paid on those as well. The ex-American may want to shirk her tax responsibility, but the IRS has other plans.

Another pitfall to watch out for, says Reed: a permanent ban on reentering the U.S. While those who acquire second passports from countries like Canada, France, and the U.K. won’t have much of a problem visiting the U.S. in the future, those who attain passports through other means—including potentially investment programs in some countries—might have a harder time, he says.

“That’s why it’s not really a strategy for Americans just trying to avoid income tax; it doesn’t really work that well,” Reed says of renouncing citizenship purely to save on taxes. “What second citizenship are you getting? And will you be able to get back in?”