By Kara Brandeisky, Money

If Tax Day has you worrying about an IRS audit, you probably have little reason to be nervous. Last year, the IRS audited less than 1% of all taxpayers—and the federal agency is on track to audit even fewer people this year.

“The math is pretty simple. There are fewer audits because we have fewer auditors,” IRS commissioner John A. Koskinen told the New York State Bar Association in February. “The IRS lost more than 2,200 revenue agents since 2010. Last year alone, there were 600 fewer auditors, with the total falling to 11,600—the lowest level in more than a decade.”

Still, some Americans are subject to more scrutiny than others. The IRS doesn’t spell out why auditors single out some returns for special treatment, but a look at the agency’s track record provides some clues. Here are the groups that are more likely to get the government’s attention:

1. People who report more than $10 million in income—or none at all

It’s like the old saying about why the bank robber robbed the bank: “Because that’s where the money is.” With limited resources, the IRS takes a harder look at people with the most money (and the most to hide). In 2010, then-commissioner Doug Shulman told the New York State Bar Association targeting the rich was part of a new strategy to “work smarter.”

“This is a game-changing strategy for the IRS,” Shulman said. “Initially, we will be focusing on individuals with tens of millions of dollars of assets or income. Going forward, we will take a unified look at the entire complex web of business entities controlled by a high wealth individual, which will enable us to better assess the risk such arrangements pose to tax compliance.”

In 2014, the IRS audited more than 16% of returns reporting more than $10 million in income. But, as you can see in the table below, single-digit millionaires should take care with their tax returns as well.

Another group with a high-than-average chance of getting audited? People who report no income. If you’re reporting an operating loss on your business, the IRS might double check that you’re being honest. In 2014, the IRS audited 5.3% of the taxpayers who reported no income.

Otherwise, if you—like the majority of American taxpayers—earn between $25,000 and $200,000, you have a better-than-average shot of dodging an IRS audit. Here’s the breakdown:

Source: Internal Revenue Service Data Book, 2014

2. People who file estate tax returns for assets worth more than $5 million

Likewise, a huge estate tax return could raise some eyebrows at the IRS. Overall, 8.5% of estate tax returns were singled out for special scrutiny in 2014, way more than the 0.9% of individual tax returns.

And the bigger the estate, the more likely the IRS flagged the return for an audit. More than 21% of estate tax returns with assets between $5 million and $10 million were audited in 2014, and 27% of returns with assets worth over $10 million were audited.

However, estate tax returns are pretty rare: The IRS received 33,719 in 2013, and only 3,359 of those were for estates worth $10 million or more.

3. People who file international returns

If you’re mailing your return from the Cayman Islands, you can bet that the IRS is onto you. Over the past several years, the IRS has increased scrutiny of international returns.

“On the individual front, we have made putting a big dent in offshore tax evasion a major priority,” Shulman told the American Institute of Certified Public Accountants in 2012. “We view offshore tax evasion as an issue of fundamental fairness. Wealthy people who unlawfully hide their money offshore aren’t paying the taxes they owe, while schoolteachers, firefighters and other ordinary citizens who play by the rules are forced to pick up the slack and foot the bill.”

In 2014, the IRS audited 4.8% of international returns.

But there’s a cost to fewer audits

Law-abiding citizens have little reason to celebrate the limited number of tax audits. Koskinen expects that, thanks to federal budget cuts, the IRS will lose out on at least $2 billion in revenue that auditors would otherwise be able to collect. Plus, sometimes when the government takes a second look at your return, you get more money back: In 2014, the IRS decided 38,029 individual filers had paid too much in taxes and sent them additional refunds.

Watch more business news from Fortune: