The majority of Americans do vote on Election Day in presidential years—though even that has dropped from 57% in 2008 to 54% in 2012. History suggests there will be a 20% or so drop-off between a presidential election and the midterm two years later.

But the real action—the place where victors are decided—is increasingly in party primaries, where only a tiny fraction of voters go to the polls. Most congressional districts today are solidly Republican or Democratic, owing to advances in technology that enable legislatures to gerrymander congressional districts with impeccable precision. Veteran electoral analyst Charlie Cook has estimated that the number of swing districts declined from 168 in 1998 to just 99 in 2012. And there are even fewer this year.

The result? Whoever wins the primary in the predominant party will probably win the general election. Primaries have become a breeding ground in both parties for ideologues who resist compromise and refuse to govern. “If you combine low turnout with one-party districts, the primary is essentially the election,” says Curtis Gans, whose nonpartisan Center for the Study of the American Electorate calculates voting patterns. “An organized minority of 3% to 4% can propel someone into office.”

[c

The U.S. Chamber of Commerce got its wake-up call in last year’s special election in Alabama, when it managed to stop a candidate who applauded the government shutdown, wanted to overthrow House Speaker John Boehner, and claimed that President Obama was born in Kenya. During this year’s primary season, the Chamber poured $15 million into creative media support for candidates facing similar challenges.

Scott Reed, the veteran GOP strategist recruited to run the Chamber’s blitz, refuses to apply the Tea Party label to the candidates it opposes, saying the Tea Party is an important part of a center-right coalition. Instead, he says, he’s going after candidates from the “caveman caucus.” “When you get [to Washington],” says Reed, “you have to be willing to govern.” That’s a new standard that Chamber strategists are using in assessing whom to support.

Primary candidates supported by the Chamber, along with other establishment GOP groups like Karl Rove’s American Crossroads, won in 14 out of 15 races this year. “It allowed us to set the table with varsity candidates for general election,” said Reed.

Working with local chambers, the Washington office also mounted get-out-the-vote efforts to encourage more of Main Street to go to the polls. Still, the nationwide primary turnout has continued its steady decline. Gans estimates the national primary turnout this year to be in the 14% to 15% range, down from 18.3% in 2010.

That level of disengagement by most Americans, who trend toward the center, still leaves wide running room for politicians less interested in governing than in making a fiery point. The Republican right gets most of the attention on this score, but in Congress there is similar intractability on the left. Activists would have President Obama’s head if he dared raise the retirement age or cut the world’s highest corporate tax rate.

On the GOP side, Main Street dodged a bullet this year: There was no rerun of the 2012 debacles like Todd Akin or Richard Mourdock. But the caveman caucus will reemerge with well-funded ferocity in the 2016 presidential campaign. Once again, the primaries will be where the action is.

So the next time you feel like complaining about gridlock and dysfunction in Congress, look in the mirror—and then, come the winter of 2016, go plant a campaign sign in your front yard.

This story is from the October 27, 2014 issue of Fortune.