Last week, as part of the first-ever White House Demo Day, an event that highlighted female and minority entrepreneurs, President Barack Obama issued a call to action to the tech industry, asking companies to step up their game on workforce diversity. Of the 14 tech companies that responded to the President’s challenge, seven announced that they would attempt to hire more women and underrepresented minorities by implementing the so-called Rooney Rule. Two of the participating tech giants—Facebook and Pinterest—had already revealed their plans to try out the rule earlier this summer, while the others, including Intel, Xerox and Amazon, are joining the Rooney club for the first time.

What is it?

So, what is this Rooney Rule that’s suddenly the darling of the tech world? It’s actually very simple: Under the rule, at least one woman and one underrepresented minority must be considered in the slate of candidates for either every open position or every open senior position (the details vary from company to company).

The original Rooney Rule was implemented in 2003 in the NFL by Pittsburg Steelers chairman Dan Rooney. In his version of the rule, Rooney mandated that every team with an opening for a head coach position interview at least one minority candidate for the role; the rule was expanded to include general managers four years later.

Does it work?

The success of Rooney’s rule is debatable. From 1992 to 2002, seven out of 92 NFL head-coaching vacancies (less than 10%) were filled by minorities, according to ESPN. In its first decade, the Rooney Rule seemed to make a notable difference: Beginning in 2003, ESPN reports that minorities filled about 20% of head coaching vacancies—and that percentage edged up steadily each year.

In 2012, however, progress seemed to come to a halt. According to the New York Times, no minority candidates were hired for any of the open coaching vacancies or GM positions that entire year. As a result, an advisory committee was formed by the league in 2013 to help identify and train candidates, resulting in two minority hires. The only new head coach from a minority background for the 2015 season is Todd Bowles of the New York Jets.

What about metrics?

This tapering off highlights one of the biggest criticisms of the rule: It sets no goals for how many candidates should be hired. This is a major part of the reason the tech world likes the Rooney Rule so much, says Joelle Emerson, co-founder and CEO of diversity consultancy Paradigm, whose clients include Pinterest, Slack and Airbnb.

“There are companies where you’re just not going to see [hiring goals] happen right now. They’re resistant to setting goals before they understand what its going to take to get there,” she says. Much of the resistance to putting hard numbers on diversity comes from the belief that “goals equal quotas, and quotas will lower the talent bar,” says Emerson.

However, some companies are making their diversity targets public, which is “a really powerful strategy,” according to Emerson. Just last week, her client Pinterest announced its hiring goals for 2016; for full-time engineering roles, the goal is for 30% of roles to be filled by women and 8% of roles to be filled by people from underrepresented ethnic backgrounds. Xerox has long assigned numbers to its employment objectives, which is perhaps why the company is setting its sights even higher: It is aiming for 34% of the executive leadership team to be female within the next five years (up from 28% currently), according to Damika Arnold, global diversity and inclusion leader at Xerox.

But specific targets come with their own set of issues, says Emerson. For example, focusing on gender and race can end up hurting other type of minorities, such as members of the LGBTQ community, or people who come from universities that are not considered “feeders” for tech jobs. Students of all ethnic backgrounds who attend community colleges are not recruited as heavily as students from Ivy League schools, although they may have the skill-set necessary to succeed in a given role.

Who’s accountable?

Accountability is another potential problem of the Rooney Rule. How are companies going to ensure compliance? Within the NFL, senior hires are closely monitored by both the league itself and the Fritz Pollard Alliance, a nonprofit organization that tracks the league’s minority hiring practices. Teams that don’t comply with the rule face financial penalties, says Robert Gulliver, the chief HR officer of the NFL. In the corporate world, however, it becomes more challenging to oversee who is hired and to hold hiring managers and company leaders accountable for inclusive hiring practices.

At Xerox, managers’ ability to foster diversity and inclusion will become part of the annual review process, says Arnold. While it won’t be directly tied to compensation, linking diversity goals to a manager’s performance keeps the issue “at the forefront,” she says.

Intel, on the other hand, has a slightly less formal review process. Managers will be expected to report on the diversity of their teams to their own supervisors, says the company’s chief diversity officer, Roz Hudnell.

Still, while the rule may sometimes work well for filling a senior-level position—of any role where only a handful of candidates are in contention—it’s challenging to implement the strategy across all levels, says Emerson. “I just can’t imagine how you’re going to do it,” she said. “Are companies just going to consider one minority out of every pool no matter how big the pool is?”

Hudnell shared this sentiment: “At the NFL, it’s not like there are thousands of job openings. At a company of [Intel’s] size, it’s less about a slate, and more about a pool,” she said. “It’s less about considering one or two diverse candidates per job, and more about casting a wider net.”

The verdict

The Rooney Rule is just one method of addressing a seemingly daunting problem and its simplicity makes gives it a certain appeal. However, to truly tackle the lack of diversity in tech, leaders must commit to finding solutions that work for them. And they don’t all need to be massive or complex initiatives, says Sarah Nahm, co-founder and CEO of talent management platform Lever. Something as simple as putting a dollar into a “Guys Jar” every time an employee uses gender-biased language (something Nahm has seen work well with her clients, which include Box, GitHub, and Reddit) can help increase awareness of unconscious bias.

The Rooney Rule is a step in the right direction, and it’s likely to have some measurable, near-term impact. However, as in the NFL, that impact will plateau at a certain point if firms lose sight of why they put the Rooney Rule in place, say diversity experts. Considering one woman and one minority candidate for every job without understanding the business case for diversity—which has been proven time and time again to increase profits—isn’t going to get us very far.

As Hudnell put it, “We have to keep our focus on [diversifying companies], otherwise we as human beings will revert back to what is the most convenient and what is easiest. Until we get to a point where minorities are a critical mass in our corporations, we have to keep working at it.”

Subscribe to The Broadsheet, Fortune’s daily newsletter on the world’s most powerful women.

Editor’s Note: The original article incorrectly stated Xerox is aiming for 34% of its executive leadership team to be female by next year. The correct time frame is within five years.