Seventy-five of the roughly 100 colleges with billion-dollar or greater endowments enjoyed increases in the size of their funds in fiscal year 2015, a year that also saw record-setting donations, with 30% of the money going to a few familiar colleges.

Such high-profile donations to Yale and Harvard this year produced biting criticism online that questioned why donors are so willing to give a mega-gift to “institutions that clearly don’t need it.”

A research report last year found that tax laws benefit wealthy private colleges to a far greater degree than their poorer public college counterparts. And earlier this month, a New York lawmaker proposed requiring wealthy colleges to spend more of their annual endowment returns on financial aid or face loosing their tax-exempt status.

Suffice it to say that there’s growing public discussion today about inequality between the haves and the have-nots in higher education.

So, should colleges sitting on more than $1 billion in endowment money be required to spend more to make college more affordable?

Most of the colleges, perhaps unsurprisingly, say no.

Colleges generally aim to have a 10-year average return of about 7.5% on their endowments, according to the National Association of College University and Business Officers.

Most colleges aim to spend between about 4% and 5% of their endowment year. Finance officers say that allows them to contribute some money to the operating budget and pay their investment expenses, while not letting the overall endowment lose ground to inflation.

The average spending rate for fiscal year 2015 was 4.2%, down slightly from 4.4% for the two previous years, according to an annual report from NACUBO. But more than three-quarters of colleges said they actually increased the dollar amount they spent from their endowments last year, made possible by donations that increased the overall size of the funds.

On average, colleges fund roughly a tenth of their annual operating budgets through their endowments. At many of the wealthiest colleges, though, the share is much higher. About a third of Smith College’s and Harvard University’s annual budgets come from endowment returns. Amherst, Princeton, and Grinnell all fund about half of their operating budgets through endowment sources.

About a decade ago, the last time lawmakers talked seriously about forcing colleges to spend more of their endowments, wealthier colleges voluntarily upped their financial aid for low- and middle-income families. Harvard, Princeton, Yale, and Stanford now all have extremely generous aid policies, awarding some amount of need-based aid to families earning up to $100,000 or more.

“It’s really important to recognize that at these schools, the only families that are paying full tuition are ones who are able to afford it,” says Catharine Hill, president of Vassar College and an economist specializing in higher education finances. “It’s not clear to me you’d want to further subsidize tuition for those families.”

But other critics say that while the wealthiest colleges are often generous with the low-income students who enroll, they could do a better job of recruiting and admitting more of them. At all but one of the 25 wealthiest schools, less than one in five of students qualify for federal Pell Grants, meaning they have a family income generally less than $50,000.

Washington University-St. Louis has $6.8 billion in its endowment, but just 6% of its students were considered low-income in 2014. At Northwestern University ($10.2 billion endowment), 30% of students still take out student loans to pay for their education, with an average $5,600 borrowed in 2014. And at the Emory University ($6.7 billion endowment), families earning between $30,000 and $48,000 a year still have an annual net price of attendance of nearly $17,000. (See the table below for a breakdown of the Pell rates, student loan borrowing, and grant aid at the wealthiest schools.)

With colleges facing increased pressure every year to spend a higher share of their endowment earnings, the key challenge for college business officers is to balance present demands with future interests, says John Walda, president and CEO of NACUBO.

While it may be hard or unpopular to withstand that pressure, Walda thinks colleges should try. Increasing the spending rate in a low-return environment like today’s could compromise long-term sustainability and hinder colleges’ ability to increase endowment spending in the future, he said during a call with reporters on Tuesday.

Returns on endowments in the past decade have also been volatile: down 18.7% in 2009, up 19.2% in 2011, down 0.3% in 2012.

Vassar aims to spend about 5% of its endowment each year, Hill says. It has focused some of that in recent years on increasing need-based financial aid and has nearly doubled the share of Pell-eligible students on campus.

During the worst of the recession, endowment spending at Vassar crept up over 6%, so the college could avoid cutting its spending dramatically. That’s the way it’s supposed to work, Hill says. A rigid rule that requires all colleges to spend a certain amount each year would take away colleges’ ability to smooth out the up-and-down cycles.

“If you’re always forced to hit that cap, you wouldn’t have the flexibility to save for when the bad times comes,” she says.

Diana Newman, a consultant who specializes in endowment building and fundraising, agrees that flexibility is key. She’s not convinced there’s a larger problem with colleges stockpiling their endowments—though she admits that sounds a little ridiculous when you realize Harvard has $36 billion. But, she points outs, it’s impressive what Harvard is able to do with the earnings on those billions each year.

“There’s nothing in it for them to be hoarding it,” she says. “It’s all about (showing donors) they’re spending it wisely.”

Money also reported on the 10 richest colleges in the U.S., which you can find here.