When we think about America’s prison system, we think almost exclusively about young men—and for good reason, as more than 93% of inmates are male, according to the Federal Bureau of Prisons.

But while the proportion of incarcerated women is small, the actual number is surprisingly large: There are more than one million women currently behind bars in the U.S., and that number is on the rise. In fact, women are the fastest growing segment of the prison population in the country, and the rate of incarceration for women has been growing nearly twice as fast as that of men since 1985, according to the ACLU.

The drastic increase in the number of women is due to the past two decades’ war on drugs, said Congresswoman Karen Bass (D-CA). Many of the women in today’s prisons are there because of romantic or other relationships with male drug dealers, who give their names to officials in exchange for more lenient sentencing, she explained. In 2000, 40% of women’s criminal convictions leading to incarceration were for drug crimes, according to the ACLU.

Yet when efforts are made to reform the prison system, women are almost entirely left out of the conversation, said Bass. “When they were making the efforts for criminal justice reform and they forgot women, they didn’t mean to,” she said.

Bass pointed out that women tend to be in prison often for different reasons than men are—the majority are non-violent offenders—and have different needs than male prisoners. “It should absolutely be gender-specific,” she said, referring to the conversation about women’s prison reform.

Among the women-specific measures that the panelists discussed was the shackling of pregnant prisoners, which has been outlawed in 21 states, according to the New York Times, yet continues to occur. “The policy is that inmates transported to medical visits get shackled. For pregnant women, you can’t fake the pregnancy. There’s an armed guard in the room. Why on earth would you torture women this way?” asked panelist Pat Nolan, director of the American Conservative Union Foundation’s Center for Criminal Justice Reform.

Incarcerated mothers face other obstacles, including the sky-high cost of phone calls, and strict reunification laws that prevent mothers who are in jail for longer than a specific period of time to regain custody of their children once they are released. Mignon Clyburn, chairwoman of the Federal Communications Commission, said that some families spend up to $30,000 on phone bills so that children can stay in touch with their mothers—and as a result, only 38% do.

Many women are “totally lost in the system,” said Nolan, pointing out that while a hardened criminal may know how to survive prison, many incarcerated women, who are there because of relationships to criminal men, are unable to cope.