I recently watched an old news clip of Gretchen Carlson, years before she sued former Fox News CEO Roger Ailes for harassment. On national television, and while Gretchen attempts to do her job, two men on either side of her discuss her outfit as if she isn’t there. They approve. The flinch I see on her face is familiar. I feel my shoulder blades squeeze together (my version of gritting my teeth). Most women watching this scene and who work in a place where backslapping bro talk prevails would have this same reaction. It’s hard to watch because we’ve been there. And, like most women, Gretchen didn’t try to fit in or add to the conversation. She knew it would only prolong the moment.

Many women who work primarily with men find themselves feeling left out of the very culture they want to succeed within. While some will try and fit in, most become some version of invisible when men talk about them or other women they’d like to “tap.” I’m guessing that was Gretchen’s tactic. Her face says, “I am going along with your stupid banter and yes, I am attractive and yes, I am a sexy mommy. Now can we please get back to work?” But her voice just lets out the laughter of the smaller person in the room.

Gretchen’s experiences remind me of a time over a decade ago when a toy beaver lived on the Wall Street trading floor I worked on—a paddle-tailed, big-toothed animal that was stuffed within an inch of his made-in-China existence. He had been a gift, a wink, from one trader to another in admiration of a well-executed trade. Sexual connotation was the point: The guy was getting some beaver. The beaver continued to live with us, going from one male owner to the next, traveling by air as he was flung across the room, right over women’s heads when the win went to someone else. The few women I worked with didn’t allow themselves to feel revulsion. We all knew we were lucky to have our jobs, and the ones who didn’t, left.

Today, the culture varies at different banks, but they all have an overwhelming majority of men in senior positions. In recent years, Wall Street has made sizable investments in righting the gender disparity. Countless groups committed to helping women advance in the industry have formed, yet the efforts can’t seem to police the bro talk that’s more often felt than heard. The result is an environment that both discourages, and yes, holds women back. Almost all employees attend some version of discrimination and harassment-awareness training, so how is it that the bro speak still exists at all? With only 10% of female professionals on the trading floor and only 16% of senior executives being female (at all banks plus financial instruments), it’s unlikely that the few women at that level even want to discuss it. They’ve beaten the odds and have a good thing going. So what is it that will shut bro speak down? It’s about having an employee base that more closely models the human race. By having more women on the floor, the fraternal yuks will soften and fade.

Beyond distasteful chatter, it’s hard to define exactly what bro talk is. Often, I felt men were trying to out-funny a television sitcom, not necessarily trying to directly hurt anyone. But does it feel exclusive and gross at the same time? Of course it does. Bro talk encourages the invisible chest-bump of the brotherhood, that team feeling, that got-your-back acceptance that most companies yearn for their employees to have. Bro talk encourages a bonding experience that money can’t buy, and that ensures employees go the extra mile for one another. So what is the cost to women when they stay silent?

While doing research for my recent novel, Opening Belle, I interviewed women working on trading floors. They don’t complain at work because they fear retribution, but they were thrilled to talk openly on this topic. While they’d like to tune out the trash talk they overhear, where could they possibly hide? In language both crude and creative, they report men wondering aloud as to whether or not a new hire across the room was a Heisman (think of the statue’s posture, one arm hugging the chest, the other in a protective block, meaning she looked good from afar, but up close, it’d be better to block her image). Others heard men discuss whether or not a co-worker had real breasts and implored another woman to find out for them (she didn’t take the bait). Others loudly considered whether another was Spanx–worthy, meaning whether or not the tight-fitting undergarments would be worth the effort to remove.

The women who work with these guys just nod and look away when the talk gets debased. Speaking up would be career limiting. I know. I spoke to a woman who, when a summer intern was shocked at the language of the brotherhood, went up many levels of seniority to complain at her firm. Her boss saw no punishment to the guilty party, but instead re-assigned most of her most profitable accounts to others. Bros don’t tell on one another.

In a recent article for The New York Times, Sam Polk explains that he believes this bro speak is another reason why career advancement is anemic for women on Wall Street—that breaking into the boys club and the career advancement therein is impossible if you don’t speak the language. In my findings, women consistently reported that over-hearing bro talk made them feel small and embarrassed, so how could it not be affecting their work? Office bonding doesn’t happen if you aren’t a person the majority wants to bond with.

The bro speak of some men and the negative effect on the women who work with them fits snugly into the description of an uncomfortable workplace. The chest bump of the brotherhood that runs the place includes giving promotions to the men they feel unconsciously biased toward. Many women report seeing careers take off for lesser-producing men who are able to speak a language they can’t—nor should they want—to be fluent in.

When I searched for examples of women feeling included, they bordered on the bizarre. One went so far as to go to a strip club with her co-workers, only to have multiple lap dances bestowed on her and her embarrassed self. She laughed along with the men who watched another woman grind into her, but was she really laughing? One mentioned a paintball outing where she spent the day getting ambushed with balls of oil. Another wrapped herself in Velcro and was thrown against a wall, and another considered joining some men in a Navy SEAL-like training weekend. They admitted that putting themselves out there did little to move the career needle, but filled a need for them to feel “better liked.”

Most of us don’t have the stamina or the desire to go to those lengths. Most just redden, look the other way, and pretend to not know why the better account was just assigned to a junior guy—or why a beaver just flew over her head.