Federal officials say they won’t evict the Oceti Sakowin, or Seven Council Fires camp, due to free speech reasons, even though it’s on U.S. Army Corps of Engineers land near the confluence of the Missouri and Cannonball rivers that many Native Americans believe is still rightfully owned by the Standing Rock Sioux under a nearly 150-year-old treaty.

“We’re not leaving until we defeat this big black snake,” camp spokesman Cody Hall said of the pipeline.

But residents in the area have expressed feeling unsafe and frustrated with how the protest has swelled to scores of self-described “water protectors” who have joined the tribe’s fight, and Republican U.S. Rep. Kevin Cramer—North Dakota’s lone voice in the House—says the camp is illegal. He blames the agency for looking the other way.

“If that camp was full of people advocating for fossil fuels, they would have been removed by now,” Cramer said. “There is some discretionary enforcement going on.”

The Fight Over the Dakota Access Pipeline Could Be the Next Keystone

The camp is the overflow from smaller private and permitted protest sites nearby and began growing in August. The gathering has been called the largest gathering of Native American tribes in a century, all there to protest Dallas-based Energy Transfer Partners’ $3.8 billion Dakota Access pipeline, which tribal officials believe threatens sacred sites and a river that’s a source of water for millions.

Corps spokeswoman Eileen Williamson said the agency is “encouraging” people to relocate to areas where there is a permit, such as a nearby smaller camp where demonstrators have been allowed to legally protest on federal lands managed by the agency.

“We don’t have the physical ability to go out and evict people—it gives the appearance of not protecting free speech,” she said. “Our hands are really tied.”

Cramer believes the “bigger problem” of the camp is “the illegal activity that may be orchestrated from there”—meaning a base to launch interference with pipeline construction miles away. About 95 people have been arrested in several pipeline protests — some as far as 70 miles away from the camp—since early August, but none at the camp itself. Protesters deny state officials’ charge, with Hall saying the camp promotes “peaceful” protest.

“People don’t leave from the camp with malicious intent to do harm,” Hall said. “There are always going to be a few bad eggs in any group you can’t the message to.”

North Dakota Pipeline Protest Turns Violent After Native American Burial Grounds Allegedly Damaged

Ranchers and farmers in the area are wary of the growing number of protesters and are fearful of damage to their fields and pastures, Morton County Sheriff Kyle Kirchmeier said.

“We’ve had several reports of people down there getting stopped on the road and being intimidated to hay that has been stolen to grazing animals without permission,” he said.

Matthew Rebenitsch, who has a ranch just a few miles from the encampment, said he and his neighbors worry about talking publicly about the situation.

“To be honest, no one around here wants to say anything because we’re afraid they will come and threaten us,” he said. “I’ll say this, every rancher around here is packing and people are locking their doors—and no one has ever locked a door in their entire life.”

Republican state Rep. James Schmidt, who also farms and ranches nearby, said he and his neighbors are more cautious.

“I’m starting to see a lot more guns in the back of pickup trucks. All it is going to take is one incident and emotions are going to overtake the situation,” he said.

North Dakota Oil Pipeline Delays Could Hit Producers, Shippers

Complicating the issue is a grazing lease on that land. The renter, David Meyer, is responsible for the condition of the land. He could enforce his lease and have people removed, Williamson said, but hasn’t requested any intervention. Meyer recently sold part of his historic Cannonball Ranch to Energy Transfer Partners for an undisclosed price; he hasn’t returned telephone calls from The Associated Press seeking comment.

The Standing Rock Sioux has challenged the Army Corps of Engineers’ decision to grant permits for the pipeline that is intended to carry North Dakota oil to Illinois. Due to a federal court decision, construction has stopped within 20 miles on either side of Lake Oahe, a Missouri River reservoir that is near the encampment. The Departments of Justice, Army and Interior also has said it will “reconsider any of its previous decisions” on land that borders or is under the lake.

Hall, a member of South Dakota’s Cheyenne River Sioux Tribe, said the Sioux “never ceded” the land on which they’re camping. He said they are preparing to stay through the winter by stockpiling firewood, heating stoves and warmer clothes. But there’ll be no permanent structures built, Hall said.

“We will leave this camp as we found it,” he said.