PALOS VERDES ESTATES, Calif. » The days appear numbered for the “Stone Fort,” a venerable edifice that was illegally erected decades ago by a group of surfers and became a beachhead in their ongoing war to keep outsiders away from some of the best waves in Southern California.

Under pressure from the California Coastal Commission, the City Council in tony Palos Verdes Estates voted unanimously Tuesday to have the concrete-and-stone structure torn down.

A public hearing is scheduled in September to discuss how to do it, but the city manager in the seaside community of multimillion-dollar homes and priceless views says jackhammers are the likely solution.

The impressive structure is made of rock walls and sits on the beach just above the tide line at the base of towering cliffs. Its amenities include a table, bench, fire pit, shaded patio and a place to store boards and kayaks.

Critics have claimed in a lawsuit that a murky group of local surfers known as the Lunada Bay Boys use it as an observation point to spy on arriving outsiders then harass them when they reach the shore.

The lawsuit indicates most members of the Bay Boys are middle-aged men and longtime residents of Palos Verdes Estates. Members, however, make a point of not advertising their affiliation in public or on T-shirts or surf gear.

Many residents deny the group exists at all, and no one identified themselves as a Bay Boy at the council meeting.

Surfer Magazine recently listed the Lunada Bay beach as one of five places to avoid — no matter how good the waves break.

“To get there you’ll have to duck rocks chucked by the Bay Boys,” the magazine said while mocking the wealthy locals. “And your car windows will get waxed. And your tires might get slashed. You’ll endure chest-thumping and lip-quivering threats shouted from the beach shack the Boys inhabit near the shore.”

While calling for the removal of the fort, councilmembers cited liability issues, including concerns that an outsider might fall from one of the fairly treacherous pathways to the breakwater while toting a board.

But the fact is that the Stone Fort has been under assault for months by an increasingly agitated group of outsiders who say the Bay Boys use it as a staging area to keep others off the beach and away from the primo waves.

The class-action lawsuit filed against the city and several residents in March by a pair of outsiders describes the Bay Boys as more thugs than surfers. It accuses them of running out-of-towners over with their boards in the water and even taking their wallets, wetsuits and surfboards.

Police say the fort is almost always deserted by the time officers respond to complaints about the treatment.

The Coastal Commission, noting the structure was built without a permit, told the city to either remove it or apply for a permit with the promise of ending the harassment and providing better access to the beach.

Locals say waves along the beach can be spectacular but only occur about 30 days a year in the winter. When the surf is up, everyone wants to be there and that’s when the Bay Boys get busy.

“They tend to be aggressive in keeping other surfers out or intimidating people who look like they are going to surf,” 15-year resident Cliff Filepe said as he walked his dogs along the bluffs, noting that he has never been bothered by the group.

“I don’t know that they bother anybody who just wants to stand around here, but they consider the surfing their territory,” he said.