Boris Johnson faces a drip-drip of Conservative MPs calling on him to resign, as former Attorney General Jeremy Wright said Monday the so-called “partygate” scandal had done “lasting damage” to the government.

The prime minister had been hoping that with Parliament on holiday to mark Queen Elizabeth II’s 70 years on the throne, Tory rebels would be keeping relatively quiet back in their districts. The government is also rolling out a slew of announcements it regards as popular with Conservative voters, including a plan to bring back imperial measurements.

But the plan does not appear to be working. The number of Conservatives calling for a change of leadership is rising slowly but persistently, with 10 more going public since the publication last week of an internal probe into the illegal gatherings in Downing Street during the pandemic.

Though that did not contain an obvious smoking gun, it did lay bare the extent of drinking and partying that took place in government offices on Johnson’s watch. Wright said the events outlined in the report risked undermining the current administration but also the “authority of government more generally.”

“That matters because it is sadly likely that a government will again need to ask the citizens of this country to follow rules it will be difficult to comply with,” Wright said in a statement on his website. “What we now know happened in Downing Street during months of COVID restrictions imposed by government policy makes that consent less likely.”

Wright served as Attorney General under Johnson’s predecessors David Cameron and Theresa May. Other prominent Tories to criticize Johnson include Tobias Ellwood, chair of Parliament’s defense committee, who told Sky News on Monday the party will lose the next election under the current leadership.

In total, there are 26 Tory MPs who have publicly called for Johnson to quit — including Elliot Colburn later on Monday — and a further 10 who have criticized his conduct but without seeking his resignation. Those numbers are still short of the 54 it would take to trigger a vote of no confidence in the prime minister.

The problem for Johnson is that only Graham Brady, who collects the letters from disgruntled Tory MPs, knows the exact number of rebels. That makes it difficult for the prime minister’s team to prevent the threshold being triggered.

The premier has focused on policies designed to please his party, ranging from expanding the grammar school system to allowing Britons to return to the use of pounds and ounces. The latter is one of his Brexit promises, to repeal legislation introduced when the UK was a member of the European Union.

As Johnson tries to get Tories on side, comments from Education Secretary Nadhim Zahawi that emerged on Monday are unlikely to have helped. “Every MP must feel that they can put themselves forward to lead their party,” he told a podcast. Cabinet ministers are usually extremely careful to never suggest even the possibility of a leadership challenge, for fear of appearing disloyal.