Cabinet ministers call for Jonathan Ashworth role at 10 Downing Street after weeks of infighting

Cabinet ministers are demanding a rapid shake-up of Keir Starmer's Downing Street operation, which they say has failed to spot obvious political banana skins, indulged in weeks of post-election infighting and been unable to promote a sufficiently positive story of Labour's mission in government.

Some senior figures in Starmer's team want the prime minister to bring in Jonathan Ashworth, the former frontbencher who was a regular and, many believe, effective media performer during the general election campaign, to beef up the team and give it a sharper political edge.

In one of the shocks of election night, Ashworth was narrowly defeated in Leicester South by Shockat Adam, a pro-Palestinian independent candidate . The ex-MP now heads the Starmerite thinktank Labour Together. Former colleagues believe he is wasted there and that Starmer should install him at the heart of his operation.

The calls are likely to intensify after Saturday's shock announcement by Rosie Duffield, the MP for Canterbury, that she was resigning the Labour whip and would sit as an independent. Duffield told the prime minister she was quitting because of "cruel and unnecessary" government policies such as cuts in the winter fuel allowance and the two-child benefit cap.

Frustration at the way No 10 has squandered what many believed could and should have been a long political honeymoon was evident among ministers at all levels of government at last week's Labour conference in Liverpool.

Instead of being a celebration, the party met amid plummeting poll ratings, rows about "freebies" for the PM and others, stories of top aides falling out and leaking negative information about one another, and with few substantial new policies or ideas.

Several senior ministers said stories about freebies, and internal rows between the likes of Starmer's chief of staff, Sue Gray , and cabinet secretary Simon Case should have been headed off or, once they had emerged, shut down earlier. As it was, they had been allowed to gather momentum because too many senior figures had been preoccupied and at loggerheads, briefing against one another as they tried to establish their own power bases after the election. Another cabinet source said: "We need to sort this out. It can be dealt with. But it has to be done soon if we are to be seen as what we say we are, a government of service."

Bemused newly elected Labour MPs, who had looked forward to some form of welcome party in Liverpool, summed up the atmosphere. "It is very strange," one said. "Not exactly joyous is it?" Another MP, when asked if he was positive about the government's chances of success over the next five years, said: "I am maybe 60% upbeat, not more."

Since the conference ended on Wednesday with a defeat for the leadership over its controversial plans to limit winter fuel payments to pensioners, the Guardian has revealed that Starmer was given a further £16,000 of clothes before the general election by the Labour peer Waheed Alli .

For weeks, Starmer and cabinet members – who promised to end Tory sleaze – have been facing criticism for accepting gifts from Alli, which in the PM's case included £2,400 for glasses and tens of thousands of pounds of clothes, as well as concert and football tickets from other organisations.

Much of the blame for problems inside No 10 is being laid by officials and ministers at the door of Case, who is expected to leave his post in the new year. He and Gray – two of the officials with most access to Starmer – do not get on and figures in government suspect he has been briefing against her. The BBC recently revealed that she was earning £170,000 a year – more than the PM.

"It is completely mad that we have allowed this Sue versus Case thing to go on," said a cabinet source. "They don't get along. We know that. Only two people would know how much Sue Gray is paid and he is one of them. He should be shipped out now."

Another well-placed source said there had not yet been a proper transition of officials from opposition to government, with a clear enough allocation of roles. "You have the election campaign lot coming in to No 10 and those who worked on policy. They are kind of all in their own lanes still, but veering all over the place."

Stories that Gray has been involved in rows with Starmer's key aide, Morgan McSweeney , including claims that Gray had twice moved McSweeney's desk further away from the PM's office, have circulated but been dismissed by insiders as "completely ridiculous, total fiction".

One minister said the response to stories about freebies as slow and ineffective. "First we said nothing, then tried to put up a defence, then tried to say it was all OK, then that it wasn't. There was no line for ages."

Ministers also say that damaging stories – true and false – have been allowed to fill a policy vacuum caused partly by the timing of chancellor Rachel Reeves's first budget on 30 October and the five-year spending review next spring. Ministers say they want to reveal more long-term policy ideas but do not feel able to do so before knowing what is in both documents.

One source said: "We are not able to reveal what we want to do because we are in a weird no man's land. But we could have come up with a few more stocking fillers in the form of floating ideas and plans."

The consensus within the government is that at least two or three more people with serious political and media experience need to be installed alongside the director of communications, Matthew Doyle, who helped mastermind the landslide election win, in order to spread the load and beef up the operation.

"The trouble is that to do that you have to get any plan past Sue and Matthew first," said an insider.

"Keir is loyal to his people. He will be furious about people criticising the operation. But things have to change."