Urgent memo from his anxious ministers to Sir Keir Starmer: you need to get a grip of No 10 | Andrew Rawnsley

Over a cuppa in Liverpool, the Labour MP was getting nostalgic about the election campaign. Drivers would spot her on the street and stop their cars in order to wish her luck and give her a hug. Then she became wistful: "They're not hugging me now."

The Labour conference was a curious cocktail of the jubilant and the jittery, the gleeful and the grumbly. Celebration of the largest parliamentary landslide in a generation was mixed with angst about Labour's tanking popularity. Don't let anyone tell you it was an entirely overcast occasion. I met plenty of buoyant delegates along with ministers saying how brilliant it was to be in power after so many miserable years in opposition. One cabinet member expressed delight that "after 14 years howling into the void, we can now do things". The mantra "the worst day in government is better than the best day in opposition" was often to be heard.

Don't let anyone tell you that it was universally cheery either. One senior figure, who was a student when he attended the Labour conference immediately after the first Tony Blair landslide, remarked dolefully: "It is not like 1997, is it?" It probably never could have been, given the state of the realm that Labour inherited from the Tories. Realists in the cabinet didn't expect voters to be happy with Labour for all that long, but few thought they'd be evicted from the honeymoon suite with such brutal rapidity. After just three months in power, the government's approval ratings are slumping and Sir Keir Starmer's personal score has plunged precipitously .

There's a pervasive feeling that the desire to ram home the direness of the Tory legacy and prepare the ground for difficult decisions on tax and spend has led to an excess of miserabilism about what faces Britain. At a pre-conference meeting of the cabinet, I am told, Pat McFadden warned his colleagues that they had struck the wrong "balance between light and shade". The Cabinet Office minister usually projects the persona of a dour Scot to the point of conscious parody. So it tells you something when even the lugubrious Mr McFadden thinks they've overdone the doom and gloom.

Cabinet members were encouraged to expect the speeches from the prime minister and chancellor to sound more optimistic notes. Both did, up to a point. Rachel Reeves claimed that " Britain's best days lie ahead " while Sir Keir spoke of "a Britain built to last". What neither quite managed to conjure up was an inspirational vision of a shining city on a hill.

Their essential message was unchanged: it will take time to put the country on the right track and it will involve doing things that a lot of people are not going to like. The removal of the winter fuel allowance from most pensioners, the subject of a hostile conference vote on the last day, is going to be followed by a budget that will be flinty about day-to-day spending even if there is tweaking of the fiscal rules to allow more funding for capital investment.

The neighbours of Downing Street pride themselves on being hard-working, practical, methodical people. Neither are lyrical orators. More, they are instinctively wary of anything that sounds like what the Labour leader derides as "the politics of easy answers" and "false hope". Sir Keir keeps saying that he is "prepared to be unpopular" almost as if it would be a disgrace not to be disliked. Both prime minister and chancellor are calculating – perhaps I ought to say gambling – that they will ultimately get credit for confronting the public with the tough trade-offs necessary to reform the country.

One of the standout lines from the Starmer speech was the declaration: "We will turn our collar up and face the storm." This bad-weather warning told us that he still thinks things are going to get worse before they get better. Referencing the furore over winter fuel payments, one party veteran remarks: "Labour backbenchers better be prepared to defend a lot worse shit than this."

To endure the tempest in decent shape, the government will need to be organised, resilient, deft and united. So another source of anxiety, which I frequently hear expressed by members of the cabinet, is that the centre is not functioning anything like as smoothly as they expected from a Starmer-headed Downing Street. "Number 10 hasn't worked out its role," observes one cabinet member. "Too much time is being spent looking inwards trying to sort themselves out and not enough of it is being spent looking outward." There's more to this than fretting about the failure to shake off the freebies saga and dismay about rancorous power struggles and personality clashes within Number 10. I hear ministers lament that there has been inadequate preparation for power in some vital areas, while also complaining that agreements made in opposition about how things would be run have been torn up since they arrived in office. Says one cabinet source: "We had it all agreed who were going to be ministers and who were going to be spads (special advisers). Then it all got junked. So what was the point of all the preparation?"

The pivotal post of principal private secretary to the prime minister is unfilled because of feuding about who it should be. There are also growing concerns that the "mission boards", supposed to drive delivery of the government's ambitions, are already proving too sluggishly bureaucratic for the task. Some assign the blame for dysfunctionality to Sue Gray, the chief of staff who has been in the headlines far more than she wants to be . Others assign responsibility to her enemies operating in the shadows. When talking to ministers, I find many fingers angrily jabbing in the direction of Simon Case , the Boris Johnson-appointed cabinet secretary. If I were to sink a beer every time I hear a Labour person say he needs to be ushered out of the building as soon as possible, I would require hospital treatment for alcohol poisoning. This can't be left to drift until after Christmas. Sir Keir needs a new cabinet secretary pronto and one with the qualities required to command not just the prime minister's confidence, but also that of the civil service and the cabinet.

The thorniest challenge is convincing the country that "change" was not just an election slogan, but a plan to make palpable improvements to people's lives. One difficulty, in both managing Labour MPs and telling a persuasive story to the public, is that we have entered what you might call the post-euphoria, pre-delivery phase. Delight about the removal of the Tories is dissipating without being replaced by enthusiasm for what Labour is attempting to do. For sure, ministers can reel off a list of actions they've taken since the July election: from making a start on the creation of GB Energy ; to changing the planning regime to facilitate more building; to announcing the plan to bring all the train operators into public ownership ; and to establishing a new border security command .

It is a long list with a snag attached. Politicians don't win public affection by publishing white papers and announcing reorganisations. They rarely win applause for issuing a new regulation or passing a piece of legislation. Apart from the pay rises agreed for public sector workers, as far as many voters are concerned, there's little yet that has made a visible, feelable, tangible change to their lives. Breakfast clubs in every English primary school are one of the government's better ideas and also one that shouldn't be too taxing to bring into existence. Parents will notice that as an appreciable result of Labour government. Yet the rollout will not start until the beginning of the next summer term in April 2025, and at first on a trial basis covering just 750 of England's 16,783 primary schools . To ensure that policy is implemented effectively, there's sense in testing what works rather than rushing in and repenting later. But the incremental approach delays the day when voters start to see the difference that Labour can make.

"Treasury says no" is another obstacle I hear complaints about. Cabinet members with initiatives they're itching to take are chafing against the restraints imposed by Rachel Reeves' powerful bailiwick because it is blocking any announcements with a cost implication before the budget at the end of October.

There's plenty to commend about the candour of the chancellor and the prime minister when they ask for patience because a better Britain won't be built in a day. But after committing too many avoidable own goals, the government could now do with some quick wins.