Andrew Grice: The Week in Politics

Forget all the fuss about Labour's campaign against the "Tory toff" in the Crewe and Nantwich by-election. To paraphrase Al Gore, it is a convenient untruth for Labour to hide behind. The people of Crewe did not vote against a silly campaign by Labour workers wearing top hats. They voted against Gordon Brown.

The catalyst was his decision to abolish the 10p income tax rate. Not only did this alienate the low-paid, it became a symbol for wider concerns about the economy and the rising cost of living, which Mr Brown was remarkably slow to understand.

The toff who leads the Tory party (how some in Team Brown still see David Cameron) got there first. For months he has noted how much more it costs to fill up the family car. Images matter. Voters can probably imagine Mr Cameron at the pumps. When Mr Brown eventually talked about prices at the supermarket checkout, it didn't ring so true.

Yesterday, the Tories hailed Crewe as "a sea change", the end of New Labour and the start of "a positive coalition for change". They may be right, but we can't be sure. The landmark by-election shows that people are prepared to vote Tory to give Mr Brown a kicking. That doesn't mean they see the Tories as a government-in-waiting, the next task for Mr Cameron. Rightly, he has no intention of rushing out an encyclopedia of policies from Angling to Zoos. With Labour showing signs of imploding, he is tempted to bide his time. But he should also use this platform to display some of his most important wares. To win a general election, he will need positive support, not merely a protest vote.

In the short term, the pressure is mounting on Mr Brown. To lift the morale of the Parliamentary Labour Party off the floor, he will need to do the same with Labour's poll ratings, which won't be easy. To me, Labour MPs seem as depressed as in the darkest days of their 18 wilderness years after 1979. That's serious when Mr Brown doesn't need to hold a general election for two years, and reveals a remarkable lack of confidence in him.

You can't have a serious discussion with a Labour MP these days without the question arising: should the party drop its pilot? The media doesn't need to stir the pot; the stirring is being done by jittery backbenchers. Crewe has increased the panic. By-elections come and go, but most Labour MPs know such a seismic shift in a seat held by Labour since the Second World War cannot be dismissed as a mid-term blip.

"We're not a party which does regicide; we didn't get rid of Michael Foot," one senior Labour MP told me yesterday. "But it could happen, and if it does, it could happen very quickly."

Another said: "We have hit rock bottom. There comes a point where you can't fight back because the voters have made up their minds about you. Crewe suggests we have now reached that point."

How might it happen? Probably not a "stalking horse" leadership challenge to Mr Brown, which he might well see off. Nor would there be a round-robin signed by 50 or 70 backbenchers saying the Prime Minister had lost their confidence. What the backbench plotters have in mind is private pressure on Mr Brown to stand down by his own Cabinet. "The Cabinet has a responsibility, a duty, to the party," one said.

The three key figures are seen as Jack Straw, the Justice Secretary, Alan Johnson, the Health Secretary, and Geoff Hoon, the Chief Whip, greybeards Mr Brown would find hard to ignore if they told him the game was up. The plotters claim that, privately, a majority of the Cabinet doubt that Mr Brown can turn it round. But, for now, there little sign of cabinet ministers pushing him out, whatever their private doubts.

Why? There is no obvious successor waiting in the wings, let alone one certain to transform Labour's fortunes. When Labour backbenchers staged an attempted coup to force Tony Blair to name his retirement date in 2006, they had a king over the water in Mr Brown. The absence of a king now may save Mr Brown. One of his sternest Labour critics puts the chances of ousting him no higher than 20 per cent, even though many of the party's MPs believe Mr Cameron could still be beaten.

The odds may change. Mr Brown probably has until the Labour conference in September to show signs of direction, strong leadership and opinion poll progress. That will be make-your- mind-up time for his critics. Yesterday, the Prime Minister insisted he is the right man to steer Britain through difficult economic times. He will need to do better than that, because Crewe suggests many voters have reached the opposite conclusion.