Andy Sawford's victory was the highpoint for Labour in a day of results which was otherwise dominated by the poor turnout in elections to the newly-created post of Police and Crime Commissioner (PCC) across England and Wales.
The Electoral Commission watchdog announced a review of the PCC elections, warning that record low participation levels of well below 20pc were "a concern for everyone who cares about democracy".
Conservatives sought to portray their first loss of a Westminster seat to Labour in a by-election for 15 years as a case of "mid-term blues", and cast some of the blame on former MP Louise Mensch for resigning to be with her family in the USA.
Mr Cameron said: "It's a classic mid-term result and obviously made difficult by the fact that the Conservative MP left the seat in question."
But a jubilant Labour leader told supporters: "Middle England is turning away from David Cameron and the Conservatives because Middle England feels let down by David Cameron and the Conservatives."
Elsewhere, there were mixed results for Mr Miliband's party, which won Westminster by-elections in the safe seats of Cardiff South and Manchester Central, but was beaten in the race for Bristol mayor by independent George Ferguson and also lost the North Wales PCC election to an independent, former judge Winston Roddick.
Former Labour MPs Tony Lloyd, Jane Kennedy, Alun Michael and Vera Baird won PCC polls in Greater Manchester, Merseyside, South Wales and Northumbria. But the party's most prominent candidate, former deputy prime minister Lord Prescott, faced a nail-biting finish in Humberside after securing only 25pc of first-preference votes to go into a second-round face-off with Conservative Matthew Grove.