The Prime Minister said "wider questions" had been raised about the welfare system.
ITV News Political Correspondent Libby Wiener reports:
Senior Liberal Democrats have not entered the row which has seen David Cameron back Chancellor George Osborne's comments that the case prompted "wider questions about our welfare system."
Mr Alexander said: "The Philpott case is an individual tragedy. Children have died in that case.
"I think that is where we should let that case lie. I would not want to connect that to the much wider need to reform our welfare system."
The Prime Minister agreed with the Chancellor that the Philpott case prompted "wider questions about our welfare system".
The Prime Minister said the Chancellor's remarks about the Philpott case "were absolutely right".
He said: "I think what George Osborne said was absolutely right.
"He said that Mr Philpott was the one to blame for his crimes and he should be held responsible but what the Chancellor went on to say is we should ask some wider questions about our welfare system - how much it costs and the signals that it sends.
"We do want to make clear that welfare is there to help people who work hard, it shouldn't be there as a lifestyle choice and I think that's entirely legitimate."
The Tory Party Chairman has defended the Chancellor over remarks in which he appeared to suggest state handouts were partly responsible for the lifestyle of child killer Mick Philpott.
Grant Shapps told me this morning that it was Labour's Ed Miliband and Ed Balls who were "out of touch".
Speaking outside the Tory party HQ where he was launching a local election poster, he said: "This debate is raging in the country. For politicians to think they can just step away from it, not be involved in that debate even though everybody is talking about it, is completely out of touch."
But he conceded that the numbers of people with 11 children, who claimed benefits, was small.
Conservative Party chairman Grant Shapps denied the Chancellor was exploiting the deaths of the six Philpott children for party-political advantage by linking them to the welfare debate.
It is deeply irresponsible for such a senior politician to seek to capitalise on public anger about this case, and in doing so demonise anybody who receives any kind of welfare support. Mr Philpott should be held fully accountable for his awful actions and it is reprehensible to seek to explain it away by blaming the welfare system which Osborne has been so happy to wage war on.
– Sarah Teather, Liberal Democrats
On Tuesday, when answering a question about living on £53 a week, Osborne said that it's not sensible to reduce the debate to an argument about one individual's set of circumstances. It makes you wonder what has changed in 48 hours.
Shadow Chancellor Ed Balls launched a scathing attack tonight on what he called the "cynical, nasty and divisive" way George Osborne linked the Philpott case with the broader issue of state benefits.
Mr Balls said the "desperate" Chancellor had offended millions of hard-working people and was playing politics with a tragic case for his own political gain.
The shadow chancellor Ed Balls has condemned the "nasty and divisive" comments by George Osborne in which he questions why taxpayers' money was being used to "subsidise lifestyles" like Mick Philpott:
I believe George Osborne's calculated decision to use the shocking and vile crimes of Mick Philpott to advance a political argument is the cynical act of a desperate Chancellor.
Our main thought at this time should be about the six children who tragically lost their lives, and the others in the family who have been left to mourn their loss.
We should have a proper debate about welfare reform.
And we should discuss what action needs to be taken to tackle the scourge of long-term unemployment including the need for a compulsory jobs guarantee so that people cannot languish on the dole for years and years on end.
But for the Chancellor to link this wider debate to this shocking crime is nasty and divisive and demeans his office.