The head of Theresa May's policy unit has come under fire and been accused of dragging the Tories into the "gutter" after defending changes to disability benefits.
George Freeman said personal independence payments (PIP) should go to "the really disabled people who need it" as opposed to those "taking pills at home, who suffer from anxiety".
He defended government reforms to PIP, saying they were needed to change the "bizarre" decision of a tribunal, which said claimants with psychological issues who can't travel alone must be treated like those who are blind.
The tribunal also said claimants who need help taking medication should be assessed the same way those manage therapies like dialysis at home.
Responding to the upper tribunal rulings on Thursday, disabilities minister Penny Mordaunt said she was reforming the payments to "restore the original aim of the benefit".
Ms Mordaunt said no claimants would see a reduction in the amount of PIP previously awarded.
But Labour said that the government's equality assessment showed that 160,000 people would miss out on money that's "rightfully" theirs.
Mr Freeman, head of Number 10 Downing Street policy board, said it was the right decision.
He told Pienaar's Politics on BBC Radio 5 Live: "These tweaks are actually about rolling back some bizarre decisions by tribunals that now mean benefits are being given to people who are taking pills at home, who suffer from anxiety.
"We want to make sure we get the money to the really disabled people who need it."
When he was challenged on anxiety, Mr Freeman said: "I totally understand anxiety and so does the Prime Minister. We've set out in the mental health strategy how seriously we take it.
"My point was that these PIP reforms are partly about rolling back some frankly bizarre decisions in tribunals which have seen money that should go to the most disabled spent on people with really much less urgent conditions."
Shadow chancellor John McDonnell said it was "an insult".
In a statement, he said Labour would push the Chancellor to reverse the changes in next month's Budget.
Mr McDonnell said: "This is a return to the worst politics of spin that so tarnished our politics for so long. It is an act of immense bad faith. She is degrading politics and demeaning the role of Prime Minister."
Reacting to Mr Freeman's comments, Labour MP Louise Haigh tweeted that the Conservatives are "in the gutter".
The Prime Minister has made tackling mental health a priority for this government, saying last month that mental illness is "shrouded in a completely unacceptable stigma and dangerously disregarded as a secondary issue to physical health".
Disability charity Scope criticised the "crude" distinction between physical and mental health, and said it was concerned about "worrying" changes to PIP.
Scope chief executive Mark Atkinson said: "It is unhelpful to make crude distinctions between those with physical impairments and mental health issues because the kind of impairment someone has is not a good indicator of the costs they will face.
"Many disabled people will be now be anxiously waiting to hear as to whether or not these tighter rules will affect their current PIP award.
"The Government must offer clarity and reassurance that these new measures will not negatively affect the financial support that disabled people receive now or in the future, and that they stand by their commitment to making no further changes to disability benefits in this Parliament."
Mr Freeman tweeted later reiterating that the government was "committed to tackling mental illness".
Ms Haigh later called on Mr Freeman to resign unless he apologises for his comment.
She posted on Twitter: "Given (the) PM's commitment to mental health surely she can no longer rely on adviser who dismisses serious issues as 'not really disabled'.
"If Freeman doesn't apologise he surely has to resign."