Lib Dem Care Minister Norman Lamb has said the evidence was "clear" that the controversial "spare-room" policy was not working.
He told BBC2's Newsnight that in his North Norfolk constituency he came across "many cases" where people may be willing to move, but the state of the housing market and the shortage of social housing just makes it impossible.
"So they are stuck in a situation," he said.
"It was also wrong for disabled people who needed an extra room for a carer to use or to be adapted in some way to be hit".
The Liberal Democrats called for a reform of the bedroom tax in a dramatic withdrawal of support for the controversial welfare reform.
The deputy chief executive, at deafblind charity Sense has welcomed the Liberal Democrats' "change of heart" over the bedroom tax issue:
Mr Richard Kramer said:
It is now time for the rest of government to reconsider this disastrous policy. This policy has unfairly targeted disabled people, causing them undue hardship and risks marginalising them from society.
This tax has had a disproportionate impact on disabled people as many have been found to have a so called 'extra room' despite requiring it because of their disability.Alongside other benefits being cut, housing benefit has been the final blow for many disabled people and can lead to rent arrears and serious financial pressures.
The Liberal Democrats have called for a reform of the "bedroom tax" in a dramatic withdrawal of support for the controversial welfare reform.
Speaking to The Mirror, Chief Secretary to the Treasury Danny Alexander said it's time to "take stock and change our approach."
With only one in 20 of affected claimants having successfully downsized, it’s clearly time to take stock and change our approach in this particular area.
Those already in the social rented sector would only see a reduction in their housing benefit if they are offered a suitable smaller home and, crucially, turn it down.
Disabled adults should be treated in the same way as disabled children, by permanently exempting them.
Almost 60% of households affected by the "bedroom tax" changes were in arrears as a result of the cut to their housing benefit, an internal Government review has found.
Under the policy, social tenants deemed to have a spare room see their rent eligible for housing benefit reduced by 14%, rising to 25% if they have two or more extra bedrooms.
Some 20% of those affected had paid none of the shortfall and 39% had only paid their landlords part of the money owed, the interim report for the Department for Work and Pensions (DWP) found.
The report found 522,905 households were affected by the policy in August 2013, which equates to 11.1% of social tenancies. Some 57% of claimants were cutting back on household essentials and 35% on non-essentials in order to pay their shortfall.
The grandparents of a severely disabled teenager have lost their legal challenge against the so-called "bedroom tax".
Paul and Susan Rutherford, from Pembrokeshire, are full-time carers to 13-year-old Warren, who suffers from a rare genetic disorder which means he is unable to walk or talk and cannot feed himself.
The case concerned regulations, introduced in April 2013, which have led to reductions in housing benefit payments to tenants in social housing assessed through controversial "size criteria" to be under-occupying their accommodation.
Mr Justice Stuart-Smith, at London's High Court, rejected their challenge to the lawfulness of the regulations and dismissed the application for judicial review.
A father of a disabled boy who will have to pay money back under the "bedroom tax" has hit out at the "dreadful" scheme.
Jimmy Daly from Stoke told Daybreak he was moved into his current two-bedroom home to help him cope with his son's disabilities, but because his son does not live with him full time, Jim has to pay some money back.
"I was moved in here because of his disabilities, and now they're saying because your son is disabled you have to pay this dreadful tax. It's wrong."
The bedroom tax has been "mismanaged from the start" and has failed to free up larger properties, according to Labour.
Shadow welfare minister Chris Bryant said:
This ill-conceived policy has been mismanaged from the start.
It's failed to free up larger properties because there simply aren't smaller properties to move to.
It's hitting the most vulnerable hardest, and failing to save the money that David Cameron had predicted.
The Government has no reason to keep it - if they won't repeal it, a Labour government will.
The bedroom tax has failed to save the taxpayer £115 million of the £445 million that was expected, a leading social policy charity has found.
A report by the Joseph Rowntree Foundation said savings from the so-called bedroom tax - a reduction in housing benefit for social housing tenants deemed to have more rooms than they need - were lower than expected but fewer people had been affected.
Around 498,000 people had been affected altogether, the report said, compared with an initial Government estimate of 660.000.
Report author Steve Wilcox said: "There are options to alleviate the worst effects of the policy - particularly in the provisions for people with disabilities, bedrooms too small to share, and those unable to move.
"The savings from the size criteria are modest, and will decline over time, but they have been achieved at disproportionate costs for tenants and landlords."
After a committee of MPs said the bedroom tax unfairly affects disabled people, Tim Farron says his party will push for change.Read the full story ›
The Liberal Democrats wants to see the so-called 'bedroom tax' changed so that people do not face moving out of their home with nowhere to go, the party's president has told ITV News.
Current policy means social housing tenants who have a spare room face having their housing benefit reduced.
Speaking to Deputy Political Editor Chris Ship, Tim Farron said Nick Clegg shared "the view that this is something that we want to see changed".