Tim Farron MP says people having empty second homes in Cumbria is a "real concern".
Following the Queen's speech, it was announced new council tax rules for second and empty homes will be introduced as part of the UK government’s levelling scheme.
Local Authorities in England will be able to double council tax bills for second homes - which are furnished but not occupied as a sole or main residence.
Homes left empty could also see their standard council tax rate doubled sooner - after 12 months rather than two years - which aims to discourage owners from leaving properties vacant for a long time.
Mr Farron said: "My real concern is that you've got rural communities massively struggling at the moment with excessive numbers of second homes, growing numbers of holiday lets, local communities being cleared out of those communities because they simply can't afford to live there or they're even actually evicted because the Government has rules that allows that."
There are currently 72,000 homes that have been left empty for more than two years in England.
The MP for Westmorland and Lonsdale added: "We've heard some stuff about maybe the government allowing councils to double council tax and that would be welcome for second home owners.
"But, let's be clear, 90% of second homes are bought by people who use them as investments.
"They let them out for two or three months, therefore they pay no council tax and they also pay no business rates and somebody going to the food bank in Kendal or Penrith is having to subsidise somebody with a second or third or fourth home in Ambleside or in Appleby and that's not on."