Older people living in a row of bungalows in Prudhoe, in Northumberland, say they feel "hurt" by plans to demolish their homes.
The housing association Karbon Homes, which owns the site, says it wants to build new houses for working families.
It has pledged to find the existing residents suitable places to live, but they fear they won't be as accessible.
'The Haven' is made up of 10 one-bedroom bungalows, rented by older people, just off Front Street in Prudhoe.
Sheena Gibson has lived there for more than 40 years. She is now 81, and has arthritis and other health issues which make it difficult to get around.
"[I am] very annoyed and very angry, and very hurt. Where they want us to go is up a hill, or down a hill, there's nowhere else in Prudhoe as level as what this is."
Karbon Homes is planning a £1.2m redevelopment of the site, to build seven two or three-bedroom houses.
It says existing residents will have good options to choose between for their new accommodation.
"We are really sorry for the disruption this will cause to our residents’ lives.
The social landlord says it will pay residents' moving costs, and anyone who has lived in their home for more than a year is eligible for compensation of more than £6,000.
It says its staff have visited every household in person, and appreciates it may take a year or more for residents to find a suitable home to move to.
Local representatives have also criticised the housing association's approach.
"We need more properties like this in the town centre, for the older generation - not moving them out to the outskirts of town, that's where the families can be.
Karbon Homes says the new houses at 'The Haven' will be available via the Rent to Buy scheme.
It has not yet applied for planning permission.