Two Conservative councillors have quit the party after the leader of Windsor and Maidenhead Borough Council called for rough sleepers to be cleared from the streets before the royal wedding in May.
It comes after leader Simon Dudley survived a no-confidence vote on Monday night.
Mr Dudley told ITV News Meridian he now wants to get on with the job of making the area "an even better place for every resident".
Councillors Paul Brimacombe and Asghar Majeed announced their resignation from the council's Conservative group as a result of the comments and are continuing as independents.
They said in a statement: "It was not considered appropriate by them to wholeheartedly disagree with the will of the group on this fundamental issue and yet to remain within the group."
Mr Dudley's comments were made at the beginning of this month.
He suggested rough sleepers may present the town in an "unfavourable light".
Whilst Mr Dudley has not expressed regret for his comments he said last night he wanted to "look after every member of our community".
Despite the council leader successfully coming through the vote there are still calls for him to resign.
Murphy James, from the Windsor Homeless Project, said "he should recognise the mistake he's made and take responsibility for it".
Mr Dudley might still lose his job over the coming days.
He faces another no confidence vote at a full council meeting next week.