Resident, Alan Cowburn said: "It seems they are just trying to drive me out of my home. The smell is awful on some days and we get a lot of dust.
"The lorries are so large they have to back in because they can't turn around at the bottom and we get beep beep beeping all day long sometimes from six o'clock in the morning."
"We've even had lorries coming in at midnight. It's so much dust and in the winter it's all mud. And you get a lot if flies and the odd rat running around so all in all it's a total nightmare. The house next door sold recently and they knocked £25,000 off the price."
More top news
The plaque at number 16 Cowley Street in Westminster marks the brown brick Georgian townhouse where Sir John lived for 31 years.
A man reportedly armed with several knives has been arrested in Whitehall on suspicion of planning a terror attack, police have said.
Metropolitan Police officer, Tom Harrison, has been crawling the course since the race began on Sunday morning.