Speaking to our reporter, Liz Wickham, the travellers say they're Irish and have nowhere else to go.
The six caravans are parked on football pitches, preventing some of the regular Saturday morning matches from being played.
More top news
The plaque at number 16 Cowley Street in Westminster marks the brown brick Georgian townhouse where Sir John lived for 31 years.
A man reportedly armed with several knives has been arrested in Whitehall on suspicion of planning a terror attack, police have said.
Metropolitan Police officer, Tom Harrison, has been crawling the course since the race began on Sunday morning.