Speaking to our reporter, Liz Wickham, the travellers say they're Irish and have nowhere else to go.
The six caravans are parked on football pitches, preventing some of the regular Saturday morning matches from being played.
More top news
A lamp in the tower which houses Big Ben is to be switched off for the first time in more than 70 years.
Sir Bruce entertained audiences for more than seven decades.
Work in the tower will also pause over the bank holiday weekend.