Firefighters based at the fire station nearby had heard shouting in the street and went out to investigate, where they were confronted and threatened by the man.
The 26-year-old suspect remained hostile and struck the window of one of the first police cars when they arrived.
Eyewitness Jenilee Graham said she was unaware there was a situation developing until a woman stopped her and told her to get away.
The 23-year-old City worker, from south London, said:
"I was coming back from Asda with my mum when we saw a woman who told us to stop where we were going. We didn't know what was happening, but then we saw the man who the woman was talking about pop out from behind a bus. He ran towards us, then he darted down a side street. It was obviously pretty scary."
More top news
A Labour MP tipped as a possible candidate for London mayor has admitted crashing into a motorcylist her second accident in just over a year
The 48-year-old man was held on suspicion of theft from heritage and protected sites.
A Met Police officer was left red-faced after attempting to stop a cyclist riding along a dedicated cycle path.