Daniel Grainger, the chairman of Young Conservative Network (YCN), was told to leave and “take anyone with the same views with you” by Andy Street after sharing the remark on Twitter.
Mr Grainger later apologised and deleted the post – which had read: “Birmingham is a dump.” – claiming it “was not about the city” and that he had been the victim of a mugging attempt earlier on Saturday.
The city is hosting the Conservative Party’s annual conference this week.
In a later tweet, Mr Grainger wrote: “My tweet in relation to Birmingham was not about the city or its people. I’ve always enjoyed my visits to your city, but this morning I was greeted by an individual who threatened me with a mugging.
“I was angry and tweeted without thinking, I apologise for any offence caused.
“I’d also like to directly apologise to (Andy Street), the (Conservatives) and other representatives of Birmingham for my ill conceived comments. I have now deleted the tweet.”
Mr Street and Labour MP for Birmingham Yardley Jess Phillips were among those to pour scorn on the comment.
“Off you pop then, and take anyone with the same views as you. Try and take some time to grow up and understand just why this city is so special whilst you’re gone,” the mayor wrote.
Ms Phillips tweeted: “I see the Tories doing their best to endear themselves.”
Mr Street added: “I may be a Conservative, but I’m also a bloody proud Brummie. I won’t have anyone who’s never lived or breathed this place trying to put us down.”
The exchange comes against a backdrop of wider divisions emerging within the Tory party, after the Chancellor’s mini-budget prompted market turmoil.
In an article for The Daily Telegraph, Kwasi Kwarteng claimed he will be setting out a “credible plan” to get the public finances back on track with a “commitment to spending discipline”.
Writing in The Sun, Prime Minister Liz Truss insisted the Government was right to act and will keep “an iron grip” on the national finances.
But the move has come under attack from prominent backbenchers who are concerned about the pound hitting a record low and the impact the financial fallout could have on them at the ballot box.
Tory MP Robert Largan labelled as a “mistake” the decision to cut the top income tax rate while colleague Simon Hoare, chairman of the Northern Ireland Select Committee, branded the financial plans “inept madness”.