- Video report by ITV News Political Correspondent Angus Walker
The former minister is the third former Tory MP to commit to the party in recent weeks, joining ranks with Phillip Lee and Sarah Woollaston.
His admission brings its number of MPs in the Commons to 18 representatives.
Mr Gyimah told the Liberal Democrat conference he has known and admired party leader Jo Swinson for a long time.
He said: "Despite what has been said, the Withdrawal Agreement was the worst of all worlds for the country.
"It was sold to us as a deal, but what I see is if we exit, and we negotiate the future afterwards, we will be doing so with our hands tied behind our back as a country."
On not backing the government and voting to block a no-deal Brexit, he said: "I was disheartened to find that the way the whipping had been framed was for us MPs to choose our careers, in other words our own salaries, over putting the country first."
He went on to criticise the government's agenda, hitting out at Boris Johnson's pledge for 20,000 more police officers on Britain's streets.
Mr Gyimah said: "Essentially, a government that has made law enforcement the centre-piece of its programme, more police officers to make sure its citizens obey the law, is saying to us it will pick and choose which laws it respects.
"That is incredibly dangerous. When you are playing fast and loose with our constitution, when you are playing fast and loose with people's livelihoods, when you are playing fast and loose with people's security, I think it is right that we all stand up.
"But when people stand up, you're called collaborators. You're called saboteurs. You're called mutineers. Something has gone wrong."
In an earlier interview with The Observer, Mr Gyimah said: "It is sad that I find myself at a crossroads. I am an outcast in the Conservative party.
"But that’s Brexit. It has divided families. The country is divided. This is a huge fault line."
Three former Labour representatives have also defected the party; they are Luciana Berger, Angela Smith and Chuka Umunna.