Iain Anderson, who was named by Boris Johnson as Whitehall's LGBT+ business champion in 2021, announced his support for Labour leader Sir Keir Starmer on Monday.
Mr Anderson had been a member of the Conservative Party for almost 40 years and is a known friend of former prime minister Liz Truss.
"It’s not the party it used to be - I can’t defend it," he told the Financial Times.
Mr Anderson said he was leaving the party due to its "relationship with business", claiming its approach looked to mirror Mr Johnson's alleged "f*** business" comment, which he reportedly made in relation to Brexit.
He added: "I’ve tried to understand why a Conservative leader could say that and nobody in the party blinked an eyelid about it."
A recent meeting with Sir Keir about business policy convinced Mr Anderson to switch sides of the political divide, who added that the Labour leader will "do what he says".
Last April, he quit his role within the government, alleging that ministers were "trying to drive a wedge" between trans and lesbian, gay and bisexual people.
"Britain needs a strategy for trans people and I can’t see one at the moment," he told ITV News.
"We have a tabloid debate going on about people’s lives. It’s not a respectful debate, it's turned into a woke war. It's turned into a wedge issue... I was LGBT Business Champion not LGB or T, and that’s why I’m walking away."
Want a quick and expert briefing on the biggest news stories? Listen to our latest podcasts to find out What You Need To Know