Former Tory minister Damien Green has described Norman Baker as a "natural lone wolf, not a natural team player".
He said Baker did not like the fact he had to work under Home Secretary Theresa May and wanted to clear everything himself.
"I think he maybe he had unrealistic expectations of what his job was", Green added.
One of the reasons Lib Dem Home Secretary Norman Baker stepped down is to concentrate on his passion for music.
The former minister is in a band called The Reform Club, for which he is "lead singer and lyricist", and even released a pop single Piccadilly Circus last year.
In his resignation letter to Lib Dem leader Nick Clegg, Baker said his time as minister had "squeezed the time available for my family and outside interests, including my music".
He will still continue in his role as MP for Lewes.
Here is the video for The Reform Club's single Piccadilly Circus:
Former Tory former Damian Green has accused Norman Baker of trying to act as if he had the same ministerial rank as Home Secretary Theresa May.
The ex-Home Office minister told BBC's Radio 4 Today programme: "Norman's problem was he came in ... he gave an interview to his local paper saying 'I'm the Lib Dem home secretary'.
"He regarded himself as being on a par with the Home Secretary and asked for papers from other ministers, he wanted to check what everyone else was doing.
"The world doesn't work like that. If you are a minister of state, in the end you work for the Secretary of State in that department."
Nick Clegg has said he will announce Norman Baker's replacement shortly after calling him an "outstanding" minister.
Labour have claimed the resignation of Norman Baker shows Theresa May is "losing control" of the Home Office.
Shadow Home Office minister Diana Johnson said:
More chaos at the Home Office. Alongside the immigration shambles, the failed experiment of Police and Crime Commissioners and the failure to undertake basic checks into chairs of the child abuse inquiry, it's clear Theresa May is losing control of her Department.
Nick Clegg has written to Norman Baker thanking him for his "brilliant" service as a minister, in spite of well-documented difficulties with his Tory coalition colleagues.
The Deputy Prime Minister wrote:
Thank you for the brilliant job you have done as a Minister over the past four and a half years, first at the Department of Transport and more recently at the Home Office.
However complex the issues have been, or challenging the coalition relations have proved to be, you have handled the political relationships within Government with great skill, always focusing on how to achieve liberal reform wherever you can.
Norman Baker has cited a "lack of good will" at the Home Office as one of the reasons for his resignation as a minister.
His resignation letter to Nick Clegg also reveals that he first raised the prospect of leaving as a minister as early as August, before a recent row with the Tories over drug policy.
In the letter, Mr Baker said he was proud to have brought through initiatives on areas such FGM and animal testing.
But he said the Conservatives' attitude had made it hard to work together on policy.
"In stark contrast to the Department for Transport, I regret that in the Home Office, the goodwill to work collegiately to take forward rational evidence-based policy has been in somewhat short supply," he wrote.
Nick Clegg has said he is "extremely grateful" for Norman Baker's service in the Home Office.
A spokesman for the Deputy Prime Minister issued a statement referencing Mr Baker's attempts to pursue "liberal reforms" in the department.
A spokesman for the Deputy Prime Minster said:
“Norman Baker has decided that now is the right time to step down from Government as a Minister after four and a half years to concentrate on his constituency, his family and his great passion for music.
“The Deputy Prime Minister is extremely grateful for the skill and determination shown by Norman Baker in pursuing liberal reforms during his time as a Minister in the Department for Transport and Home Office.”
Norman Baker has resigned his role as a Home Office minister, saying working with Home Secretary Theresa May was like "walking through mud".
The Lib Dem MP accused May of looking on her coalition colleagues as "a cuckoo in the nest rather than part of government".
His decision comes days after he became embroiled in a fresh disagreement over drugs policy and accused the Tories of suppressing a report which backed his case for a review of the current law.
Baker told the Independent: "They have looked upon it as a Conservative department in a Conservative government, whereas in my view it's a Coalition department in a Coalition government.
"That mindset has framed things, which means I have had to work very much harder to get things done even where they are what the Home Secretary agrees with and where it has been helpful for the Government and the department.
"There comes a point when you don't want to carry on walking through mud and you want to release yourself from that."