Matt Hancock has said he did not resign immediately after images emerged of him kissing his close aide (and therefore breaching social distancing guidelines) as his first focus was on his "personal life".
"The first thing that I focused on was my personal life," he said.
"I'd blown up every part of my life and I concentrated on my personal life first, as you can probably imagine," he told ITV's Peston programme.
The former health secretary addressed the scandal on Wednesday during his first major interview since resigning from government.
He spoke to ITV News Political Editor Robert Peston about the aftermath of when photos surfaced of him and Gina Coladangelo in a passionate embrace.
Mr Hancock, who is still the MP for West Suffolk, used the interview to apologise again.
"I let a lot of people down, sorry to the people that I hurt," he said.
Asked if he was keen to return to a cabinet position, Mr Hancock said: "I'm not in any hurry".
The former health secretary also addressed accusations his neighbour Alex Bourne won a contract to supply Covid medical equipment after a story in The Guardian newspaper.
Mr Hancock dismissed the reports as "rubbish".
Hancock denied in Parliament on Tuesday that a contract was awarded to Mr Bourne's company, Hinpack.
What appeared to be a screengrab of a contract, posted on Twitter, between the government and a firm called Alpha Laboratories stipulated the manufacturing of equipment had to be by Hinpack.
Robert Peston quizzed the former health secretary, saying he too had seen the contract and that a subclause revealed the link with Hinpack.
"It is inconceivable the department of health was not aware that the subcontract was going to Alex Bourne," Robert Peston told Mr Hancock.
"All these accusations are complete rubbish, people are trying to insinuate there is a problem where there was no problem," Mr Hancock said.
"I had absolutely nothing to do with that contract," the former health secretary said.
He added that there will be an inquiry into the matter and that it won't find any wrongdoing, just "people working hard."
Mr Hancock said the concept of "a great scandal" is "just not true".
"At that time, we were doing everything we possibly could to save lives," he added.
But Labour's Jess Phillips dismissed Mr Hancock's comments as "an absolute load of rubbish" asking the other guests: "Did anyone else's friends get a contract?"
"I watched lorries take PPE equipment out of my constituency while my care homes were ringing me and asking if I could go to the local school and borrow goggles," she said.
"I remember filling in endless contracts for charities for victims of domestic abuse. I've got all the notes of how I helped them and every single bit of rigour that they had to go through to get any public money."
The Labour MP said some contracts awarded by the department of health did not go through the same level of rigour.