Speaking to the Acting Prime Minister podcast, Mr Stewart said if the PM "ends up being toppled" it will be over his response to the Covid-19 crisis.
He said: "What defeated Donald Trump in the end was Covid, what always defeats populists is their inability to manage seriously, to take an interest in the details in a crisis.
"If he ends up in a difficult situation, he ends up being toppled, it will be because of this. It will be because this is exactly the kind of issue that is his greatest weakness."
The former MP, who quit Parliament ahead of the 2019 General Election, added: "He's good when things are optimistic and happy and he can make optimistic and happy statements, encourage people. What he doesn't like is difficult, detailed, bad news."
The former Cabinet minister, who ran against Mr Johnson for the Tory leadership last year, attacked the prime minister over his honesty, saying he "doesn't mind letting people down".
He told podcast host Paul Brand that Mr Johnson "isn't very worried about whether, what he says he's going to do, he's actually going to do".
"On the spur of the moment when he's on a spot, he will say whatever is convenient to that moment.
"Whatever's going to help him with votes. Sometimes he just doesn't tell the truth because he just likes making people happy."
He added: "He doesn't mind letting people down. In fact, in some ways he's got through life, more than 50 years, by doing that again and again, and never really getting in trouble for it".
Mr Stewart, who was formerly the international development secretary, said the move by Chancellor Rishi Sunak to cut the foreign aid budget by 0.2% was a "terrible thing to do".
"The prime minister promised repeatedly that he wouldn't do this - so you've got to keep your promises," he said.
He added: "If you reduce that international development aid, it completely undercuts our ability for example, to do international environment and climate stuff."
Mr Stewart, who had planned to run for London Mayor this year before the Covid crisis postponed the election, was asked if he would ever consider a return to politics.
"Yes, if I was very lucky, I think it would be the most extraordinary privilege," he said.
"But I suspect now at this stage of my career, it would be more a question of whether somebody would want me back in that Cabinet."
Listen to the podcast here: