Boris Johnson is "away" with his family in England's West Country, Number 10 has revealed, but his spokesman insisted the prime minister is not on holiday.
The PM headed off for a four-day break away from Downing Street on Sunday, after the final British troops left Taliban-controlled Afghanistan.
But Downing Street insisted the PM was "continuing to work" as Britons and Afghans remained stranded after the withdrawal of forces from Kabul.
His spokesman refused to say Mr Johnson's time away amounted to a holiday in the face of repeated questioning from journalists at Monday's behind-closed-doors media briefing.
Mr Johnson and his Foreign Secretary Dominic Raab both came under fire recently for being on holiday during the weekend in which the Taliban completed its takeover of Afghanistan.
Mr Raab told ITV News on Tuesday that the number of Britons stuck in the country was down to the "low hundreds" but he admitted the number of Afghan allies still there could be much higher.
Defence Secretary Ben Wallace had previously estimated that up to 1,100 eligible Afghans who had aided the British mission would not be evacuated in time.
Number 10 was unable to say what time the PM departed Downing Street, but said it was at some point on Sunday, around a day after the final British troops left Kabul.
He said: "The prime minister is away for the next two days in the west of England but is continuing to work and will be back in the office in Downing Street on Thursday."
Asked what he would say to critics who say Mr Johnson should not go on holiday during such a crucial period, his spokesman said: "I would say the prime minister is continuing to work."
Pressed whether Mr Johnson is working full time, he responded: "The prime minister is continuing to work."