Lord Coe, a former athlete and member of the International Olympic Committee, told ITV's Peston that athletes' movements during the Games will be tightly controlled.
He said: "The thought that there are going to be 10,500 athletes descending on the city of Tokyo is actually quite wide off the mark.
"They are going to be going in, very carefully, they are going straight to the village. Their day during their time at the Games is going to be literally village to venue, possibly venue to training track - they're not going to be mixing with local communities."
On Friday, just 50 days before the start of the Olympics, Japan extended a coronavirus state of emergency in Tokyo and other areas with high infection rates.
The current state of emergency in the capital and eight other metropolitan areas was to end on Monday, but hospitals in some regions are still overflowing with Covid-19 patients.
Lord Coe said some concerns about the Games "quite wide off the mark"
Olympic organisers have to yet to decide whether domestic fans will be allowed at the Games, after overseas spectators were banned.
"Even the arrangements in the village, which I'm very familiar with, are very tough," Lord Coe added.
"Athletes are not going to be mixing between countries, the village cafeteria which I remember from London is sort of 10,500 athletes eating in a mingle, which is what the Olympic spirit's about, that isn't going to happen."The Tokyo Olympics are scheduled to start on July 23 after the Games were delayed by a year due to the pandemic.
A plan to prioritise vaccinations for Japanese athletes has been delayed, according to media reports. Just 4% of the population has had at least one dose of a Covid vaccine and only 2.3% of the population has been fully vaccinated, while doctors in Osaka warned this week that hospitals were close to collapse. Japan has lagged on vaccinations due to bureaucratic and planning missteps, along with shortages.
Listen to the ITV News politics podcast, Calling Peston