The fixed spectator limits for the event will be 50% of a venue’s capacity, up to a maximum of 10,000.
Fans from overseas have already been banned from the Games but it has been confirmed spectators from Japan will be allowed.
The decision was taken following a meeting on Monday morning involving local organisers, the International Olympic and Paralympic Committees, the Tokyo metropolitan government and the national government.
"In light of the government’s restrictions on public events, the spectator limit for the Olympic Games will be set at 50% of venue capacity, up to a maximum of 10,000 people at all venues," a statement from Tokyo 2020 read.
Listen to our coronavirus podcast:
Dr Shigeru Omi met representatives of the Tokyo 2020 local organising committee on Friday and told them “the best way” to ensure safety amid the pandemic was to hold the Games without spectators.
But organisers have opted to follow the existing government limits for sports events in the country.
The Olympics are scheduled to start on July 23, while the Paralympics follow a month later, from August 24.
Earlier on Monday, Prime Minister Yoshihide Suga said he would not rule out holding the summer Olympics without spectators if the capital was under a state of emergency for Covid-19.
"In the event a state of emergency was declared then we can't rule out not having spectators," Mr Suga told reporters during a tour of vaccination sites in Tokyo.