British Athletics' Ed Warner says he expected the bulldozers to be in the Olympic stadium by now.
Warner admits the three-day London 2012 anniversary meet in the stadium in July is "opportunistic".
Warner says the event is probably the last chance to see sport in the stadium until 2016 - with the possible exception of rugby.
Organisers of the 2015 Rugby World Cup are keen to use the stadium but may lose out if West Ham moves in.
WHUFC and the London Legacy Development Corporation are reported to be close to a deal with an announcement expected on Tuesday.
If West Ham is successful, conversion work may mean the stadium is a building site in 2015 and the rugby will not happen.