But Scolari insisted his team were destined to win the World Cup before the tournament started and he is sticking by his words.
"Do we have one hand on the Cup? Yes," the Brazilian said in a fiery press conference on the eve of the match.
"Don't forget those statements (I made about winning the World Cup before it started). They were all fantastic. Those statements could not be any different now.
"We are going on to the fifth step now. There are seven in total."
Brazil go into the match with criticism from some of the nation's biggest names still ringing in their ears.
Zico and Carlos Alberto have led the criticism of Scolari's squad in recent days.
Of particular annoyance to the two greats is the outpouring of emotion shown by the current squad at this World Cup.
Neymar, Julio Cesar and David Luiz all broke down in tears after the shootout win against Chile while Neymar also looked overcome with emotion when he sang the national anthem earlier in the tournament.
Carlos Alberto, Brazil's 1970 World Cup-winning captain, said recently: "The team is crying when they're singing the national anthem, when they get hurt, when they take penalties. Stop crying!"
Brazil's psychologist Regina Brandao visited the squad at their Teresopolis base on Tuesday, but Scolari denied that was anything out of the ordinary.
"It was scheduled and organised. Please stop thinking (otherwise)," the former Chelsea manager snapped when asked about Brandao's visit.
"I admire psychologists. She has participated in a very nice way. That's it. The players love to talk. She isn't even paid anything to do this."
Next in Scolari's sights were a bunch of disgruntled Brazilian journalists who questioned why they were excluded from a briefing with him at the training ground earlier this week.
He said: "I can't talk to all of you. I just sat down with a few people. If you don't like it, go to hell."
The one thing Scolari has to be happy about is the fact that Neymar is fit and ready to take on Colombia.
The Barcelona star suffered knee and thigh injuries during the bruising battle against Chile, but he is ready to go.
Scolari will have to make at least one change for the quarter-final though as Luis Gustavo is suspended.
The 65-year-old is likely to recall Tottenham midfielder Paulinho although he could draft in Napoli defender Henrique instead.
Scolari is wary that whatever team he picks, his men will come up against the form team of the tournament so far.
Colombia swept through their group with three wins and they easily saw off Uruguay at the Maracana on Saturday to qualify for the last eight.
They also have James Rodriguez, the tournament's top scorer, and the in-form Juan Cuadrado, but Scolari has no specific plans to stop the duo.
"There will not be any special marking on James Rodriguez nor on Cuadrado," he added.
Overall, Scolari is braced for a tough match against Colombia.
"Chile had strength, they played with spirit and they have a game dynamic which is very difficult to play against, but Colombia is a much better team, and they play good football," Scolari added.