He told Iain Dale at the Edinburgh Fringe Festival that Labour "would let the Scottish people decide" if the country should be independent because "that's democracy".
He added: "It will be for the Scottish Parliament and the Scottish people to decide that."
But Scottish Labour MP Ian Murray branded the comments "ludicrous", saying it was a "thoroughly dreadful way to try and change policy".
He added: "I think he should apologise to Richard Leonard and he should apologise to the Scottish Labour Party members."
Mr Leonard, the leader of Scottish Labour, aimed to clarify Mr McDonnell's remarks and the Labour Party policy, saying "we're opposed to a second independence referendum".
He said: "I met with John McDonnell this morning to make clear that not only does the Scottish Labour Party oppose it, the people of Scotland do not want it."
He added: "It's my view that the manifesto that we go into the next general election on will be categorical in saying that we will oppose a second independence referendum, that was the platform we stood on in 2017 and I am determined that will be the platform we will stand on at the time of the next general election."
A Labour party spokesperson also moved to clarify the shadow chancellor's comments, said he "was clearly not advocating a second independence referendum."
The spokesperson added: "He made clear the huge benefit a UK Labour government will bring for the people of Scotland.
"Labour stands for an end to the status quo in the UK: economically, politically and constitutionally."
However, when asked about his comments during another Fringe event, Mr McDonnell reiterated "I'm not in favour of blocking (a second vote), I've said it time and time again in interviews."
But he added: "I've reinforced the view that a referendum isn't the solution to the problems in this country.
"But I don't want to use parliamentary devices to block it."
Following the event Mr McDonnell told ITV News Correspondent Peter Smith that Jeremy Corbyn shares his view: "As the situation is at the moment, we will not block it."
Labour MP Stella Creasy appeared to disagree with McDonnell, tweeting: "Nationalism is antithesis of socialism - to prioritise passports over principles isn't progressive.
"Egalitarian devolution isn't about separation but how by working together as nations within the UK each stronger and more successful. Labour abandons such commitment at our peril."
Mr McDonnell also ruled out an election pact with the SNP, suggesting it is not a progressive party.
He stressed Labour would reject any opportunity to form an alliance, saying "we're a socialist party, they are not."
"I don't want to be derogatory in any form but let me try. In my own view, I think they're Tories, it's as simple as that and always have thought that.
"I think there is a small section of SNP supporters who I think have been illusioned by them in some way that they're a progressive party."
Speaking to the Guardian, Ms Sturgeon said: “I have fought two general elections now as SNP leader, and, in both of them, I have been pretty candid.
“We would always want to be part of a progressive alternative to a Tory government. That remains the case.
“I think in politics you’ve got to be careful. But it would not be my intention, to go into a formal coalition.
She added: "But some kind of progressive alliance that could lock the Tories out of government. It wouldn’t be a blank cheque-type scenario.”