Justice Secretary Chris Grayling has said he is not convinced by the case for an immediate judge-led inquiry of the kind Harriet Harman has proposed.
Mr Grayling told BBC1's Andrew Marr Show: "There is always a danger if you set up a very substantial inquiry process of that kind that it takes much longer to get to the truth.
"What should be happening right now first and foremost...is we should be looking to see who is still around who was involved, and criminal proceedings should follow if people were guilty of participating in these offences alongside Jimmy Savile. That is of paramount importance."
Mr Grayling accepted that the case indicated that the authorities did not pay enough attention to the complaints of abuse victims in the past.
"I hope every single person in the law enforcement world today is accepting the fact that things went badly wrong and it should not happen again," he said.
Mr Grayling added: "Clearly what has happened is absolutely horrendous. It is shocking. There was clearly a culture that should never, ever, ever have been allowed to exist."
Labour deputy leader Harriet Harman has said there needs to be an over-arching independent inquiry into the Savile case.
Speaking to Andrew Marr on the BCC, she said too often the criminal justice system gave the victim the impression they would not be believed if they came forward with allegations of sexual abuse.
Ms Harman said: "There are big lessons to be learned here...
"Not just for the BBC although the epicentre of it was at the BBC, but elsewhere because when something like this comes out, there is an assumption, 'how could we have gone astray from our normal policy that protecting vulnerable children must take priority over the rights of protecting adults'?
"That is not actually the situation because that is always under challenge. Just this month the law has been changed so if there is an allegation of a sex offence against a teacher, the teacher has anonymity right up until the moment of charge.
"That is based on an assumption that you have to protect the adult from false allegation. I think what we do is that we push to protect children but that is always resisted."
Lord Patten, who heads the BBC Trust, has pledged to get to get to the truth following the Jimmy Savile scandal, stating "our public expects no less".
Writing in the Mail on Sunday, Lord Patten says: "even those of us who weren't there are the inheritors of shame".
He adds: "Can it really be the case that no one knew what he was doing? Did some turn a blind eye to criminality? Did some prefer not to follow up their suspicions because of this criminal’s popularity and place in the schedules? Were reports of criminality put aside or buried?"
There was a crucial week at the end of November 2011:
On November 25th, Newsnight editor Peter Rippon sent an email saying he was still very enthusiastic about the investigation.
On 29th November, the BBC Christmas schedules were published.
On 30th November, Rippon sent another email to his team saying he'd had a change of heart and that he was no longer sure the story was strong enough to run.
On 1st December he pulled all editing on the programme entirely, effectively ending the project.
The question is: Did he feel he had to pull the programme because of the tribute programmes? Or was it because of the new information he was given the previous day by the reporter which made him question the project editorially?
The BBC said this will be examined in the investigation run by former head of Sky News Nick Pollard.
The chairman of the BBC Trust Lord Patten has described the Jimmy Savile affair as "an appalling tsunami of horror" which he concedes the BBC is "at the centre" of.
He also admitted to ITV News' UK Editor Lucy Manning that he and other BBC bosses had given misleading statements.
Arthur Sulzberger Jr. the chairman of the company and publisher of The New York Times has voiced his support for Mark Thompson, the incoming president and chief executive, in the wake of the Jimmy Savile scandal.
Mr Thompson was the Director-General when the Newsnight investigation in to Savile was shelved.
In a letter to the Times staff Mr. Sulzberger said, "I want to address a topic that has been on many people’s minds. Mark has provided a detailed account of that matter, and I am satisfied that he played no role in the cancellation of the segment."
Mr. Sulzberger goes on to describe Mr Thompson as having, "high ethical standards and the ideal person to lead our Company."
BBC Trust chairman Lord Patten has said a "great tsunami of filth" broke over the corporation's Director-General, George Entwistle as he tried to deal with the Jimmy Savile scandal just eleven days into his new job.
He told Radio 4's World at One programme that the BBC's main concern "is to get to the bottom of what happened".
BBC Trust chairman, Lord Patten, has told Radio 4's World at One programme that he insisted that Newsnight editor Peter Rippon's blog should be corrected after he was informed of its inaccuracies on Sunday.
He had previously supported Mr Rippon's explanation for dropping the programme's Jimmy Savile investigation.
Should I have disbelieved what a senior editor said? I have at a certain point to believe. I can't disbelieve everything that is said.
The BBC warned Conservative Party chairman, Grant Shapps, against raising damaging allegations over the Jimmy Savile scandal during his recent appearance on Question Time, according to the Daily Mail.
An email leaked to the newspaper, reportedly insisted claims Newsnight dropped an investigation into the affair because of internal pressure were "malicious".
The BBC's head of public affairs, Julia Ockenden, is quoted as having said in the email: "No pressure was applied to drop this investigation. None. To suggest otherwise is to risk impugning the professional reputation and integrity of a number of journalists."
Yesterday, Mr Shapps made a formal complaint to BBC Trust chairman, Lord Patten, the newspaper said.
He is said to have accused the corporation of "trying to provide guidance for an elected representative appearing on a BBC opinion show – guidance which in the event has turned out to be wholly inaccurate"
The BBC's former director-general, Mark Thompson, faces questions over whether he is the right person to take charge of the New York Times newspaper in the wake of the Jimmy Savile scandal.
In an interview with the Guardian, Mr Thompson, who is due to become the paper's chief executive on November 12, said, it is "totally reasonable for institutions like the New York Times and the BBC to be free to examine everything, including subjects of corporate interest in the institution itself".
The New York Times' ombudsman had written that the newspaper must consider if Mr Thompson is right for the job, a move he said was "completely correct".
He also said that it was the BBC's head of news, Helen Boaden, who told him there was nothing about Newsnight's investigation into Savile that should concern him.