John Whittingdale's role in press regulation has been compromised and the public "can't have faith" in it, according to campaign group Hacked Off.
The group believes the Government's policy on the press hacking inquiry has even "reversed" under Mr Whittingdale's tenure as Culture Secretary.
Under his stewardship the Government has failed to carry out a promise to look into an alleged press hacking and police cover up, Hacked Off's Executive Director said.
Dr Evan Harris' comments came after Mr Whittingdale revealed on Tuesday he had a relationship with a prostitute.
He claimed he did not know her real occupation and ended the relationship as soon as he found out when he was approached by a journalist in February 2014.
But now questions have been raised over Mr Whittingdale's role in regulating the press when he knew that newspapers had been aware of the relationship for some time.
- Government "reversed" hacking inquiry policy
Hacked Off claims the Government failed to let the Leveson inquiry finish its work, while it reneged on a promise to enact laws designed to give press victims better access to justice.
Dr Evans said official policy had reversed to the extent that Mr Whittingdale was "personally blocking them from taking effect".
Hacked Off's Brian Cathcart told BBC2's Newsnight that since becoming Culture Secretary with responsibility for the media, Mr Whittingdale had taken a number of decisions which had been welcomed by the press.
"The public cannot have faith in his judgment, in his independence in making decisions about the media", he said.
"It is not a story about John Whittingdale's private life. It is a story about why the press didn't cover this".
- Press held a "sword of Damocles"
Labour shadow cabinet minister Chris Bryant, who was previously shadow culture secretary, said it appeared the press were "quite deliberately holding a sword of Damocles" over Mr Whittingdale.
Speaking to the BBC, he said: "He has a perfect right to a private life but as soon as he knew this he should have withdrawn from all regulation of the press".