The Labour peer Baroness Morgan of Huyton accused David Cameron of secretly trying to pack public bodies with Conservative Party supporters.
Her claims come as she was told she would not be appointed for a second three-year term as the chairman of the schools inspectorate, Ofsted.
"I am the latest of a fairly long list of people now who are non Conservative supporters who are not being re-appointed. I think there is absolutely a pattern. It's extremely worrying," she told the BBC Radio 4 Today programme.
Lady Morgan, who has agreed to stay on until the autumn while a successor is found, stressed she had not had any falling out with Education Secretary Michael Gove, and that the impetus behind the appointments appeared to come from No 10.
"There is a lot of concern about it. Often they are people who have been working really well with their organisations and, indeed, with their host departments, so I do think this is coming from No 10. I don't think it is coming from individual departments," Lady Morgan added.