The Labour peer Baroness Morgan of Huyton accused David Cameron of secretly trying to pack public bodies with Conservative Party supporters.
Her claims come as she was told she would not be appointed for a second three-year term as the chairman of the schools inspectorate, Ofsted.
"I am the latest of a fairly long list of people now who are non Conservative supporters who are not being re-appointed. I think there is absolutely a pattern. It's extremely worrying," she told the BBC Radio 4 Today programme.
Lady Morgan, who has agreed to stay on until the autumn while a successor is found, stressed she had not had any falling out with Education Secretary Michael Gove, and that the impetus behind the appointments appeared to come from No 10.
"There is a lot of concern about it. Often they are people who have been working really well with their organisations and, indeed, with their host departments, so I do think this is coming from No 10. I don't think it is coming from individual departments," Lady Morgan added.
More top news
American scientist, who was the co-discoverer of DNA, is selling his Nobel Prize medal at auction and it could sell for up to $3.5 million.
A British paleontologist has discovered a new species of dinosaur while studying fossils in a Canadian museum.
Sebastian Michaelis, one of a small group of tasters employed by Tetley, can grade a variety of tea in just 15 seconds.