The Labour peer Baroness Morgan of Huyton accused David Cameron of secretly trying to pack public bodies with Conservative Party supporters.
Her claims come as she was told she would not be appointed for a second three-year term as the chairman of the schools inspectorate, Ofsted.
"I am the latest of a fairly long list of people now who are non Conservative supporters who are not being re-appointed. I think there is absolutely a pattern. It's extremely worrying," she told the BBC Radio 4 Today programme.
Lady Morgan, who has agreed to stay on until the autumn while a successor is found, stressed she had not had any falling out with Education Secretary Michael Gove, and that the impetus behind the appointments appeared to come from No 10.
"There is a lot of concern about it. Often they are people who have been working really well with their organisations and, indeed, with their host departments, so I do think this is coming from No 10. I don't think it is coming from individual departments," Lady Morgan added.
More top news
Tougher regulations coming into effect today are intended to crack down on motorists driving under the influence of drugs.
A Finnish punk band whose members have autism and Down's Syndrome have been chosen as the country's entry for the Eurovision Song Contest.
The PM promises to double to 200,000 the number of homes built under an existing scheme aimed at easing Britain's housing shortage.