The Labour peer Baroness Morgan of Huyton accused David Cameron of secretly trying to pack public bodies with Conservative Party supporters.
Her claims come as she was told she would not be appointed for a second three-year term as the chairman of the schools inspectorate, Ofsted.
"I am the latest of a fairly long list of people now who are non Conservative supporters who are not being re-appointed. I think there is absolutely a pattern. It's extremely worrying," she told the BBC Radio 4 Today programme.
Lady Morgan, who has agreed to stay on until the autumn while a successor is found, stressed she had not had any falling out with Education Secretary Michael Gove, and that the impetus behind the appointments appeared to come from No 10.
"There is a lot of concern about it. Often they are people who have been working really well with their organisations and, indeed, with their host departments, so I do think this is coming from No 10. I don't think it is coming from individual departments," Lady Morgan added.
More top news
Figures from the Local Government Association show 60,000 households will be affected by plans to end subsidised rents for high-earners.
Crime figures show a sixfold increase in people reporting a rape by someone they met on a dating app or website between 2009 and 2014.
The launch was made in defiance of United Nations sanctions barring North Korea from using ballistic missile technology.