The couple said in an Instagram post they will not be returning for the new series of the show.
Viewers said that the judges sounded like Daleks.
Williams and Ayda Field are reportedly set to join the panel.
The judge was rushed to hospital on Friday after he took a tumble in his London home.
The singer was allegedly assaulted outside a karaoke bar in Brighton in the early hours of Friday.
The X Factor star appeared to pop the question on the same day gay marriage was legalised in her native Finland.
Former X Factor hopeful Lucie Jones has been named as Britain's entry to this year's Eurovision song contest.
ITV said the move was to "focus our efforts on our digital platforms, in support of the main show."
The boyband were signed to the label a month after fellow X Factor contestant Honey G announced she is to release a single with the company.
The former X Factor hopeful was signed to Simon Cowell's label Syco, but her first single failed to make an impact.