David Hoare has resigned as chairman of education watchdog Ofsted just weeks after causing controversy by branding the Isle of Wight a "ghetto" where there was "inbreeding".
David Hoare said he had informed Education Secretary Justine Greening that he was resigning with immediate effect.
The former banker's resignation comes just weeks after he was forced to apologise for his comments about the Isle of Wight.
"Most people go there for sailing for two weeks a year. There's a sailing club that is one of the best in the world, where there's champagne," he said in the recording, obtained by the Times Education Supplement.
"But just within inches, there are people who live in a ghetto ... They think of it as holiday land. But it is shocking. It's a ghetto; there has been inbreeding."
He later issued an apology after his comments were reported by the TES, which prompted calls for his resignation.
Mr Hoare also made a telephone call to the leader of Isle of Wight Council, Jonathan Bacon, to make a personal apology, during which he offered to visit the island.
In a statement announcing his decision to quit, Mr Hoare said: "I have today informed the Secretary of State that I will be resigning from my position as chair with immediate effect.
"It has been a great privilege to chair the Ofsted board for the past two years. I am pleased that the organisation now has an excellent board in place with expertise across all of our remit areas, including early years and further education."
He added: "I will miss working with an excellent team, making a real difference."
An official reason for his resignation has not yet been given.
Education Secretary Justine Greening said in a statement: "I have accepted David Hoare's resignation from the post of Chair of the Board of Ofsted.
"I would like to thank David for his hard work in this role over the past two years."
Neil Leitch, chief executive of the Pre-school Learning Alliance, said: “Mr Hoare’s comments about the Isle of Wight were offensive, thoughtless and damaging, and it is right that he has stepped down from his position as Ofsted chair.
“Ofsted plays a pivotal role in supporting the provision of high quality education services in all areas of the country. We hope that whoever replaces Mr Hoare takes this responsibility more seriously."