Video report by ITV News Correspondent Richard Pallot
The company said a committee of directors established to look into the claims had been made aware of "further serious allegations" regarding Mr Kelvin.
Ted Baker said Mr Kelvin had agreed to go on leave "for the benefit of the business and the people who work in it".
Accusations were also made that he massaged employees, kissed their ears and asked some to sit on his lap.
Mr Kelvin said in a statement the accusations had "raised some very serious and upsetting issues".
He said: "It is only right that the Board and HSF should investigate fully.
"Ted Baker has been my life and soul for 30 years."
He added: "I love this company and I care deeply for all my colleagues.
"It’s for that reason that I have decided to take a temporary leave of absence.
"Ted Baker means everything to me and I can’t bear to see it harmed in any way."
In Mr Kelvin’s absence, chief operating officer Lindsay Page has been appointed interim CEO.