Mr Orban said he understood why his comments about "Muslim invaders" were offensive but added it was "the fact".
The meeting was criticised by opposition figures and human rights groups due to Mr Orban's views on migrants, alleged Islamophobia and for assaults on democracy and the freedom of the press.
Speaking to journalists on Downing Street, Mr Orban defended his country's record on human rights and press freedom and said claims he was anti-Semitic were "simply ridiculous".
However, he did not deny being anti-migration saying "to be anti-migrant...is to be on the good side" and said he understood why remarks about "Muslim invaders" were offensive but added it was "the fact".
A Downing Street spokesperson said Boris Johnson had raised "his significant concerns about human rights in Hungary, including gender equality, LGBT rights and media freedom."
Listen to our politics podcast - Calling Peston