Mr Khan made the comments on Sunday in a live tele-chat dubbed “Prime Minister On Call With You”, in which he answers questions from the public. The programme was streamed by state-owned Pakistan Television.
According to women’s rights activists, the remarks excuse rapists and put the blame for assaults on women.
The 68-year-old also said vulgarity and obscenity combine to destroy societies and families.
“If you keep increasing vulgarity in a society, then definitely there will be this impact,” he said.
“What is the whole concept of observing the veil? It is so that there is no temptation in society. Each individual does not have the willpower or strength, if you keep increasing obscenity in the society and if you don’t care, then there are impacts of such things.”
The remarks caused an uproar among Pakistani rights groups, which say such misogynistic remarks shame women and excuse rapists.
The independent Human Rights Commission of Pakistan called Mr Khan’s comments unacceptable and appalling for suggesting the veil can contain sexual assault.
“Not only does this display a baffling ignorance of where, why and how rape occurs, but it also lays the blame on rape survivors,” said the Human Rights Commission.
The commission’s chairperson Hina Jilani is a member of The Elders, a group formed in 2007 by Nelson Mandela.
The commission demanded an apology from Mr Khan and a commitment his administration would tackle rape “as an act of violence, of power”.
In the call show, Mr Khan also attacked Hollywood and the Indian cinema centre known as Bollywood as purveyors of obscenity and vulgarity that he said contribute to rape, high divorce rates and the break-up of family units in society.
“If our religion gives us the concept of observing veil, then there is some philosophy behind it and the philosophy is to save the family system and to protect the society from such things,” Mr Khan said.
Mr Khan’s information adviser Raoof Hasan said the prime minister’s remarks were misinterpreted and said he advocated a “holistic” approach to sexual assault that includes both strong legal repercussions for rapists and sexual predators and efforts by society as a whole working to find remedies.
“Plucking a single line out distorts the prospective and does not serve the cause of the actual statement,” Mr Hasan said.
Pakistan has been rocked by several high-profile sexual attacks, including an assault in September of a mother gang-raped before her children after their car broke down on a major road at night.
The following day a senior police official questioned why the woman was alone, had not checked her fuel gauge before travelling or chosen a busier road. Campaigners’ calls for his censure were ignored.