Home Office Minister Jeremy Browne said he believes people should not be told what to wear but warned against being "fearful" over holding a national debate on banning veils.
The Liberal Democrat MP said: "I think we have to be careful about being fearful on discussing these issues.
"I think there is a perfectly legitimate discussion that liberal-minded people can have."
When I asked if it was time for a national debate on the wearing of the veil, as suggested by Jeremy Browne, the Prime Minister's official spokesman said that David Cameron's view was that Parliament should not be legislating on what people can and cannot wear in the street.
But he added that is not incompatible with institutions having their own rules and schools should be able to set their own dress code.
Deputy Prime Minister Nick Clegg said it was not "appropriate" for people to wear a full veil while going through airport security or studying in a classroom.
But Mr Clegg said he did not want to see a ban on the wearing of religious items of clothing in particular circumstances.
His comments came as Liberal Democrat Home Office Minister Jeremy Browne said the Government should consider banning Muslim girls and young women from wearing the veil in public places.
Mr Clegg said: "My own view, very strongly held, is that we shouldn't end up like other countries issuing edicts or laws from parliament telling people what they should or should not wear.
"This is a free country and people going about their own business should be free to wear what they wish. I think it is very un-British to start telling people what pieces of clothing they should wear.
"I think there are exceptions to that as far as the full veil is concerned - security at airports for instance. It is perfectly reasonable for us to say the full veil is clearly not appropriate there."
Mohammed Shafiq, chief executive of the Ramadhan Foundation, said he was "disgusted" by Jeremy Browne's calls to consider banning Muslim girls and young women from wearing the veil in public places.
This is another example of the double standards that are applied to Muslims in our country by some politicians.
Whatever one's religion they should be free to practise it according to their own choices and any attempt by the Government to ban Muslim women will be strongly resisted by the Muslim community.
We take great pride in the United Kingdom's values of individual freedom and freedom of religion and any attempt by illiberal male politicians to dictate to Muslim women what they should wear will be challenged.
I call on the Liberal Democrats to disown these illiberal comments from Jeremy Browne and ensure that no government with Lib Dem participation introduces this stupid Bill.
If they fail to do this then they will face the consequences at the next general election from Muslim voters.
We would expect these sorts of comments from the far right and authoritarian politicians and not from someone who allegedly believes in liberal values and freedom.
The Government should consider banning Muslim girls and young women from wearing the veil in public places, a Liberal Democrat minister has said.
Home Office Minister Jeremy Browne called for a national debate on whether the state should step in to prevent young women having the veil imposed upon them.
Mr Browne told the Telegraph he was "instinctively uneasy" about restricting religious freedoms but said there may be a case to act to protect girls who were too young to decide for themselves whether they wished to wear the veil or not.
His intervention came after a row erupted over the decision by Birmingham Metropolitan College to drop a ban on the wearing of full-face veils amid public protests.