Former cabinet secretary Lord Butler of Brockwell said he backs an amendment calling for the public to be consulted again on the Government's final Brexit deal.
Lord Butler's comments came as the Lords debated the European Union (Notification of Withdrawal) Bill for the second day.
Lord Butler said that while he accepts Britain is leaving the single market to regain control of immigration, a final deal should be put to the people.
Lord Butler said: "Is the outcome of last June's referendum to be interpreted as meaning that a majority of the United Kingdom want to leave the EU whatever the terms? The Government clearly thinks so.
"But on a matter of this importance has not the Government a duty to be sure before our departure becomes final?
"My lords, one has to ask why those who base their arguments for Brexit on the will of the people are now opposed to consulting the people on the outcome of the negotiations.
"One has to suspect that they fear that they will get a different answer. If so, we ought to know."
Former Labour chancellor Alistair Darling has said the EU referendum result must be accepted, but ruled out giving the Government a "blank cheque".
Lord Darling of Roulanish's comments came as the House of Lords debated the European Union (Notification of Withdrawal) Bill for the second day.
The legislation, which would allow Theresa May to trigger the formal two-year Brexit process, has already cleared the Commons unamended with big majorities.
During his speech Lord Darling said none of the negotiations could be conducted in secret, since with 27 other countries involved there would be a running commentary every day.
"Let's be grown-up about it. Let's engage positively," Lord Darling said.
"From the Brexiteers and Government point of view, they have to accept there is a large section of the population in this country, a large membership in the Lords and the Commons, who won't accept some of the extreme arguments being put forward and think the voice of reason must prevail for the good of our country."
Lord Darling also warned the "middle ground of British politics" had been abandoned, adding: "That's a very dangerous place for us to be."
Prime minister's ethics adviser suggests rigorous interviews could avoid repeat of 'Cameron's cronies' scandal.Read the full story ›
The Prime Minister's decision to shelve a review of the Upper House is a "big mistake" in light of Brexit vote says peer.Read the full story ›
Report recommends cuts to the number of bishops to make way for other faith leaders and reflect the 'pluralist character of British society'Read the full story ›
Members of the House of Lords argued five million Britons living abroad should get a vote in the referendum on the UK's EU membership.Read the full story ›
Labour leadership contender Andy Burnham has called for the House of Lords to be scrapped, saying it's "a national embarrassment".Read the full story ›
There are now 826 peers in the House of Lords. But with only 200 seats how will they all fit in?Read the full story ›
Peers claimed £360,000 in attendance fees and expenses over the last five years even though they failed to vote in the House of Lords.Read the full story ›