Conservative Cabinet minister Ken Clarke said he does not believe it is possible to return money to the Government after Iain Duncan Smith urged wealthy pensioners to hand their benefits back.
Mr Clarke, 72, refused to say whether he returned the universal benefits he is entitled to.
Speaking on Sky News' Murnaghan programme, Mr Clarke said, "It is certainly the case when it comes to a bus pass and when it comes to the winter fuel all taxpayers should decide and recipients should decide what to do with it themselves".
"You can't hand it back to the Government. I don't think it is a system for doing that. Every pensioner and retired person like myself has to make up their own mind about whether they really need it and whether they are going to give it to some worthwhile cause", he continued.
"No doubt most pensioners who are reasonably prosperous give quite a lot of money to charity and worthwhile causes in any event".
More top news
MPs could be asked to authorise airstrikes against IS in Syria in the wake of the Tunisia massacre, the defence secretary has suggested.
Cooler temperatures are forecast after thunderstorms and heavy downpours hit parts of Britain after the hottest July day on record.
Only a third of police cases involving vulnerable children are handled to a good standard, inspectors have found.