The 74-year-old told ITV News that he spent more time at the cricket test match last week than in his office.
Justice Secretary Kenneth Clarke insists that Britain has steered through real reforms to the European Court of Human Rights.
The Government is to go ahead with changes that would allow more trials to be held behind closed doors after opposition from MPs
Conservative minister Ken Clarke has joined forces with Liberal Democrat Danny Alexander and Labour's Lord Mandelson to warn that leaving the EU would be an "historical error".
The politicians insisted the UK should work to reform the organisation from within rather than walking away.
Mr Clarke, regarded as the sole Conservative europhile in the Cabinet, said: "David Cameron has already shown what can be achieved in Europe through strong, determined leadership.
"We earn our living in a globalised economy and a world in which nations are interdependent in tackling global and regional problems.
"We can help to promote our own interests best by reinforcing our role as a leading nation in the EU."
The former chancellor Ken Clarke has told ITV News that the party needs to 'move on and focus on the result of the referendum'.
The Minister without Portfolio said that the draft legislation being bought out by his party 'underlined the commitment that David Cameron was making to holding a referendum'.
He added that the most 'important thing is to make sure we get the right result from a referendum that's held to avoid the catastrophe to the countries economy and our political standing in the world. If we were to be so very very reckless as to leave the European Union.'
Conservative Cabinet minister Ken Clarke said he does not believe it is possible to return money to the Government after Iain Duncan Smith urged wealthy pensioners to hand their benefits back.
Mr Clarke, 72, refused to say whether he returned the universal benefits he is entitled to.
Speaking on Sky News' Murnaghan programme, Mr Clarke said, "It is certainly the case when it comes to a bus pass and when it comes to the winter fuel all taxpayers should decide and recipients should decide what to do with it themselves".
"You can't hand it back to the Government. I don't think it is a system for doing that. Every pensioner and retired person like myself has to make up their own mind about whether they really need it and whether they are going to give it to some worthwhile cause", he continued.
"No doubt most pensioners who are reasonably prosperous give quite a lot of money to charity and worthwhile causes in any event".
Until now the backroom deliberations within the Conservative party over the Falklands have remained largely private but the notes are among those released by the Margaret Thatcher Archive Trust as it opens its files from a year which came to define Lady Thatcher's career.
They range from Ken Clarke, then a junior minister, arguing to "blow up a few ships but nothing more" to West Devon MP Peter Mills who warned "my constituents want blood".
On April 6, four days after the incursion, the Chief Whip, Michael Jopling, prepared a note for the Prime Minister saying: "You may like to have general re-action to events in the Falkland Islands."
Former chancellor Ken Clarke has warned of the dangers of an economy based on oil revenues, as he attacked the SNP's predictions of a post-independence referendum North Sea boom.
Speaking in Edinburgh, Mr Clarke issued advice to Scottish First Minister Alex Salmond and Finance Secretary John Swinney not to "bet the ranch" on figures "pulled from the air".
Mr Clarke, UK Minister without Portfolio, said, "If we were to have an independent Scotland, it is not the case that it would not somehow have to face up to the consequences of economic crisis".
The former Conservative Cabinet minister Ken Clarke has warned David Cameron that any attempt to imitate UKIP will "drive moderate people to stick with the Liberal Democrats", adding: "I can't think of a more certain way to lose the general election than to go for a lurch to the right."
The now Minister Without Portfolio went on to say that talk of the Human Rights Act being scrapped was not something he recognised as government policy, or "any policy likely to be adopted by a Conservative Party that I can recognise."
Cabinet Minister Ken Clarke has admitted it would be "very reckless" for the Government to threaten to withdraw from the European Union after the next general election.
In an interview with the Daily Telegraph he said:
To start threatening, throwing into the air our relationship with the outside world, with the global economy, would I think be very reckless.
I frequently say that to several of my colleagues: there seems little point in opening up the debate at the moment about our membership of the European Union.
The idea that somehow the present problems are caused by Britain's membership of the European Union is a theory that I can't quite follow.
Married couples should not "count on" the Government delivering them tax breaks before the 2015 general election, Cabinet minister Kenneth Clarke said today.
Conservatives fought the 2010 election on a promise of a transferable tax allowance that could be worth £150 a year to married couples where one spouse stays at home.
But the measure was opposed by Liberal Democrats, who secured the freedom under the Coalition Agreement to abstain in any vote on the issue, and Mr Clarke indicated today in an interview with the Daily Telegraph that tough economic conditions made it unlikely to be introduced before 2015.
Despite the better-than-expected growth of 1% in the third quarter of 2012 announced earlier this week, Mr Clarke said it was too early to be certain the economy is bouncing back.
Saying that he believes a "long hard road" still lies ahead, Mr Clarke indicated that married tax allowances may be a casualty of the tight economic conditions.
"We never committed ourselves to married couples' tax by the end of the Parliament," the minister without portfolio told the Telegraph.
"I'm married, I'm not counting on it. I don't remember anyone promising that kind of thing."On the economy, Mr Clarke - a former chancellor - said: "It would be absolute folly to turn around and say it will all be fine by Christmas."
The UK is "plainly losing" the war on drugs - and may even be going backwards, Justice Secretary Kenneth Clarke has said.
Danny Kushlick from drug charity Transform told Daybreak that Clarke's admission is a breakthrough in the fight against drug crime.
Ken Clarke has said the government has no intention of changing the law on drugs and that problem will not be solved by decriminalisation.
He said: "The Government has no intention whatever of changing the criminal law on drugs.
"I have frankly conceded that policy has not been working. We are all disappointed by the fact that far from making progress it could be argued we are going backwards at times.
"But my own purely personal view is that I would be worried about losing the deterrent effect of criminalisation of youngsters who start experimenting.
"The really key thing is to try to work out how to get fewer young people to start experimenting with drugs.
"One thing that does put them off is that they would get into trouble with the police."