The 74-year-old told ITV News that he spent more time at the cricket test match last week than in his office.
Justice Secretary Kenneth Clarke insists that Britain has steered through real reforms to the European Court of Human Rights.
The Government is to go ahead with changes that would allow more trials to be held behind closed doors after opposition from MPs
Ken Clarke has claimed the Tory press office tried to keep him away from the television cameras during his time in government, even attempting to prevent him from going on Question Time.
"I had a great row with them [Conservative press officers] when they told the producer of Question Time that I was ill and they were able to provide a replacement," Clarke said during an interview with the Observer.
" It never occurred to them that I could ring up the producer. She said: 'I'm told you are ill.' After that, I got even more freelance."
The former minister admitted there was "constant friction" between himself and David Cameron, saying his views sometimes "didn't coincide with No 10's".
Former Tory Cabinet minister Ken Clarke has questioned the strength of the recovery, claiming the economy is still "fragile".
Mr Clarke, who retired from the government in the reshuffle, said the economy is "not firmly enough rooted" in a balance between manufacturing and services, leaving it prone to shocks.
The former minister did however declare himself a "great fan" of Chancellor George Osborne and said the Conservatives had saved the country from calamity by reining in spending.
Clarke, 74, told the Observer that the economy still has a "long, long way" to go before sustainable levels of growth could be maintained.
The former chancellor added that Britain must break out of the "ludicrous cycle" of house price booms followed by crashes.
The Prime Minister has written to Kenneth Clarke saying he will be 'hugely missed' around Parliament, after the MP stepped down from his post as Minister Without Portfolio.
On the resignation, Mr Cameron wrote to him saying:
– Prime Minister David Cameron
Since you first entered Parliament in 1970, and over forty years since your first Front Bench role as a PPS, your passion for getting things done and the energy you have brought to your Cabinet posts has not diminished one bit.
To have that level of experience at my own Cabinet table has been incredibly helpful – both to me as Prime Minister and to the whole Cabinet.
You have never been timid to raise issues of importance or to stand up for causes that matter to you, but you have also brought a keen sense of humour to the Cabinet table – and you will be hugely missed.
Nigel Farage decided not to stand in the Newark by-election, as he knew he could not win, according to Conservative Cabinet minister Ken Clarke.
"I am not really surprised. Whatever else Nigel is, he is not an idiot and I don't think he'd have the faintest chance of winning in Newark," he told the BBC.
Earlier the Ukip leader said: "I don't have any links with the East Midlands, I would just look like an opportunist and I don't think that would work."
But he added: "We will fight the Newark by-election and fight it damned hard and get a good local candidate.
A senior Conservative MP has said "it is truly shocking" that Britain helped the United States to kidnap and torture British detainees, as he warned that the Government would come to regret its decision not to allow an independent judge-led inquiry to run its course.
Andrew Tyrie said today's confirmation by Cabinet Office minister Ken Clarke that the investigation would instead be handed over to Parliament's controversial Intelligence and Security Committee (ISC) was "a mistake".
It is truly shocking that Britain has facilitated kidnap and torture. And the decision to end, to abandon this judge-led inquiry will, I think, come to be seen as a mistake.
What confidence can the public have in their conclusions when that same body wrongly concluded that Britain was not involved in 2007 only to be flatly contradicted by a High Court written ruling the following year?
Conservative minister Ken Clarke will join ITV News Political Editor Tom Brady on The Agenda tonight.
Former BBC Economics Editor turned JP Morgan strategist Stephanie Flanders, FA director Heather Rabbatts and TV and radio presenter Terry Christian will also be on the panel.
The Agenda with Tom Bradby is on ITV at 10.35pm.
Join in the debate on Twitter using the hashtag #TheAgenda.
Former Business Secretary Lord Mandelson said: "If Britain quit the EU, amongst the losers would be businesses foregoing trade and investment opportunities, their employees whose jobs and workplace rights would be at risk, the police and security agencies who would sacrifice vital collaboration."
"The British people would lose the freedom to travel, live and work as they wish across Europe," he continued.
"In return we would get the dubious satisfaction of standing alone in the world."
Treasury minister Danny Alexander said the EU was "one of the central pillars of British prosperity and security".
"Membership of the EU gives market access to British firms, makes us more attractive to overseas investors, and underpins more than three million British jobs," he said.
"EU trade deals with the US, India, Canada and Japan will be worth billions to the UK economy but could not be won by the UK alone."
"Britain must work with our allies for change from within to ensure the EU continues to take reform seriously.
"The isolationists are reckless with our prosperity and security. We cannot let them succeed."
Conservative minister Ken Clarke has joined forces with Liberal Democrat Danny Alexander and Labour's Lord Mandelson to warn that leaving the EU would be an "historical error".
The politicians insisted the UK should work to reform the organisation from within rather than walking away.
Mr Clarke, regarded as the sole Conservative europhile in the Cabinet, said: "David Cameron has already shown what can be achieved in Europe through strong, determined leadership.
"We earn our living in a globalised economy and a world in which nations are interdependent in tackling global and regional problems.
"We can help to promote our own interests best by reinforcing our role as a leading nation in the EU."