"There is no money" - the famous note was meant for Philip Hammond, he was expected to be Chief Secretary to the Treasury in 2010.
The coalition meant he didn't become Chief Secretary and find the note, but he's never forgotten what it said.
To this day the lack of money guides the Chancellor's approach, so it's not in his nature to produce the big exciting Budget measures that some in his own party would now like to see on housing.
Yet the Prime Minister says the UK hasn't been building enough houses for years and "I am seeing the work now underway to put this right and, in coming weeks and months, my government will be going further to ensure that we build more homes, more quickly."
Today the Communities Secretary Sajid Javid ratcheted up the pressure saying "In next week's Budget you'll see just how seriously we take this challenge, just how hard we're willing to fight to get Britain building."
The Treasury is more cautious. The Chancellor has said there is "no silver bullet" to solve the housing crisis.
A Treasury source says there will be a number of detailed and practical policy measures to tackle the housing issue on different fronts, but not one 'big whacking figure' to fix housing.
A number of ideas are being lobbied for; taking Local Authority borrowing off Government books so they are free to raise more cash to build, letting them keep all the money raised from selling off council houses and perhaps exempting first time buyers from Stamp Duty.
Will a raft of less expensive policies do the job? The Chancellor's future may depend on how the Prime Minister answers that question.