Video report by ITV News Royal Editor Chris Ship
In unseen footage from Harry and Meghan's interview with Oprah Winfrey, Harry said his grandmother had initially invited the couple to come up to Sandringham and stay for dinner after they arrived in the UK from Canada on January 6, 2020.
But when they landed in the country, he received a text from his private secretary telling him not to "cannot come to Norfolk" as the Queen was "busy all week".
Harry said he was uninvited after he and Meghan made it known to the Royal Family that they intended to quit as senior royals.
He told Oprah: "That announcement that we put out on the 8th January 2020, the contents of that was put in a letter to the institution, to my father, which was then shared at the end of December while we were in Canada.
ITV News Royal Editor Chris Ship discusses the fallout from the interview
"And to then get back on the 6th after my grandmother had said: 'The moment you land, come up to Sandringham, would love to have a chat. Why don't you stay for dinner because it's going to be a long drive and you're going to be exhausted.'
"Wonderful. I'd love that.
"The moment we land in the UK, I got a message from my private secretary, Fiona, at the time, cutting and pasting a message from the Queen's private secretary saying: 'Please pass on to the Duke and Duchess of Sussex, he cannot come to Norfolk. The Queen is busy, she's busy all week.'"
Oprah asks: "After she's just invited you?"
Harry replies: "She'd just invited me."
He continued: "So I rang her from Frogmore that night and said: 'I was thinking about coming anyway but I hear you're now busy.
"And she said: 'Yes, there's something in my diary that I didn't know that I had.'
Listen to our podcast about the Royal Family:
"And I said: 'Well, what about the rest of the week?' And she said: 'That's busy now as well.'
"I didn't want to push, because I kind of knew what was going on."
Oprah asks: "But doesn't the Queen get to do what the Queen wants to do?"
After a pause, Harry says: "No. When you're head of the firm, there's people around you that give you advice. And what has also made me really sad is that some of that advice has been really bad."
ITV News Royal Editor Chris Ship says it will take Buckingham Palace some time to respond to the Harry and Meghan interview
Harry and Meghan's explosive interview aired in the US overnight on Sunday. It will be available for UK audiences to watch on ITV at 9pm on Monday.
During the interview, the Duchess of Sussex opened up about her suicidal thoughts during her time in the Royal Family and said there were concerns raised by a member of Harry's family before the birth of son Archie about "how dark" his skin colour might be.
Oprah With Meghan and Harry will be broadcast in the UK on ITV on Monday 8 March at 9pm. It will also be available on ITV Hub.