The transport secretary's decision to award Seaborne Freight a contract worth £13.8 million had attracted widespread criticism.
His department said it had decided to terminate the contract after Irish company Arklow Shipping, which had backed Seaborne Freight, stepped away from the deal.
A DfT spokeswoman said: "Following the decision of Seaborne Freight's backer, Arklow Shipping, to step back from the deal, it became clear Seaborne would not reach its contractual requirements with the Government. We have therefore decided to terminate our agreement.
"The Government is already in advanced talks with a number of companies to secure additional freight capacity – including through the port of Ramsgate – in the event of a no-deal Brexit."
Mr Grayling last month defended the Seaborne Freight contract, insisting it was "not a risk".
It was one of three firms awarded contracts totalling £108 million in late December to lay on additional crossings to ease the pressure on Dover when Britain leaves the EU, despite having never run a Channel service.
The department said it had been Arklow Shipping's backing that gave it confidence in the viability of the deal, and that it stands by the robust due diligence carried out on Seaborne Freight.
Labour seized on the situation to say Mr Grayling should quit or be sacked.
"As we predicted, the Seaborne Freight contract has been cancelled," shadow transport secretary Andy McDonald said.
"This cannot go without consequence. The Chris Grayling catalogue of calamities grows bigger by the day.
"This contract was never going to work but this Secretary of State, true to form, blunders from one disaster to another.
"Whilst Theresa May needs the few friends she has right now, we cannot have this incompetent Transport Secretary carry on heaping humiliation after humiliation on our country. He has to go."
The DfT said no taxpayer money had been transferred to the company.
Brexit-backing Tory MP Jacob Rees-Mogg questioned whether Leo Varadkar's Irish government had any influence on Arklow's decision.
He told the Daily Telegraph: "One has to hope that the Irish government has not leant on or put any pressure on Arklow to persuade it to pull out.
"That would be a very unfriendly act of a neighbour to obstruct no-deal preparations and one has to hope very sincerely that this is genuinely a corporate decision."
Mick Cash, general secretary of the Rail, Maritime and Transport union, said: "RMT has taken a number of protests over the fiasco of the Government's Brexit ferry contracts to both the Department for Transport and the ports, and the news this morning comes as no surprise to us.
"The whole exercise is a complete and utter shambles with the government ignoring union calls on what needs to happen. Instead they are blundering on from crisis to crisis."