The sites in Norwich, Nottingham, Lakeside in Essex and Metrocentre in Gateshead, will face closure in the new year.
The landlord of the sites is London-listed Intu, which is in the midst of a potential sale to a group of investors.
Sports Direct said it had engaged with the company but was ultimately “unable to agree reasonable terms for these stores to continue trading”.
“Unfortunately, these stores now face closing in the new year. I urge other institutional landlords to be more proactive to help save the HoF stores in their schemes.”
Sports Direct is now in consultation with staff at the affected stores.
It comes on the same day as Intu pushed back the deadline for a second time on its talks with a consortium of buyers looking to take over the business.
A group led by John Whittaker’s Peel Group now has until November 22 to make a firm offer.
The tycoon has warned that time is “running out” and said that some stores will have to close.
Following his acquisition of the chain in August, Mr Ashley pledged to save around 47 of House of Fraser’s 59 outlets.
Wednesday’s announcement adds to previously announced closures in Edinburgh, Hull and Swindon.
Intu has been contacted for comment.