Many of the victims died as they jumped from the building to escape the flames, a police official said, while witnesses added the death toll could rise.
The cause of the blaze was not immediately clear.
Bangladesh has around 4,500 garment factories that make clothes for brands including Tesco, Wal-Mart, JC Penney, H&M, Marks & Spencer, Kohl's and Carrefour.
Ready made garments make up 80 percent of the country's $24 billion annual exports.
More top news
Bodies are being buried in backyards and overwhelming morgues as officials try to find another plot of land to lay them to rest.
Authorities have warned that the death toll is likely to rise as much of the building is yet to be searched.
The PM will sound out the possibility of new free trade agreements with Gulf states for once the UK has left the EU.