Many of the victims died as they jumped from the building to escape the flames, a police official said, while witnesses added the death toll could rise.
The cause of the blaze was not immediately clear.
Bangladesh has around 4,500 garment factories that make clothes for brands including Tesco, Wal-Mart, JC Penney, H&M, Marks & Spencer, Kohl's and Carrefour.
Ready made garments make up 80 percent of the country's $24 billion annual exports.
More top news
Atlantic frontal systems spread rain in from the northwest on both weekend days. Windy - with gales across the north and west but mild.
The band announced at a free concert on Friday that they would take over the event's Pyramid stage on Saturday June 25.
The victims were collecting permits and escorts a day after their home town was liberated from the extremists.