The National Fire Agency identified two of the victims as Indonesian, another as Filipino and the fourth body has not been identified.
All the victims are believed to be fishing boat workers from Indonesia and the Philippines.
The 140-metre-long arched bridge collapsed on Tuesday into a bay on Taiwan’s east coast, about 40 miles southeast of Taipei, the capital.
A typhoon swept by the island earlier, but the weather was sunny when the bridge collapsed and it was not clear if the storm was a factor.
Ten people were taken to hospitals with injuries, including the Taiwanese driver of an oil tanker that fell off the bridge.
The other nine are Filipino and Indonesian fishermen, with one, an Indonesian, in serious condition, Taiwan’s official Central News Agency said.
Many Filipinos and Indonesians work on fishing boats registered in Taiwan, where the pay is better than in their home countries.
The 18-metre-high high bridge in Nanfangao, a Pacific coast fishing port, opened in 1998 and replaced a lower span that prevented large fishing vessels from passing underneath.
The company that designed the bridge, MAA Consultants, says it is the only single-span arch bridge in Taiwan supported by cables and the second single arch-cable steel bridge in the world.