US Coast Guard Lieutenant Commander Matthew Kroll said authorities had recovered 20 bodies and discovered five more.
Those five have not been recovered due to unsafe conditions under the boat, which sank about 20 yards (18 metres) from shore.
Four bodies were recovered on Monday morning and 16 others were recovered later in the day.
Authorities will search for the nine people still unaccounted for through the night.
"We should all be prepared to move into the worst outcome," Coast Guard Captain Monica Rochester said at a news conference.
The fire broke out aboard the vessel Conception at around 3am local time off Santa Cruz Island.
Five of six crew members escaped by jumping off the boat and taking refuge on an inflatable boat called The Grape Escape which was anchored nearby.
The Grape Escape’s owners, Bob and Shirley Hansen, told The New York Times they were asleep when they heard pounding on the side of their 60-foot (18-metre) fishing vessel at about 3.30am and discovered the frightened crew members.
They told the couple they had fled when the fire grew out of control.
"When we looked out, the other boat was totally engulfed in flames, from stem to stern," Mr Hansen said, estimating it was no more than 100 yards from his craft.
“I could see the fire coming through holes on the side of the boat. There were these explosions every few beats. You can’t prepare yourself for that. It was horrendous.”
“The fire was too big, there was absolutely nothing we could do,” he added.
Mr Hansen said he and his wife gave the crew clothes and two of them went back towards the Conception looking for survivors.
The Conception, based in Santa Barbara Harbour on the mainland, was on the final day of a Labour Day weekend cruise to the Channel Islands when the fire erupted.
"At 3.15 this morning the Coast Guard overheard a mayday call. The call was garbled, it was not that clear, but we were able to get some information out of it to send vessels on scene," Coast Guard Petty Officer Mark Barney said.
Santa Barbara County Sheriff Bill Brown said the elements of the tragedy were daunting for rescuers.
The boat was in a remote location with limited firefighting capabilities, passengers were sleeping below deck in the middle of the night and there was a quick-moving fire.
"You couldn’t ask for a worse situation," the sheriff said.
The Conception was chartered by Worldwide Diving Adventures, which says on its website that it has been taking divers on such expeditions since the 1970s.
It was owned and operated by Truth Aquatics, a Santa Barbara-based company founded in 1974.
Coast Guard records show inspections of the Conception conducted last February and in August 2018 found no deficiencies. Earlier inspections found some safety violations related to fire safety.
A 2016 inspection resulted in owners replacing the heat detector in the galley and one in 2014 cited a leaky fire hose.
Records show all safety violations from the last five years were quickly addressed by the boat’s owners.