Authorities found around 50 bodies stored on ice inside four trucks rented by the Andrew T. Cleckley Funeral Home.
It's thought the facility resorted to the measure following overwhelming demand during the Covid-19 outbreak in the city.
Officers were called to the address in Brooklyn after a passerby complained about a smell coming from the vehicles, officials said.
The discovery was made on Wednesday according to an official who spoke anonymously to The Associated Press.
The Andrew T. Cleckley Funeral Home was cited for failing to control the odour - the home was then able to obtain a larger, refrigerated truck later in the day.
No criminal charges were brought against the home, the source said.
Brooklyn Borough president Eric Adams, who was at the scene on Wednesday evening, told the Daily News: "While this situation is under investigation, we should not have what we have right now, with trucks lining the streets filled with bodies.
"It was people who walked by who saw some leakage and detected an odour coming from a truck."
Funeral homes in New York City have struggled to cope under increased demand during the pandemic.
The city's death toll from the outbreak has risen to at least 18,000 since the end of March.
New York has set up temporary morgues and hospitals have been using refrigerated tractor trailers to cart away multiple bodies at a time.
Crematoriums in the city have been backed up, and funeral directors across the city have pleaded for help as they have run out of space.
The New York Police Department notified the state Department of Health - which oversees funeral homes - about the discovery at the Andrew T. Cleckley Funeral Home.
The Department of Health has not responded to an approach for comment.
Coronavirus: Everything you need to know