All survivors were said to be in good condition, but taken to hospital for further evaluation.
A video posted online by the USCG Southeast showed responders clapping and cheering as the final man, wearing only shorts, climbed out of a hole in the hull and stood up.
Three of the South Korean crew members emerged from the wreckage in the mid-afternoon.
The fourth man, who was trapped in a separate compartment, was saved three hours later.
In a Twitter video with the final crew member, USGC Southeast Captain John Reed could be heard exclaiming: "This is amazing! The best day of my career because you guys did that! Outstanding!"
The crew member - looking happy and healthy - could be heard thanking everyone.
The rescues followed nearly 36 hours of drilling into the hull’s steel plates to release the men from the capsized cargo ship.
The Golden Ray, carrying thousands of vehicles and 24 crew members, rolled to its side early on Sunday morning as it left Brunswick for Baltimore.
It is not currently clear what caused the ship to overturn.
In the hours immediately after the accident, the Coast Guard lifted 20 crew members into helicopters before determining that smoke and flames and unstable cargo made it too risky to venture further inside the vessel to retrieve the four remaining men.
At first, rescuers thought the noises they were hearing inside the 656ft vessel could be some of the vehicles it was transporting crashing around.
But by dawn on Monday, they were confident that the taps were responses to their own taps, indicating someone was alive inside.
“It was outstanding when I heard the news this morning that we had taps back throughout the night,” Captain John Reed said.
Those sounds helped lead rescuers to the right place and provided motivation.
“They were charged up knowing the people were alive,” Mr Reed said.
On Monday morning, rescuers landed on the side of the Golden Ray and rappelled down the hull.
Lietenant Lloyd Heflin, who was coordinating the search, said they found three men in a room close to the propeller shaft, near the bottom of the stern.
Responders began drilling, starting with a three-inch hole.
Coast Guard officials brought the ship’s chief engineer, who was rescued on Sunday, out to the ship to translate, and found the three men were “on board and okay,” as Mr Heflin put it.
Mr Reed said rescuers passed food and water through the hole to the men. They also provided fresh air to the propeller room, which he said was even hotter than outside, where the heat reached 34 degrees.
Responders set up a tent on the hull and began drilling additional holes, eventually making an opening large enough to insert a ladder and help the men climb out.
“It was like connect the dots,” Mr Reed said of the hole, which grew to two feet by three feet.
The fourth rescue was a greater challenge, as the last crewman was behind glass in a separate engineering compartment on another deck.
The cause of the capsizing remains under investigation.
Marine Traffic shows the Golden Ray overturned as it was passed by another car carrier entering St Simons Sound.
At the time, the skies were clear and the weather calm, with a southerly breeze of only five miles per hour, according to National Weather Service records.
The vessel is owned by Hyundai Glovis, which carries cars for automakers Hyundai and Kia as well as others.
In a statement, the company thanked the Coast Guard for saving the crew and sought to assure the public that it would now focus on “mitigating damage to property and the environment”.