The sale of new cars fell by 5.6% last year amid declining consumer confidence, the Society of Motor Manufacturers and Traders has said.
Around 2.54 million new cars were registered in 2017 compared with 2.69 million the previous year, while the month of December suffered an even sharper decline of 13.9%.
Demand for diesels fell by 31% last month and 17% over the year.
The market's decline is likely to continue into 2018, said SMMT chief executive Mike Hawes, who described 2017 as a "very, very volatile year."
Hawes said uncertainty over transitional arrangements after the UK's March 2019 exit from the EU has caused some manufacturers to delay investment decisions.
If there is no clarity by the end of March they will have to start implementing "contingency plans" which could harm funding for UK operations, he added.
Hawes said the decline in diesel cars is down to "underlying confusion" among consumers about whether buying such vehicles is "the right thing to do".
Chancellor Philip Hammond announced a tax hike on new diesel cars from April 1 in his Autumn budget. All diesels are expected to be subjected to a one-band increase in the first year vehicle excise duty (VED) rate after the Government unveiled plans to ban the sale of all conventional diesel and petrol cars by 2040.
Petrol registrations dropped 1.6% in December but rose 2.8% in 2017 as a whole, while alternatively fuelled cars reached a record market share of 4.7% last year.
New car sales to private motorists were down 6.5%, fleet down 4.4% and business down 7.7%.
Hawes stressed 2017 was still the third best year for new car registrations in the past decade and followed record performances in 2015 and 2016. "The market is still at historic highs," he said.
Jim Holder, editorial director of What Car? magazine, said turning the sharp decline at the end of 2017 into a loss of just 5-7% in 2018 will be a "real challenge" for the industry.
"That's going to take a massive swing in public perception," he told the Press Association.
"I would say there is a risk of double digit (decline) unless all the good news stories come true, and that includes turning some pretty entrenched views around on the economy and diesel."