Children in care in England are forced to wait an average of 20 months to move in with adoptive parents, according to new figures.
But local authorities in the south including Buckinghamshire, Portsmouth, Swindon and West Sussex are taking even longer to complete the process.
Children's Minister Tim Loughton said the first-ever local authority scorecards were a "trigger for urgent, detailed discussions" to speed up the process.
It is part of an action plan for adoption which includes proposals to reduce the length of the approval process for would-be adopters to six months.
Council leaders and children's services professionals condemned the scorecards and warned they have the potential to cause "unnecessary and avoidable concern in communities where there shouldn't be any".
The figures show that 80 local authorities met the interim thresholds of 21 months from entering care to adoption and matching a child to a family within seven months of a court order.
They include East Sussex, Hampshire, Bournemouth and Reading.
But 72 councils did not meet one or both of these thresholds which will be lowered to 14 months and four months respectively within four years.
They include Buckinghamshire at 23 months, Portsmouth at 24 months, Swindon at 27 months and West Sussex at 22 months.
Mr Loughton said: "Hundreds of children are being let down by unacceptable delays right across the country and throughout the adoption process. Every month a child waits to be placed there is less chance of finding a permanent, stable and loving home. This cannot go on.
"There has been some real progress, with local authorities beginning to bear down on adoption delays and helping in the redesign of a faster but still-thorough adopter assessment process. But these statistics illustrate all too starkly the magnitude of the challenge which we face.
"I make no apology for shining a light on the system to hold local areas to account. I have been clear that we won't hesitate to intervene where the worst delays are not tackled effectively."