The Government has rejected an offer from security firm G4S to repay the £24m it owes for overcharging in credit notes.
Officials turned down the settlement and say they will "pursue all possible avenues" to recoup more taxpayers' money.
A Ministry of Justice spokesman said it was still working with the firms and the independent auditors to work out "what the final sum will be".
"The Secretary of State has been clear: we are determined to secure a refund for the taxpayer.
"We have taken appropriate legal advice and will pursue all possible avenues."
More top news
A man from China has set a rather unenviable new world record - by covering himself in more than 1.1 million bees, weighing 17st.
Chinese goalkeeper Sui Weijie has been fined 50,000 yuan (£5,240) after allowing an opponent to score while he sipped on a bottle of water
A "caring and loving" teenage boy who "lit up the life" of everyone he met has died in hospital after taking what he thought was ecstasy.