Security firms G4S and Serco have been accused of overcharging the government tens of millions of pounds for electronic tagging services.
Senior probation officers have criticised Government's plans to involve private and charity groups in the supervision of offenders.
Frightened homeowners who react when confronted by burglars will get more protection under new plans, the Justice Secretary will announce.
Security firm Secro has confirmed it will repay any amount agreed to be owed to the Government.
Serco Group chief executive Christopher Hyman said: "Serco is a business led by our values and built on the strength of our reputation for integrity.
"These values lie at the heart of the many thousands of our people who are endeavouring to deliver the highest standard of service to our customers around the world. We are deeply concerned if we fall short of the standards expected of all of us.
"We are therefore taking this extremely seriously and will continue to work closely with our customer to resolve their concerns in this matter.
"We will not tolerate poor practice and behaviour and wherever it is found we will put it right."
Cabinet Office minister Francis Maude said Bill Crothers, the Government's chief procurement officer, will lead the review into all Government contracts held with G4S and Serco, which include running immigration centres and the welfare-to-work scheme.
Mr Maude said: "The public rightly expects government suppliers to meet the highest standards, and for taxpayers' money to be spent properly and transparently."
Electronic tagging is used as a method of monitoring offenders in the community and can be used to make sure criminals stick to court-imposed curfews.
The equipment consists of a tag fitted to the offender's ankle and a monitoring unit in the home or other place of curfew.
All movements in and out of the home, or other activities such as removal of the tag or tampering with the equipment, are reported to a control centre.
Justice Secretary Chris Grayling has told the House of Commons he is "astonished" by the revelations the government has been overcharged by "tens of millions of pounds".
– Justice Secretary Chris Grayling
This is a wholly indefensible and unacceptable state of affairs.
The House will share my astonishment that two of the Government's biggest suppliers would seek to charge in this way.
The House will also be surprised and disappointed to learn that staff in the Ministry of Justice were aware of a potential problem and yet did not take adequate steps to address it.
The audit into the Government's electronic monitoring contracts with G4S and Serco found that overcharging began at least as far back as the start of the current electronic monitoring contracts in 2005 - but could have dated as far back as the previous contracts in 1999.
It also found that contract managers in the Ministry of Justice discovered some of the issues following a routine inspection in 2008 - but did nothing to tackle the problem, Justice Secretary Chris Grayling said.
Justice Secretary Chris Grayling has told the House of Commons that security firm G4S has refused to take part in an additional audit so the Government can rule out the possibility that dishonesty was involved in any of the contracts, while Serco has agreed to this demand.
Serco has agreed to withdraw from the current tender process for electronic monitoring, while the Justice Secretary is to instigate moves to exclude G4S as the company is still attempting to bid.
The Government says it has been overcharged "tens of millions of pounds" by G4S and another firm, as part of its electronic tagging contract.
Justice Secretary Chris Grayling told the House of Commons the Government had been charged in cases when Serco and G4S were not providing electronic tagging - and in a small number of cases when the offender was known to have died.
Justice Secretary Chris Grayling has told the House of Commons he is to ask the Serious Fraud Office to investigate security firm G4S over the handling of its electronic monitoring contract after the company refused to take part in an additional forensic audit.
A network of resettlement prisons has been unveiled by the Justice Secretary in a bid to help inmates rehabilitate in the community where they are released.
The introduction of 70 resettlement prisons across England and Wales will see the majority of offenders released from jail in, or close to, the area in which they will live.
Chris Grayling said: "Rehabilitation in the community must begin behind the prison walls and follow offenders out through the gates if we are to stand a chance of freeing them from a life of crime."
Existing prisons up and down the country will function as resettlement prisons with a trial starting in north-western England in the autumn.
The Justice Secretary plans to build a £250 million super-prison in North Wales, while he announced a raft of prison closures covering some 2,600 inmate places in January.
A £250 million super-prison that could hold up to 2,000 inmates will be built in North Wales, Chris Grayling announced.
The Justice Secretary said the new jail is expected to bring around £23 million a year to the regional economy and create around 1,000 jobs.
Mr Grayling first discussed plans for a super-prison in January as he announced a raft of prison closures covering some 2,600 inmate places.
Chief Secretary to the Treasury Danny Alexander confirmed to the Commons that £100 million would be invested in the construction of the prison in 2015/2016 as part of a multi-billion investment programme.
The project will ultimately receive £250 million of investment from the Government.