Mark Gettleson from the Howard League for Penal Reform said the Government's proposals to give ex-prisoners supervision in the community is "an admission that short-term prison sentences are failing".
He added that prisoners are currently "being warehoused in enormous overflowing prisons, rather than being given something useful to do on the outside."
More top news
The letter was signed by the CBI, British Chambers of Commerce, the EEF, the Institute of Directors and the Federation of Small Businesses.
A spokesperson said Rory Stewart's remarks were in line with the British government’s stated position.
The minister's intervention will come a week after Trump decertified the landmark agreement claiming it was not in the US national interest.