Yarl's Wood immigration centre in Bedford is among those criticised for detaining people without good reason. 63% of the women detained there were released rather than removed, a report has said.
The findings by Independent Monitoring Boards (IMBs) also found that out of the 25,487 people who left detention in the UK last year, only 44% were returned from the UK to another country.
Some 40% were granted bail by the Home Office, while other cases were handled by an immigration judge.
IMBs are made up of volunteers appointed by justice ministers to scrutinise prison and detention centre conditions.
The report also said too many "vulnerable people" were detained during the year, including those with pre-existing mental health problems which "sharply deteriorated" in detention. It added: "The indefinite nature of detention also caused great distress."
The boards raised concerns over excessive use of restraints, saying: "At one point, over 90% of detainees from one centre were handcuffed for external appointments. After the IMB raised concerns, this significantly reduced."
Additionally, the report found that at Yarl's Wood, a young man with epilepsy and a learning disability suffered from deteriorating health due to the trauma and uncertainty of detention. He was released after concerns were raised.
However, the report also stated that the number of people in detention decreased after the Windrush scandal. There were fewer examples of people being held for excessive periods and conditions were seen to improve.
Rudy Schulkind, research and policy co-ordinator at the organisation Bail for Immigration Detainees (BID), said: "The impression given by these figures is that detention fails to achieve its stated aim in the majority of cases and is frequently used to coerce people into leaving the UK when the Home Office is in fact not in a position to enforce their removal."
"The Home Office treats the deprivation of liberty far too lightly and we urge the government to bring an end to this inhumane system altogether".
"Detention is an important part of the immigration system - but it must be fair, dignified and protect the most vulnerable," a Home Office spokesperson said.
"We have made significant improvements to our approach in recent years, but remain committed to going further."