Around 400 people live at Napier Barracks in Folkestone, Kent, where coronavirus has added to worries over conditions.
Residents, many of whom have crossed the English Channel in small boats, have described it as “unbearable” and say social distancing is impossible.
Listen to ITV News' coronavirus podcast
There have been reports of suicide attempts and earlier this month many residents went on hunger strike in protest at the conditions, which reportedly include 34 people sharing one shower.
The facility features barbed-wire topped fences and hosted Canadian troops in the Second World War.
A petition to shut down the site, along with a similar facility at a barracks in Wales, has amassed more than 10,000 signatures.
The Home Office, which commandeered the site last year, insists the accommodation in Kent is “safe, suitable, (and) Covid-compliant”.
An open letter to “all British citizens” from an asylum seeker living at Napier Barracks and reportedly signed by more than 200 other residents has been shared by refugee charity Choose Love.
The letter calls out Home Secretary Priti Patel and immigration minister Chris Philip over conditions at the site.
It says: “We came to this country to save our lives. Lives which were mostly in danger because of war and prosecution.
“Yet we found ourselves in an army camp and we are surrounded by fences and security guards.”
The undated letter says that at least 120 cases of coronavirus have been identified at the barracks with more test results pending.
The figure of 120 cases has been put to the Home Office, which has so far declined to comment on it.
A petition by Freedom from Torture to empty the barracks in Kent and Wales and close them down has racked up more than 10,000 signatures in less than two days.
Kolbassia Haoussou, lead survivor advocate at the charity, said: “A major crisis is unfolding in these unsanitary and dangerous places. Many of the people trapped here have low immune systems and mental health issues linked to the abuse they have fled.
“The Government has the power to end this nightmare now. Empty the barracks, close the camps, save lives.”
Bella Sankey, director of charity Detention Action, echoed calls to shut the barracks “before they are engulfed by tragedy”.
Immigration minister Chris Philp said: “We take the welfare of those in our care extremely seriously and asylum seekers can contact the 24/7 helpline run by Migrant Help if they have any issues.”
Use of Napier Barracks to house asylum seekers was initially authorised for six months under emergency provisions but the Home Office has said it is considering extending its use beyond the current deadline in around March.