A homeless charity in Taunton says the way servicemen and women are discharged from the military needs to change.
It is thought that one in ten people who don't have a home used to serve in the armed forces.
The Taunton Association for the Homeless is now calling on the Government to provide more support for people adapting to civilian life.
Joe, a former serviceman, left the army two years ago but he is now homeless.
On a number of occasions Joe has had to sleep in his car just to get shelter.
He said, "I chose to leave the army after losing my best friend in Afghanistan."
"Knowing what I know now, after being out, I probably would have stayed a little bit longer, even if I wasn't enjoying it, for the simple fact that I wasn't ready. So that big step is tough."
Joe is just one of around a 180 homeless people in Taunton, Wellington and Bridgwater - with a number of them thought to be military veterans.
They are being helped by the Taunton Association for the Homeless. The head of the charity, Justin Roxburgh, says they can only do so much and something needs to change with the way people like Joe are discharged from the military.
Justin said, "I think it needs to be done on a far more one-to-one basis rather than relying on the goodwill of charities like the Royal British Legion, and the other military charities and homeless organisations around the country as well."
"Rather than relying on them for the follow-through, I think the military services should do that themselves."
The charity is now raising money to convert a former pub into a hostel for 8 homeless veterans.
It will be just across the road from an existing 52 bed house run by Daryll Northover - a former military man himself.
He said, "I think the project will offer the support with mental health issues, drink issues, drug issues. So the project should offer all of that. But it can also offer a lot more."
"We have our own college - they can do courses, they can do building courses, woodland project courses, English, Maths. There's something for them to do all the time."
Former serviceman Joe lives in one of the charity's hostels. He is working and wants to get his own place soon.
It is thought that there is 7,000 people like Joe in the UK, who have ended up without a home after serving their country.
Joe said, "You can be skint in the army and you know that you've got a roof over your head, you've got three square meals a day and you've got all your friends around you who are like your family as well."
"So you don't have any worries about anything. So when you leave that, and you find that you maybe sometimes haven't got a roof over your head, you haven't got three square meals a day and, actually, you haven't got your friends around you."